README

This bundle helps to manage async upload of files, from the browser to Openstack Swift services.

It avoids to handle files directly on the server, which consume disk space and IO, RAM and CPU.

[[TOC]]

How does it works ?

Schema

Current limitations

only openstack is supported

If you feel free to give help to remove those limitations, do not hesitate to propose some Merge Request.

Installation

This works with symfony 3.4

Register the bundle

class AppKernel extends Kernel
{
    public function registerBundles()
    {
        $bundles = array(
            // ...
            new ChampsLibres\AsyncUploaderBundle\ChampsLibresAsyncUploaderBundle()
        );

        return $bundles;
    }
}

Create the entity which will store the filename

Create an entity is the most used way, but it might exists other (store name in redis table, ...)

Here is an example with some metadata (i.e. key which encrypted the file).

The entity must implement ChampsLibres\AsyncUploaderBundle\Model\AsyncFileInterface.

namespace Chill\DocStoreBundle\Entity;

use Doctrine\ORM\Mapping as ORM;
use ChampsLibres\AsyncUploaderBundle\Model\AsyncFileInterface;
use ChampsLibres\AsyncUploaderBundle\Validator\Constraints\AsyncFileExists;

/**
 * Represent a document stored in an object store 
 *
 * 
 * @ORM\Entity()
 * @ORM\Table("chill_doc.stored_object")
 * @AsyncFileExists(
 *  message="The file is not stored properly"
 * )
 */
class StoredObject implements AsyncFileInterface
{
    /**
     * @ORM\Id()
     * @ORM\GeneratedValue()
     * @ORM\Column(type="integer")
     */
    private $id;
    
    /**
     * @ORM\Column(type="text")
     */
    private $filename;
    
    /**
     *
     * @var \DateTime
     * @ORM\Column(type="datetime", name="creation_date")
     */
    private $creationDate;
    
    
    public function __construct()
    {
        $this->creationDate = new \DateTime();
    }

    public function getObjectName()
    {
        return $this->filename;
    }

Note: There exists a validator which check the presence of the file in the repo after creation or edition. This validator is added to the class using this annotation:

 * @AsyncFileExists(
 *  message="The file is not stored properly"
 * )

Configure openstack container

Create a container to store your data. Currently, only v2 authentification is supported (with provider OVH).

Two parameters must be added :

a temp url key, for signing the URL ;
https headers to allow CORS request

# load environment variables
swift post mycontainer -m "Temp-URL-Key:mySecretKeyWithAtLeast20Characters"
swift post mycontainer -m "Access-Control-Allow-Origin: https://my.website.com https://my.other.website.com"

To allow CORS request from all url (which is preferable during test, development and bugging):

swift post mycontainer -m "Access-Control-Allow-Origin: *"

The container must have the following data and metadata:

$ swift stat mycontainer
                         Account: AUTH_abcde
                       Container: mycontainer
                         Objects: 0
                           Bytes: 0
                        Read ACL:
                       Write ACL:
                         Sync To:
                        Sync Key:
               Meta Temp-Url-Key: mySecretKeyWithAtLeast20Characters
Meta Access-Control-Allow-Origin: https://my.website.com https://my.other.website.com
                   Accept-Ranges: bytes
                 X-Iplb-Instance: 12308
                X-Storage-Policy: PCS
                   Last-Modified: Tue, 11 Sep 2018 14:52:16 GMT
                    Content-Type: text/plain; charset=utf-8

Further references:

Configure the bundle

# app/config/config.yaml
champs_libres_async_uploader:
    persistence_checker: 'path.to.your_service'
    openstack:
        os_username:          '%env(OS_USERNAME)%' # Required
        os_password:          '%env(OS_PASSWORD)%' # Required
        os_tenant_id:         '%env(OS_TENANT_ID)%' # Required
        os_region_name:       '%env(OS_REGION_NAME)%' # Required
        os_auth_url:          '%env(OS_AUTH_URL)%' # Required
    temp_url:
        temp_url_key:         '%env(ASYNC_UPLOAD_TEMP_URL_KEY)%' # Required
        container:            '%env(ASYNC_UPLOAD_TEMP_URL_CONTAINER)%' #Required
        temp_url_base_path:   '%env(ASYNC_UPLOAD_TEMP_URL_BASE_PATH)%' # Required. Do not forget a trailing slash
        max_post_file_size:   15000000 # 15Mo, exprimés en bytes
        max_expires_delay:    180
        max_submit_delay:     3600

Note Do not forget the trailing slash with parameter temp_url_base_path

Usage in form

One can use AsyncUploaderType to generate an hidden field with additionnal data to store the filename:

namespace Chill\DocStoreBundle\Form;

use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use ChampsLibres\AsyncUploaderBundle\Form\Type\AsyncUploaderType;


/**
 * Form type which allow to join a document 
 *
 */
class StoredObjectType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder
            ->add('filename', AsyncUploaderType::class)
            ;
    }
}

The browser is responsible for:

get signature for posting a file, using this url:

/asyncupload/temp_url/generate/post?expires_delay=180&submit_delay=3600

Parameters:

expires_delay: the delay before the signature will expire, in seconds ;
submit_delay: the delay before checking the file is arriving on openstack. This verification should be done server side.

Example of response:

 {

     "method": "POST",
     "max_file_size": 15000000,
     "max_file_count": 1,
     "expires": 1592301124,
     "submit_delay": 3600,
     "redirect": "",
     "prefix": "TUPyhlXvoApgJim",
     "url": "https://storage.gra.cloud.ovh.net/v1/AUTH_c123456/container/TUPyhlXvoApgJim",
     "signature": "abcdefghijklmnopqrstuvwxyz01234567890123"

 }

generating a random suffix for the filename;
pushing the file into openstack container using the prefix, key and signature given into "data" elements associated to input elements.

The filename will be composed of the prefix given by the data elements and the suffix chosen in 1.

Example of for the POST request:
```
      <--- url given by data --> <-- prefix  -->
 POST /v1/AUTH_c123456/container/TUPyhlXvoApgJim HTTP/1.1
```
Example of Data, for one file:
```
 -----------------------------336081439295927874898201087
 Content-Disposition: form-data; name="redirect"
```

-----------------------------336081439295927874898201087
Content-Disposition: form-data; name="max_file_size"

15000000
-----------------------------336081439295927874898201087
Content-Disposition: form-data; name="max_file_count"

1
-----------------------------336081439295927874898201087
Content-Disposition: form-data; name="expires"

1592300722
-----------------------------336081439295927874898201087
Content-Disposition: form-data; name="signature"

abcdefghijklmnopqrstuvwxyz01234567890123
-----------------------------336081439295927874898201087
Content-Disposition: form-data; name="file"; filename="v0Rl2qr"
Content-Type: application/octet-stream

.
<!-- file come here -->
-----------------------------336081439295927874898201087--
```

storing the filename into the hidden field.

Example of implementation:

the bundle has an example script, which should be ready for use ;
An example of script and usage which encrypt the file client side before pushing it into openstack container:

https://framagit.org/Chill-project/Chill-Doc-Store/-/blob/master/Resources/public/module/async_upload/uploader.js

This script use the dropzoneJS library.

It allow only one file per input.

Example of js request:

var
    fileInput = ev.target, // the file input element
    uniqid = fileInput.name,
    fileObject = fileInput.files[0],
    asyncFileInput = document.querySelector('input[type="hidden"][data-input-name="'+uniqid+'"]'),
    formData = new FormData(),
    fileName = asyncupload.makeid(),
    jsonData,
    objectName,
    existingAsyncFileInputValueJson,
    url = asyncFileInput.dataset.tempUrl
    ;

// Get the asyncupload parameters
window.fetch(url)
    .then(function(r) {
      // handle asyncupload parameters
      if (r.ok) {
        return r.json();
      } else {
        throw new Error('not ok');
      }
    }).then(function(data) {
      // upload to openstack swift
      if (fileObject.size > data.max_file_size){console.log("Upload file too large");}

      formData.append("redirect", data.redirect);
      formData.append("max_file_size", data.max_file_size);
      formData.append("max_file_count", data.max_file_count);
      formData.append("expires", data.expires);
      formData.append("signature", data.signature);
      formData.append("file", fileObject, fileName);

      // prepare the form data which will be used in next step
      objectName = data.prefix + fileName;

      jsonData = { "object_name": objectName };

      return window.fetch(data.url, {
        method: 'POST',
        mode: 'cors',
        body: formData
      });

    }).then(function(r) {
      if (r.ok) {
        console.log('Succesfully uploaded');

        // Update info in the form, as upload is successful
        if (asyncFileInput.value === "") {
          existingAsyncFileInputValueJson = { "files": [ jsonData ] };
        } else {
          existingAsyncFileInputValueJson = JSON.parse(asyncFileInput.value);
          existingAsyncFileInputValueJson.files.push(jsonData);
        }
        asyncFileInput.value = JSON.stringify(existingAsyncFileInputValueJson);

      } else {
        console.log('bad');
        console.log(r.status);
        // Handle errors

      }
    }).catch(function(err) {
      /* error :( */
      console.log("catch an error: " + err.name + " - " + err.message);
      alert("There was an error posting your images. Please try again.");
      throw new Error('openstack error: ' + err );
    });

Checking for the presence of the file

The validator AsyncFileExists will check the presence of the file when the form is submitted:

/**
 * Represent a document stored in an object store 
 *
 * 
 * @ORM\Entity()
 * @AsyncFileExists(
 *  message="The file is not stored properly"
 * )
 */
class StoredObject implements AsyncFileInterface
{
}

URL to retrieve a signature GET

You can also use a GET request to download the file using javascript:

GET /asyncupload/temp_url/generate/GET?object_name=abcdefhiI

Example of response:

{

    "method": "GET",
    "url": "https://storage.gra.cloud.ovh.net/v1/AUTH_c611d5d3f457449cb709793003282426/comedienbe/FVsbQVDS0dAIvb4eqWqbDI?temp_url_sig=f10ddb5516f1b1b197a5ce63f98e2056696577c7&temp_url_expires=1592303020"

}

The DELETE and PUT method are not allowed.

Show file in template (twig filters)

The URL of the file can be get using those functions:

{# asyncFile implements AsyncFileInterface or a string (filename) #}
<img src="{{ asyncFile|file_url }}" />

You can also use a GET request to the server to get a signature:

<!-- the generate_url will be the GET url described in previous section -->
<button data-get-url="{{ asyncFile|generate_url }}">

If the container is publicly available, you can simply use the access to the file:

<img src="https://storage.gra.cloud.ovh.net/v1/AUTH_c123456/container/{{ asyncFile }}" />

Security

Before printing a file url signature, you should ensure that the user does have the right to show it.

The generation of signature for DELETE and PUT method are not allowed from http requests. You can still generate it from the PHP code.

Limit the usage of openstack container

Sometimes, it happens that users select a file for upload, which is immediatly uploaded to openstack container. Then, the user remove the file in the UI, and select a second file (which is also uploaded) and the submit the form.

The first file uploaded remains recorded on the container.

After a while, file might bloat the container.

To prevent that, you can register POST signature and store them in a queue. Then, after the submit_delay is past, check for the presence of each file under the prefix and, if the file is not store in the dabase, remove it.

The event async_uploader.generate_url will be launched when a signature is generated.

You can listen on this event to get the generation of signature and implements your own logic. You should wait for the submit delay.

Apply logic on uploaded file (image resizing, ...)