README

SilverStripe module to store assets in Google Cloud Storage rather than on the local filesystem.

Note: This is a pre-release, and does not implement any kind of bucket policy for protected assets. There is an example ACL for public/protected assets here

This was initially based off of https://github.com/silverstripe/silverstripe-s3

Requirements

Silverstripe CMS 6
league/flysystem-google-cloud-storage 3

For Silverstripe CMS 4, use the 1.x releases of this module.

Environment setup

The module requires a few environment variables to be set. These are mandatory.

GC_KEY_FILE: JSON of your google service account json file
GC_BUCKET_NAME: The name of the cloud storage bucket bucket to store assets in.

For the GC_KEY_FILE environment variable, simply copy all contents into one line and place in your .env file like -

    GC_KEY_FILE={"type": "service_account","project_id": ...

Optional CDN

If your bucket's public files are served through a CDN or a custom domain, set GC_PUBLIC_CDN_PREFIX and public URLs will use it instead of https://storage.googleapis.com/[BUCKET_NAME]/public/.

    GC_PUBLIC_CDN_PREFIX="https://cdn.example.com/"

A file is then linked as https://cdn.example.com/assets/Uploads/file.jpg. If your CDN does not serve files from /assets/, change that part of the path by redeclaring the adapter:

---
Name: app-gcs-cdn
After:
  - "#silverstripegooglecloudstorage-cdn"
---
SilverStripe\Core\Injector\Injector:
  SilverStripe\Assets\Flysystem\PublicAdapter:
    class: SilverStripe\GoogleCloudStorage\Adapter\PublicCDNAdapter
    constructor:
      bucketAdapter: '%$SilverStripe\GoogleCloudStorage\Adapter\BucketAdapter'
      prefix: "`GC_PUBLIC_BUCKET_PREFIX`"
      visibility: null
      mimeTypeDetector: null
      cdnPrefix: "`GC_PUBLIC_CDN_PREFIX`"
      cdnAssetsDir: "cms-assets" # example of a custom assets folder name

Editing images with a CDN

The image editor added in Silverstripe CMS 6.3 replaces an image under its existing filename. As the URL would not change, your CDN would keep serving the cached image from before the edit.

When GC_PUBLIC_CDN_PREFIX is set, this module does not overwrite a published image. The edited image is added to your bucket under a new versioned filename and used instead, so bird.jpg becomes bird-v2.jpg.

If you choose to back up the original image, the backup keeps the name bird.jpg and uses the original file where it is, so no copy is made and its URL keeps working. Without a backup, the original file is removed when you publish the edit.

Installation

Define the environment variables listed above.
Install Composer from https://getcomposer.org
Run composer require obj63mc/silverstripe-google-cloud-storage

This will install the most recent applicable version of the module given your other Composer requirements.

Note: This currently immediately replaces the built-in local asset store that comes with SilverStripe with one based on Google Cloud Storage. Any files that had previously been uploaded to an existing asset store will be unavailable (though they won't be lost - just run composer remove obj63mc/silverstripe-google-cloud-storage to remove the module and restore access).

Configuration

Assets are classed as either 'public' or 'protected' by SilverStripe. Public assets can be freely downloaded, whereas protected assets (e.g. assets not yet published) shouldn't be directly accessed.

'public' assets are stored by default in a directory called 'public' in the root of your bucket. If you would like to change this prefix/path simply update the environment variable of GC_PUBLIC_BUCKET_PREFIX. You will want to configure this folder and any assets under it to have an ACL that anyone can read.

'protected' assets are stored by default in a directory called 'protected' in the root of your bucket. If you would like to change this prefix/path simply update the environment variable of GC_PROTECTED_BUCKET_PREFIX. You will want to configure this folder to be private and only your google project admins and the account service key have admin access.

Protected assets are linked to with a signed URL, which expires after 300 seconds by default. To change this:

SilverStripe\Core\Injector\Injector:
  SilverStripe\Assets\Flysystem\ProtectedAdapter:
    calls:
      - [setExpiry, [3600]]

Uniform bucket-level access

By default the visibility of a file is set with an ACL on its object, which is what the bucket set up below uses. If your bucket has uniform bucket-level access enabled, objects cannot have ACLs, so access to the public folder needs to be granted with IAM, and the adapters given the matching visibility handler:

---
Name: app-gcs-visibility
After:
  - "#silverstripegooglecloudstorage-flysystem"
---
SilverStripe\Core\Injector\Injector:
  SilverStripe\Assets\Flysystem\PublicAdapter:
    constructor:
      visibility: '%$League\Flysystem\GoogleCloudStorage\UniformBucketLevelAccessVisibility'
  SilverStripe\Assets\Flysystem\ProtectedAdapter:
    constructor:
      visibility: '%$League\Flysystem\GoogleCloudStorage\UniformBucketLevelAccessVisibility'

Performance

Details of each file (whether it exists, its size, type and so on) are cached to avoid a request to Google Cloud Storage each time they are needed. This uses the default Silverstripe cache, so for a site running on more than one server use a shared backend such as Memcached or Redis, as described in the Silverstripe caching documentation. The cache is cleared on flush. To keep it:

SilverStripe\GoogleCloudStorage\Adapter\CachedGoogleCloudStorageAdapter:
  flush_enabled: false

File hashes and image dimensions

Silverstripe checks the SHA1 hash of a file when resolving its URL, and reads the dimensions of an image when rendering it. With an empty cache that would mean downloading the file from Google Cloud Storage, so this module stores both as custom metadata on each object as it is written (sha1, plus width and height for images) and reads them back with the object's other details instead.

Files uploaded before this was introduced are given their metadata the first time their hash or dimensions are needed: the object is downloaded once, as it always was, and its metadata is then updated in place. Nothing needs to be run for this. If requests shouldn't write to the bucket, turn it off, and such files are simply downloaded to be hashed or measured as before:

SilverStripe\GoogleCloudStorage\Adapter\CachedGoogleCloudStorageAdapter:
  lazy_metadata_backfill: false

To update every file up front instead of as they are used, run the backfill task.

# See what would change
vendor/bin/sake tasks:GCSBackfillMetadata --dry-run

# Try it on one folder first
vendor/bin/sake tasks:GCSBackfillMetadata --path=Uploads --limit=10

# Everything
vendor/bin/sake tasks:GCSBackfillMetadata

The task can be stopped and re-run, as objects which already have a hash are skipped. Use --force to recalculate them.

Photos which EXIF says are on their side (typically portrait phone photos) are stored with the size in the file plus an orientation value, and the width and height are swapped when read if the image manager has autoOrientation on. This needs the PHP exif extension; without it JPEGs are not given dimensions and are measured from the image as before.

Configuring Google Cloud Storage Bucket

This is an example for setting up your Google Cloud Storage Bucket. This assumes you have Google Cloud SDK and command line tools installed. https://cloud.google.com/storage/docs/quickstart-gsutil

Create the Bucket - gsutil mb gs://[BUCKET_NAME]/
Get the default ACL, you will need the project id from the ACL gsutil acl get gs://[BUCKET_NAME]/
Set the bucket so your service account key can access the bucket (note only needed if not running on google app engine or cloud compute). Replace [YOUR_ACCOUNT_NAME]/the email address with your service account key email address.
```
 gsutil defacl ch -u gcloud-[YOUR_ACCOUNT_NAME]@appspot.gserviceaccount.com:OWNER gs://[BUCKET_NAME]
 gsutil acl ch -u gcloud-[YOUR_ACCOUNT_NAME]@appspot.gserviceaccount.com:OWNER gs://[BUCKET_NAME]
```

Lets create the 'public' folder and set it so anyone can read it.

 touch test.txt
 gsutil cp test.txt gs://[BUCKET_NAME]/public
 gsutil acl ch -r -u AllUsers:R gs://[BUCKET_NAME]/public

Your bucket should now be configured and have proper protected/public folders for your assets.

Auto Generated Assets from the CMS (Tinymce.js/Error Pages)

TinyMCE

Currently anytime you run a flush on your site and access the CMS, new versions of tinymce code and other JS may be outputted to your assets directory. Instead of referencing these from your cloud storage, this module now includes an adapter to make sure this stays on the local filesystem as intended.

ErrorPages - currently error pages static files will be uploaded to GCS. Due to this you would need to update the path that these are generated to by editing your main sites config .yml file, default set from this project is:

    SilverStripe\ErrorPage\ErrorPage:
        store_filepath: 'error-pages'

You can also turn off the static file generation via -

    SilverStripe\ErrorPage\ErrorPage:
      enable_static_file: false

obj63mc / silverstripe-google-cloud-storage

Package info

Security

Statistics