swiftchase/queue-sqs-s3event

Zend Framework 2 module for Amazon S3 Event Notifications queue manager for SlmQueue

0.1.1 2015-04-05 23:53 UTC

Requires

• php: >=5.5
• slm/queue-sqs: ~0.4.0

BSD-3-Clause

• Marcus Welz

• queue
• zf2
• aws
• job
• sqs

README

Version 0.1 created by Marcus Welz.

Purpose

This module allows handling of SQS event notifications generated by S3.

For example, you might allow users to upload directly to an S3 bucket, and need to know when new file uploads complete in order to process them.

Requirements

• Zend Framework 2
• SlmQueue ~0.4.0
• SlmQueueSqs ~0.4.0

Installation

Add the library to composer:

$ composer require "swiftchase/queue-sqs-s3event:*"

Then, add the S3EventQueue module in your config/application.config.php. Afterwards, the S3EventQueue\Queue\S3EventQueueFactory is ready to manage SQS queues that contain S3 events notifications.

Follow the SlmQueueSqs documentation.

• The queue manager must be the new S3EventQueueFactory instead of SqsQueueFactory.
• In the queue configuration section, the queue must specify the job class to instantiate.

    'slm_queue' => [
        'queues' => [
            'bucket-name' => [
                'job_class' => 'My\Job\S3EventJob' // extends AbstractS3EventJob
            ]
        ],
        'queue_manager' => [
            'factories' => [
                'bucket-name' => 'S3EventQueue\Queue\S3EventQueueFactory'
            ]
        ]
    ]

How does this work?

The excellent SlmQueue allows for multiple queue backends, as well as multiple job types per queue. This is achieved by persisting metadata (the job class), and instantiating the correct job during unserialization. This happens in SlmQueue\Queue\AbstractQueue::unserializeJob.

The events generated by AWS obviously don't follow this format, so we need a way of overriding the job that gets instantiated, hence the need for the custom S3EventQueue, which is configured to a single job type.

Limitations / To-dos

It is expected that there's only a single event coming in per each SQS message. The queue manager will throw an exception if that is not the case. I haven't found documentation that indicates that multiple events could get aggregated, nor that this isn't the case. And while throwing an exception isn't pretty, it prevents potential data loss by missing an event.

Instead of having this custom queue manager, SlmQueue could be refactored to allow configuration of what type of jobs a particular queue will contain. The default could be using metadata to infer job types, a separate strategy could be to specify exactly what job is contained and treat all job data as contents (no metadata).