passcreator/jobqueue-googlepubsub

A Flowpack.JobQueue.Common implementation to support Google Pub/Sub as backend

dev-master 2019-12-17 09:33 UTC

Requires

• ext-json: *
• flowpack/jobqueue-common: ^3.0.0
• google/cloud-pubsub: ^1.0.0

MIT

README

A job queue backend for the Flowpack.JobQueue.Common package based on Google Pub/Sub. If you don't know anything about Google Pub/Sub yet have a look at their tutorials here. In general messages are pulled from Pub/Sub with this package.\ Pub/Sub offers at least-once message delivery. This means your application needs to be able to handle messages that are delivered more than once.

Disclaimer: This is heavily inspired by the RabbitMQ package by t3n!

Setup

To use Google Pub/Sub as a backend for a JobQueue you need a project on Google Cloud Platform and a service account that has permission to (at least) publish and pull messages. This package is able to create topics (the term used in Pub/Sub for queues) and subscribers but to do this the service account needs to have administrative privileges for Pub/Sub.\ If you don't grant administrative permissions you need to create the topics and subscribers on your own using the name of the queue as name for the topic and subscriber.

Install the package using composer:

composer require passcreator/jobqueue-googlepubsub

Use the setup command to create a topic and the corresponding subscribers:

./flow queue:setup myqeue

Configuration

Currently configuration options are pretty basic. If you need more, feel free to create pull requests.\ Make sure to create a service account and download the credentials JSON file as described here.\ The JSON must be passed as a base64 encoded string in the configruation.

The simplest configuration for a job queue could look like this:

Flowpack:
  JobQueue:
    Common:
      queues:
        simple-queue:
          className: 'Passcreator\JobQueue\GooglePubSub\Queue\GooglePubSubQueue'
          executeIsolated: false
          
          options:
            
            serviceAccountConfiguration: 'base64-encoded-service-account-json'
            # specifying a prefix is optional and allows you to use the same GCP project/service-account
            # in multiple instances of your projects.
            queuePrefix: 'prefix_'
            # The number of seconds Pub/Sub waits for an acknowledgement of a message
            ackDeadline: 10
            allowedPersistenceRegions:
              - 'europe-west1'
              - 'europe-west2'

Delaying messages

Google Pub/Sub doesn't have a concept of delaying message delivery which means as soon as you publish a message it will be delivered to your subscribers (for this package the job:work command will immediately receive the message when pulling).\ In order to realize delayed messages this package simply ignores them which means it doesn't acknowledge the message. This will trigger Pub/Sub to re-deliver the message later but it's not perfectly accurate.\ This means that if you delay a message for 60 seconds it will be delayed at least 60 seconds.