evolvex / laravel-remote-operations
Correctness infrastructure for durable execution, ambiguity detection, safe recovery, and reconciliation of irreversible remote operations in Laravel.
Package info
github.com/abdulsalamalkhatib96/evolvex-laravel-remote-operations
pkg:composer/evolvex/laravel-remote-operations
Requires
- php: ^8.3
- laravel/framework: ^12.0|^13.0
Requires (Dev)
- orchestra/testbench: ^10.0|^11.0
- phpunit/phpunit: ^11.5|^12.0|^13.0
Suggests
None
Provides
None
Conflicts
None
Replaces
None
README
Correctness infrastructure for durable execution, ambiguity detection, safe recovery, and reconciliation of irreversible remote operations in Laravel.
Use it when a remote side effect may have executed even though your application did not receive a response: withdrawals, refunds, provider wallet operations, casino actions, affiliate payouts, SMS sends, webhooks, external provisioning, and similar operations.
This package does not promise exactly-once execution against arbitrary external systems. It prevents blind retries, makes ambiguous outcomes explicit, uses provider idempotency/status probes when available, and escalates cases that cannot be proven safe.
Guarantees and invariants
UNKNOWNis a first-class state, never treated as a normal exception.- A business operation is locally unique by
(type, business_key). - A retry cannot silently mutate the original command payload.
- The same provider idempotency key cannot be bound to two different operations; a DB unique constraint closes the concurrent race window.
- Remote side effects and direct queue handoffs are blocked while a DB transaction is open.
- Durable async dispatch and all automatic retries are persisted through an outbox.
- Execute and reconcile jobs are generation-fenced; stale delayed jobs cannot consume a future retry.
- Outbox queue handoffs have a reclaim lease, so a lost execute job is safely redelivered instead of stranding a
READYoperation. - A crashed send worker is recovered from
EXECUTINGtoUNKNOWNafter its lease expires. - A crashed probe worker is recovered from
RECONCILINGand safely probed again. - Unknown outcomes are probed before retry whenever the provider supports a status probe.
NOT_FOUNDcan remain non-definitive for eventually-consistent providers.- Idempotent retry respects the provider's idempotency TTL measured from the first send attempt.
- Provider safety capabilities are snapshotted at first send so a later deployment cannot silently reinterpret an in-flight ambiguous operation.
- Capability contracts are validated before the first send; declaring status-probe support without implementing
ProbesRemoteOperationsfails before any remote side effect. - Probe limits/backoff are scoped to the current send episode; a new send gets a fresh ambiguity/probe window.
next_action_atis enforced by workers, so stale/early jobs cannot bypass configured backoff.- Manual/unsafe retry requires explicit duplicate-risk acknowledgement and is audited.
- State transitions are stored append-only until the configured whole-operation retention window expires.
- Terminal command payload/idempotency plaintext can be scrubbed earlier while fingerprints/hashes remain for duplicate detection.
- Events implement Laravel's
ShouldDispatchAfterCommitcontract. - Sensitive command payloads and idempotency keys are encrypted at rest using Laravel's app key.
Installation
composer require evolvex/laravel-remote-operations php artisan migrate
Optional publishing:
php artisan vendor:publish --tag=remote-operations-config php artisan vendor:publish --tag=remote-operations-migrations
Requirements: PHP 8.3+ and Laravel 12/13.
Define a durable provider handler
Do not capture an in-memory closure as the only way to contact a provider. Reconciliation may happen in another worker hours later.
<?php namespace App\RemoteOperations; use Evolvex\RemoteOperations\Contracts\ProbesRemoteOperations; use Evolvex\RemoteOperations\Contracts\RemoteOperationHandler; use Evolvex\RemoteOperations\Domain\Enums\RetrySafety; use Evolvex\RemoteOperations\Domain\Enums\TransportPhase; use Evolvex\RemoteOperations\Domain\ValueObjects\ProbeConsistency; use Evolvex\RemoteOperations\Domain\ValueObjects\ProbeOutcome; use Evolvex\RemoteOperations\Domain\ValueObjects\ProviderCapabilities; use Evolvex\RemoteOperations\Domain\ValueObjects\RemoteOperationContext; use Evolvex\RemoteOperations\Domain\ValueObjects\SendOutcome; use Illuminate\Http\Client\ConnectionException; use Illuminate\Support\Facades\Http; final class ProviderXWithdrawal implements RemoteOperationHandler, ProbesRemoteOperations { public function provider(RemoteOperationContext $context): string { return 'provider-x'; } public function capabilities(RemoteOperationContext $context): ProviderCapabilities { return ProviderCapabilities::make( supportsIdempotency: true, supportsStatusProbe: true, retrySafety: RetrySafety::IDEMPOTENT, idempotencyTtlSeconds: 86_400, probeConsistency: ProbeConsistency::eventual( visibilityDelaySeconds: 5, definitiveNotFoundAfterSeconds: 120, ), maxSendDurationSeconds: 30, ); } public function send(RemoteOperationContext $context): SendOutcome { try { $response = Http::withHeaders([ 'Idempotency-Key' => $context->idempotencyKey(), ])->timeout(30)->post('https://provider.example/withdrawals', $context->command); } catch (ConnectionException $e) { return SendOutcome::unknown( reason: 'connection_exception', phase: TransportPhase::UNKNOWN, metadata: ['endpoint' => 'https://provider.example/withdrawals', 'http_method' => 'POST'], ); } if ($response->successful() && $response->status() !== 202) { return SendOutcome::success( externalId: $response->json('id'), metadata: [ 'provider_status' => $response->json('status'), 'provider_request_id' => $response->header('X-Request-Id'), 'endpoint' => 'https://provider.example/withdrawals', 'http_method' => 'POST', ], httpStatus: $response->status(), ); } if ($response->status() === 202) { return SendOutcome::remotePending( externalId: $response->json('id'), metadata: ['provider_status' => 'processing'], ); } if ($response->status() === 422) { return SendOutcome::definiteFailure('provider_validation_rejected', httpStatus: 422); } return SendOutcome::unknown( reason: 'provider_5xx_with_unknown_commit_state', phase: TransportPhase::RESPONSE_RECEIVED, httpStatus: $response->status(), ); } public function probe(RemoteOperationContext $context): ProbeOutcome { $response = Http::get('https://provider.example/withdrawals/by-reference/'.$context->businessKey()); if ($response->status() === 404) { return ProbeOutcome::notFound(); } return match ($response->json('status')) { 'completed' => ProbeOutcome::succeeded($response->json('id')), 'failed' => ProbeOutcome::failed('provider_failed'), default => ProbeOutcome::pending($response->json('id')), }; } }
Synchronous execution
use Evolvex\RemoteOperations\Facades\RemoteOperation; $result = RemoteOperation::for('withdrawal', $withdrawal->uuid) ->handler(\App\RemoteOperations\ProviderXWithdrawal::class) ->subject($withdrawal) ->idempotencyKey($withdrawal->uuid) ->externalReference('withdrawal:'.$withdrawal->uuid) ->command([ 'withdrawal_id' => $withdrawal->id, 'reference' => $withdrawal->uuid, 'amount' => (string) $withdrawal->amount, 'currency' => $withdrawal->currency, ]) ->execute();
execute() is intentionally rejected when any resolved Laravel DB connection has an open transaction.
Durable asynchronous execution
dispatch() uses the outbox by default:
$operation = RemoteOperation::for('refund', $refund->uuid) ->handler(ProviderXRefund::class) ->idempotencyKey($refund->uuid) ->command([...]) ->dispatch();
If you explicitly want a non-outbox queue handoff, use dispatchDirect(). It is blocked inside DB transactions.
Transactional outbox
DB::transaction(function () use ($withdrawal) { RemoteOperation::for('withdrawal', $withdrawal->uuid) ->handler(ProviderXWithdrawal::class) ->idempotencyKey($withdrawal->uuid) ->command([...]) ->dispatchAfterCommit(); });
By default, if the active business transaction is on a different database connection from the remote-operation tables, the package throws NonAtomicOutboxException.
State model
CREATED -> READY -> EXECUTING
| | |
| | +-> CONFIRMED_FAILURE
| +----> REMOTE_PENDING -> RECONCILING
+-------> UNKNOWN --------> RECONCILING
RECONCILING -> CONFIRMED_SUCCESS
-> CONFIRMED_FAILURE
-> REMOTE_PENDING
-> UNKNOWN
-> RETRY_SCHEDULED -> READY
-> MANUAL_REVIEW
Recovery rules
| Provider idempotency | Status probe | Ambiguous outcome | Kernel behavior |
|---|---|---|---|
| yes | yes | timeout | wait for visibility window, probe first; same-key retry only when safe |
| yes | no | timeout | retry only while idempotency window is valid |
| no | yes | timeout | probe; never blind-retry |
| no | no | timeout | manual review |
| any | eventual probe | early 404 | wait and probe again until 404 is definitive |
| any | any | request definitely not sent | retry only if retry policy permits |
Scheduler
use Illuminate\Support\Facades\Schedule; Schedule::command('remote-operations:dispatch-outbox')->everyMinute()->onOneServer(); Schedule::command('remote-operations:recover-stale')->everyMinute()->onOneServer(); Schedule::command('remote-operations:reconcile-due')->everyMinute()->onOneServer(); Schedule::command('remote-operations:prune')->daily()->onOneServer();
Operational commands
php artisan remote-operations:list --state=unknown php artisan remote-operations:show 01K... php artisan remote-operations:reconcile 01K... php artisan remote-operations:reconcile-due php artisan remote-operations:recover-stale php artisan remote-operations:dispatch-outbox php artisan remote-operations:stats php artisan remote-operations:doctor php artisan remote-operations:providers php artisan remote-operations:prune
Manual resolution:
php artisan remote-operations:resolve 01K... success \
--external-id=TX-123 \
--reason="confirmed by provider support" \
--actor=admin:42
Unsafe retry:
php artisan remote-operations:retry 01K... \
--acknowledge-duplicate-risk \
--reason="provider confirmed operation never existed" \
--actor=admin:42
Security
- Command payload is stored using Laravel's encrypted cast.
- Idempotency key is encrypted; a SHA-256 hash is stored separately.
- Attempt/probe metadata passes through
MetadataRedactor. - Raw
endpointmetadata is hashed for attempt correlation and removed from stored metadata. - Do not place PAN/CVV/secrets in metadata.
Testing
composer lint
composer test
Limits
- This is not a payment gateway SDK.
- This is not a generic Saga/workflow engine.
- The provider adapter must classify evidence conservatively.
- A provider with neither idempotency nor a reliable status/reference lookup cannot be made exactly-once.
MANUAL_REVIEWis the safe result.
See docs/correctness-model.md, docs/provider-contract.md, docs/failure-matrix.md, and docs/operations-runbook.md.