README

Submit web-scraping jobs and verify webhooks against api.scrappy.hu from any Laravel 9 / 10 / 11 / 12 app.

Requires PHP 8.1+ (the SDK uses readonly properties for typed response objects, which Laravel 9 + PHP 8.0 doesn't support — bump PHP to 8.1+ and you're fine on any Laravel 9.x).

composer require scrappy-hu/laravel

Installation

composer require scrappy-hu/laravel
php artisan vendor:publish --tag=scrappy-config

Then in .env:

SCRAPPY_API_KEY=sk_live_...
SCRAPPY_WEBHOOK_SECRET=...

Both come from https://scrappy.hu/dashboard/api-keys: the API key, and under Webhook the key's signing secret (Create a key-level secret, shown once). Every job of the key is signed with that one secret.

The ScrappyServiceProvider is auto-registered via package discovery, and the Scrappy facade is wired up.

Receive results by webhook

Route::scrappyWebhook('scrappy/webhook'); // POST, named `scrappy.webhook`

It verifies X-Scrappy-Signature with SCRAPPY_WEBHOOK_SECRET (401 if it doesn't match), answers 200 at once, and dispatches an event:

Event	When
`Scrappy\Events\JobCompleted`	the page answered; `$e->webhook->result` holds it (check `->httpStatus`, a 404 completes too)
`Scrappy\Events\JobFailed`	no usable answer (timeout, crash, a bot wall on every attempt); `$e->webhook->error` says why
`Scrappy\Events\WebhookTestReceived`	a test delivery from `Scrappy::webhooks()->test()`

Listen with a queued listener, so the receiver answers Scrappy quickly:

use Illuminate\Contracts\Queue\ShouldQueue;
use Scrappy\Events\JobCompleted;

final class StoreScrapedProduct implements ShouldQueue
{
    public function handle(JobCompleted $event): void
    {
        $webhook = $event->webhook;
        $product = Product::find($webhook->clientReferenceId); // your id, sent back
        $product?->applyScrape($webhook->result->html, $webhook->result->httpStatus);
    }
}

Scrappy retries a delivery that doesn't get a 2xx (after 1 min, 5 min, 30 min, 2 h and 6 h), so a delivery can arrive more than once: keep listeners idempotent.

The route sits outside the CSRF check (Scrappy signs instead). Put it behind no auth middleware: the signature is the authentication.

Submit jobs

use Scrappy\Facades\Scrappy;

// Many at once: POST /v1/jobs/batch, 100 per request, in order.
$created = Scrappy::jobs()->createMany(
    $products->map(fn ($p) => [
        'url' => $p->url,
        'client_reference_id' => (string) $p->id, // comes back in the webhook
        'priority' => 'low',                      // bulk refresh: behind on-demand jobs
        'webhook_url' => route('scrappy.webhook'),
        'options' => [
            'wait_until' => 'domcontentloaded',
            'block_trackers' => true,
        ],
    ])->all(),
    idempotencyKey: 'feed-198-'.today()->toDateString(), // safe to retry the whole call
);

// One job:
$job = Scrappy::jobs()->create(['url' => 'https://example.com', 'webhook_url' => route('scrappy.webhook')]);

Set the key's default webhook URL in the dashboard and you can leave out webhook_url.

Run an action (preview)

An action job runs steps in a real browser on a Scrappy worker (log in, add to cart, place an order), at most once. The spec format is in Scrappy's ACTIONS.md. Your plan needs action jobs enabled.

use Scrappy\Facades\Scrappy;
use Scrappy\Models\ActionStatus;

$result = Scrappy::actions()->run(
    spec: $iponOrderSpec,                       // array: origin, steps, output …
    params: ['product_id' => 123, 'quantity' => 2, 'max_net_price' => 15000],
    secrets: [                                  // what {{secrets.*}} refers to
        'username' => config('services.ipon.username'),
        'password' => config('services.ipon.password'),
    ],
    session: 'xupe-ipon',                       // continue this browser session (stays logged in)
    actionKey: "xupe-po-{$po->id}",             // never runs twice for the same key
    deadlineSeconds: 120,                       // must start by then; no change starts after it
);

match ($result->status) {
    ActionStatus::Succeeded => $po->markOrdered($result->output['order_id']),
    ActionStatus::Rejected  => $po->markRejected($result->message),   // nothing changed: safe to retry
    ActionStatus::Partial   => $po->needsManualCleanup($result),      // something changed and stayed
    ActionStatus::Uncertain,
    ActionStatus::Timeout   => $po->needsVerification($result),       // NEVER resubmit automatically
    default                 => null,
};

Secrets never leave your app in the clear. run() seals them to Scrappy's public key (libsodium sealed box, ext-sodium) before sending: in your queue, logs and Scrappy's database they are ciphertext, and Scrappy keeps them only while the job runs. They are only typed on or sent to origins (default: the spec's origin; pass origins: for more).
session is your name for the browser session: the first job creates it, its cookies carry the login to the next job, and one job runs per session at a time (one account's cart is never in two jobs). Scrappy::actions()->logout('xupe-ipon') starts it fresh.
actionKey: calling run() again with the same key never starts a second job while the first is anything but rejected; it waits for (or returns) that one.
Waiting: run() holds ?wait= requests until the job ends, by default up to the deadline plus the spec's timeout_seconds. Pass waitSeconds: to wait less; you then get Timeout, and $result->mutationStarted says whether to treat it as uncertain. The final outcome also arrives by webhook (below).
Dry run: dryRun: true runs everything up to the first step that would change something and stops (Rejected, reason dry_run); $result->dryRunResult says whether that step's element was found. Start every new spec with it.
Without waiting: submit(...) (same arguments), then get($id, wait: 25), wait($id, $seconds), cancel($id).

With Route::scrappyWebhook(), every ending also dispatches Scrappy\Events\ActionCompleted ($event->action is the ActionResult), and ActionReconciled when a late worker report settles an uncertain job.

Read a job

Results also stay readable for 24 hours after a job ends:

$job = Scrappy::jobs()->get($jobId);

if ($job->status === 'completed') {
    $status = $job->result?->httpStatus;
    $html = $job->result?->html;
}

Verify a webhook yourself

Route::scrappyWebhook() does this for you. For a hand-written receiver:

if (! Scrappy::webhooks()->verify($request->getContent(), $request->header('X-Scrappy-Signature'))) {
    abort(401);
}
$event = Scrappy::webhooks()->parse($request->getContent()); // Scrappy\Models\WebhookEvent

Pass the raw body ($request->getContent()): re-encoding the JSON changes the bytes and breaks the HMAC.

Test your webhook receiver

$result = Scrappy::webhooks()->test(); // the key's webhook URL, or pass one
// $result['delivered'] === true, $result['response_status'] === 200

The test delivery is signed with your key's secret, so it exercises the same verification as real deliveries (and fires WebhookTestReceived).

Upgrading from 0.1

Webhooks are verified with one key-level secret (SCRAPPY_WEBHOOK_SECRET) instead of a secret stored per job. Create it in the dashboard first; verify() throws a ScrappyException (config_error) without one.
Route::scrappyWebhook() replaces a hand-written receiver.
Scrappy::jobs()->createMany() submits batches.
JobCreated::$webhookSecretSource and JobResult::$outcome are new.

Account snapshot

$snap = Scrappy::me()->get();

echo $snap->planName();             // 'Pro'
echo $snap->monthlyUsed();          // 1234
echo $snap->monthlyRemaining();     // 8766

Errors

Every non-2xx response throws a typed exception you can pattern-match:

use Scrappy\Exceptions\{
    ScrappyException,
    AuthenticationException,
    RateLimitException,
    QuotaExceededException,
    ValidationException,
    NotFoundException,
};

try {
    Scrappy::jobs()->create(['url' => 'https://example.com']);
} catch (RateLimitException $e) {
    sleep($e->retryAfterSeconds());
    // retry…
} catch (QuotaExceededException $e) {
    // surface upgrade CTA to the user
    return redirect($e->upgradeUrl());
} catch (ValidationException $e) {
    foreach ($e->fieldErrors() as $field => $errors) {
        // render errors next to the form field
    }
} catch (AuthenticationException) {
    abort(500, 'Scrappy api key is missing or invalid');
} catch (ScrappyException $e) {
    Log::error('scrappy', [
        'code' => $e->errorCode(),
        'status' => $e->statusCode(),
        'payload' => $e->payload(),
    ]);
    throw $e;
}

Configuration

config/scrappy.php (after vendor:publish):

Key	Env var	Default	Notes
`api_key`	`SCRAPPY_API_KEY`	—	Required.
`timeout`	`SCRAPPY_TIMEOUT`	`30`	Per-call HTTP timeout (seconds).
`webhook_secret`	`SCRAPPY_WEBHOOK_SECRET`	—	The key's webhook signing secret (dashboard → API keys → Webhook). Required for receiving webhooks.
`replay_window_seconds`	—	`300`	Reject signatures older than this.

The base URL is intentionally not configurable from Laravel — every Laravel-driven instance points at https://api.scrappy.hu. Plain-PHP users can still override it via the Scrappy constructor for tests or unusual self-hosted setups.

Plain PHP usage

The package works outside Laravel too — instantiate the client directly:

$scrappy = new \Scrappy\Scrappy(
    apiKey: getenv('SCRAPPY_API_KEY'),
);
$job = $scrappy->jobs()->create(['url' => 'https://example.com']);

Testing your own code

The SDK throws on api errors instead of returning bad data, which makes mocking straightforward:

use GuzzleHttp\Client;
use GuzzleHttp\Handler\MockHandler;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Psr7\Response;

$handler = new MockHandler([
    new Response(201, [], json_encode([
        'job_id' => 'test-id',
        'status' => 'queued',
        'created_at' => '2026-05-08T10:00:00Z',
    ])),
]);
$scrappy = new \Scrappy\Scrappy('sk_live_test', guzzle: new Client([
    'handler' => HandlerStack::create($handler),
    'base_uri' => 'https://api.scrappy.hu/',
]));

API reference

Full reference + interactive API explorer:

https://scrappy.hu/docs — long-form reference
https://scrappy.hu/docs/api — interactive (Scalar)
https://scrappy.hu/openapi.json — OpenAPI 3.1 spec (machine-readable)

Versioning

This SDK follows semver. Breaking changes go in major versions; new methods + bug fixes go in minors / patches. Tracked at https://github.com/scrappy-hu/laravel/releases.

License

MIT.

scrappy-hu / laravel

Package info

Security

Statistics