README

ScrapingBee for omnisense/omnicrawl, under any PHP host. Pages with or without JavaScript (GET /api/v1), the target's status, headers and final address reported, the family's User-Agent forwarded, no premium or stealth proxy, no retry through another proxy - for targets declared as one's own site or with written permission only, because every request goes through ScrapingBee's rotating proxies, which hide who is collecting.

It is reached through omnicrawl's gate only: every address is admitted first (the target's declaration, robots.txt read by the core, TDM reservations, the host's limits), the service is called after, and what it brings back is checked again and made a record with its provenance. Read what omnicrawl refuses to do, and why.

$registry = new Registry([new ScrapingBeeFactory($http)], [
    'scrapingbee' => ['factory' => 'scrapingbee', 'options' => ['api_key' => 'SCRAPINGBEE_API_KEY']],
], $gate);

The key goes by reference: SCRAPINGBEE_API_KEY names an environment variable (or a secret of the SecretsInterface given to the factory), never the key itself. Its usage policy is policy.json (limits, terms, what they say of lawful use), read on the day it says; the limits of each plan the service publishes are its plans, picked with the plan option (hobby (default), freelance, startup, business, business-plus, enterprise-14, -24, -41, -69, -120). See docs/index.md.

Tested on recorded answers written from the service's documentation; no test calls the service.

MIT licence.

omnicrawl / scrapingbee

Package info

Security

Statistics

README