Packages from omnicrawl · Transparency log
-
PHP
omnicrawl/apify
Omnicrawl's Apify, under any PHP host: runs over a fixed set of addresses with the Cheerio Scraper actor (no link followed, one page at a time, no retry, no Apify proxy, robots.txt honoured by the actor too), the target's raw HTML, status and headers kept, behind omnicrawl's gate. Its limits and ter
-
PHP
omnicrawl/firecrawl
Omnicrawl's Firecrawl (API v2), under any PHP host: pages fetched as raw HTML and batch runs over a fixed set of addresses, behind omnicrawl's gate (robots.txt, the target's declaration, per-host limits, provenance) - the family's User-Agent sent to the target, the basic proxy only (never Firecrawl'
-
PHP
omnicrawl/scrapingbee
Omnicrawl's ScrapingBee, under any PHP host: pages fetched with or without JavaScript, the target's status, headers and final address reported, behind omnicrawl's gate - and only for targets declared under a licence (one's own site, written permission), because every request goes through ScrapingBee
-
PHP
omnicrawl/zyte
Omnicrawl's Zyte API, under any PHP host: pages fetched over HTTP with the target's headers, behind omnicrawl's gate - and only for targets declared under a licence (one's own site, written permission), because Zyte's ban avoidance cannot be turned off and may override the User-Agent. No browser mod