Search by

omnisense / omnicrawl

GlitchArt

Omnicrawl: fetch and extract web pages under any PHP host, through crawling services or directly, responsible by default - the target site's rules applied by the family itself before any service is called. robots.txt read and honoured per RFC 9309 for the family's own token, a usage policy per targe

Package info

github.com/glitchr-omnisense/omnicrawl

pkg:composer/omnisense/omnicrawl

Security

Advisories: 0

Aikido package health analysis

Transparency log

Statistics

Installs: 6

Dependents: 4

Suggesters: 0

Stars: 0

Open Issues: 0

1.x-dev 2026-11-10 08:41 UTC

Requires

Requires (Dev)

Suggests

  • omnicrawl/apify: Apify: runs over a set of addresses with the Cheerio Scraper actor, raw HTML and headers kept
  • omnicrawl/firecrawl: Firecrawl (v2): pages fetched, rendered or not, and batch runs over a set of addresses
  • omnicrawl/scrapingbee: ScrapingBee: pages fetched, rendered or not, the target's headers reported, for targets declared under a licence only (its rotating proxies hide who collects)
  • omnicrawl/zyte: Zyte API: pages fetched over HTTP, for targets declared under a licence only (its ban avoidance cannot be turned off)
  • omnisense/omnibridge: The host contracts; omnibridge/symfony and omnibridge/laravel give a shared limiter and store
  • symfony/http-client: The HTTP client the gate reads robots.txt with and the direct provider fetches with (any HttpClientInterface does)

Provides

None

Conflicts

None

Replaces

None

MIT 0dc44e95611669b6fb2db765f75e9c8e8c89b179

  • Marco Meyer <marco.meyer.woop@glitchr.io>

JSON-LDcrawlerrobots.txtopen-graphscrapingx-robots-tagresponsiblerate-limitframework-agnosticlink-previewprovenanceomnicrawlrfc9309tdmrep

Published on Packagist.org on 2026-11-10 08:50 UTC

This package is auto-updated.

Last update: 2026-11-10 08:50:30 UTC

README

Web pages fetched and extracted under any PHP host, through crawling services or directly, responsible by default: the target site's rules are applied by the family itself, before any service is called. One contract by capability, one package per provider, as every family of the Omni ecosystem, on omnisense/contracts.

  • robots.txt honoured, read by the core with the family's User-Agent and applied per RFC 9309; no option to ignore it, an override only for one's own site or with written permission, with a written reason.
  • A policy per target site, declared as data: allowed or refused, on what authority (own site, written permission, published terms, robots.txt only), with a reference and a date, its rate limits, what may be stored and republished. A site nobody declared gets the conservative default: single pages for previews, one request every 10 seconds and 100 a day per host, no rendering, no crawl run. Each application declares the sites it refuses (the family ships no list of third-party sites).
  • An identified User-Agent with a contact, required by every factory; per-host limits beside the provider's own, the stricter winning.
  • X-Robots-Tag, meta robots and TDMRep reservations read and honoured.
  • Provenance on every record: source, fetch time, provider, authority. Records are not open data.
  • No evasion: none of the services' block-bypassing features is exposed.

Contracts: page-fetch, extract (metadata, Open Graph, Twitter cards, JSON-LD, selectors), crawl-runs. Providers: a built-in direct fetcher, and omnicrawl/firecrawl, omnicrawl/scrapingbee, omnicrawl/zyte, omnicrawl/apify.

$gate = new Gate(Agent::parse('ExampleBot/1.0 (+https://www.example.org/bot; bot@example.org)'), $http, Targets::fromFile('targets.json'));
$registry = new Registry([new DirectFactory($http)], ['preview' => ['factory' => 'direct']], $gate);

$registry->extract('preview')->extract('https://news.example.com/story')->preview();

Read what omnicrawl refuses to do, and why, then how to declare a target and the rest.

MIT licence.