crwlr/crawler

Web crawling and scraping library.

v3.0.4 2024-12-17 23:11 UTC

Requires

• php: ^8.1
• ext-dom: *
• adbario/php-dot-notation: ^3.1
• chrome-php/chrome: ^1.7
• crwlr/html-2-text: ^0.1.0
• crwlr/robots-txt: ^1.1
• crwlr/schema-org: ^0.2|^0.3
• crwlr/url: ^2.1
• crwlr/utils: ^1.2
• guzzlehttp/guzzle: ^7.4
• psr/log: ^2.0|^3.0
• psr/simple-cache: ^1.0|^2.0|^3.0
• symfony/css-selector: ^6.0|^7.0
• symfony/dom-crawler: ^6.0|^7.0

Requires (Dev)

• friendsofphp/php-cs-fixer: ^3.57
• mockery/mockery: ^1.5
• pestphp/pest: ^2.3|^3.0
• phpstan/extension-installer: ^1.1
• phpstan/phpstan: ^1.4
• phpstan/phpstan-mockery: ^1.0
• phpstan/phpstan-phpunit: ^1.0
• spatie/invade: ^2.0
• symfony/process: ^6.0|^7.0

Suggests

• ext-zlib: Needed to uncompress compressed responses

MIT

• Christian Olear

• web
• scraper
• crawler
• bot
• crawling
• scraping
• scrape
• crawl
• crwlr

README

Library for Rapid (Web) Crawler and Scraper Development

This library provides kind of a framework and a lot of ready to use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with.

To give you an overview, here's a list of things that it helps you with:

• Crawler Politeness 😇 (respecting robots.txt, throttling,...)
• Load URLs using
  • a (PSR-18) HTTP client (default is of course Guzzle)
  • or a headless browser (chrome) to get source after Javascript execution
• Get absolute links from HTML documents 🔗
• Get sitemaps from robots.txt and get all URLs from those sitemaps
• Crawl (load) all pages of a website 🕷
• Use cookies (or don't) 🍪
• Use any HTTP methods (GET, POST,...) and send any headers or body
• Easily iterate over paginated list pages 🔁
• Extract data from:
  • HTML and also XML (using CSS selectors or XPath queries)
  • JSON (using dot notation)
  • CSV (map columns)
• Extract schema.org structured data in JSON-LD format from HTML documents
• Keep memory usage low by using PHP Generators 💪
• Cache HTTP responses during development, so you don't have to load pages again and again after every code change
• Get logs about what your crawler is doing (accepts any PSR-3 LoggerInterface)
• And a lot more...

Documentation

You can find the documentation at crwlr.software.

Contributing

If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).