helturkey / php-webtrawler

v1.0.0 2026-11-11 14:36 UTC

Requires

• php: ^8.4
• ext-dom: *
• ext-json: *
• ext-libxml: *
• ext-xmlwriter: *
• guzzlehttp/guzzle: ^7.9
• guzzlehttp/promises: ^2.0
• symfony/css-selector: ^6.4 || ^7.3 || ^8.0
• symfony/dom-crawler: ^6.4 || ^7.3 || ^8.0

Requires (Dev)

• dekor/php-impity: ^1.0
• friendsofphp/php-cs-fixer: ^3.75
• orchestra/testbench: ^11.0
• phpstan/phpstan: ^2.1
• phpunit/phpunit: ^11.5 || ^12.0
• playwright-php/playwright: ^1.6

Suggests

• ext-intl: Required to fetch internationalized (non-ASCII) host names such as https://مثال.com/.
• ext-mbstring: Faster, broader charset conversion for non-UTF-8 pages (iconv is used as a fallback).
• ext-pcntl: Lets `scraper crawl` stop cleanly on Ctrl-C / SIGTERM and save its resume state.
• ext-posix: Enables ownership checks on browser-session directories.
• dekor/php-impity: Optional PSR-18 transport using curl-impersonate for browser-like TLS and HTTP/2 fingerprints. Requires the matching curl-impersonate binary.
• illuminate/support: Laravel 13 integration: service provider, Scrape facade, configuration and scraper:fetch command (Testbench 11.x in development tests).
• playwright-php/playwright: Optional Chromium automation for JavaScript rendering, browser sessions, and request/response capture. Requires Node.js 20+ and installed Playwright browsers.

MIT

• html
• crawler
• scraping
• web-scraping
• web-crawling
• playwright
• browser-automation
• adaptive-selectors
• network-capture
• tls-fingerprinting
• webtrawler

Published on Packagist.org on 2026-11-11 14:46 UTC

README

PHP Scraper is a PHP-first web scraping toolkit for HTTP fetching, DOM parsing, adaptive element matching, concurrent crawling, optional Chromium rendering, structured exports, a CLI, an MCP server, and Laravel integration.

Documentation

Choose your language:

• English: Full documentation — docs/en
• العربية: التوثيق الكامل — docs/ar
• Documentation language index

The documentation covers installation, fetching and parsing, HTTP security, adaptive selectors, spiders and resumability, transports and proxies, Playwright, CLI/MCP, Laravel, exporters, testing, troubleshooting, and API entry points.

Install

composer require helturkey/php-webtrawler

The core requires PHP 8.4+ and ext-dom, ext-json, ext-libxml, and ext-xmlwriter. Playwright, php-rnet, and Impity are optional integrations with additional requirements. See English requirements or المتطلبات بالعربية.

Minimal example

<?php

declare(strict_types=1);

use HElturkey\WebTrawler\Http\FetchOptions;
use HElturkey\WebTrawler\Scraper;

require __DIR__ . '/vendor/autoload.php';

$options = new FetchOptions(
    blockPrivateNetworks: true,
    allowedHosts: ['example.com'],
    maxBodyBytes: 10 * 1024 * 1024,
);

$scraper = Scraper::create(
    defaultOptions: $options,
    enforceDefaultSecurity: true,
);

$page = $scraper->get('https://example.com/');
echo $page->title() ?? 'Untitled';

Security note

For untrusted URLs, configure blockPrivateNetworks, an appropriate allowedHosts list, finite timeouts and a finite response-size limit, and set enforceDefaultSecurity: true. HTTP-level URL checks do not fully prevent DNS rebinding. Playwright routing is best-effort and is not a complete SSRF boundary; use network-level egress isolation for untrusted browser targets. See HTTP security and browser security.

Development checks

composer validate --strict
composer cs
composer stan
composer test
composer check

Optional integration commands and the full release checklist are documented in Testing and release and الاختبارات والإصدار.

Project links

• Issues
• Source