jooservices/crawlerx

A PHP 8.5+ Laravel package for structured web crawling with typed DTOs

v1.0.0 2026-06-07 22:32 UTC

Requires

• php: ^8.5
• illuminate/container: ^11.0|^12.0
• illuminate/support: ^11.0|^12.0
• jooservices/client: ^1.0
• jooservices/dto: ^1.0
• symfony/css-selector: ^8.1
• symfony/dom-crawler: ^8.1

Requires (Dev)

• captainhook/captainhook: ^5.23
• captainhook/plugin-composer: ^5.3
• friendsofphp/php-cs-fixer: ^3.65
• laravel/pint: ^1.18
• mockery/mockery: ^1.6
• phpmd/phpmd: ^2.15
• phpstan/phpstan: ^2.0
• phpstan/phpstan-phpunit: ^2.0
• phpstan/phpstan-strict-rules: ^2.0
• phpunit/phpunit: ^12.0
• squizlabs/php_codesniffer: ^3.8 || ^4.0

MIT

• JOOservices

• scraper
• crawler
• laravel
• dto
• http-client
• JOOservices

README

JOOservices CrawlerX is a PHP 8.5+ Laravel package for structured web crawling through site adapters and typed DTO results.

Package name: jooservices/crawlerx

What it does

CrawlerX gives Laravel code one small API for crawling supported sites:

• choose a registered site adapter
• pass a RequestDto with a URL, crawl type, page, and HTTP options
• receive either a ListDto for listing pages or an ItemDto for detail pages

All package DTOs extend JOOservices\Dto\Core\Dto.

Install

composer require jooservices/crawlerx

The package auto-registers JOOservices\CrawlerX\CrawlerXServiceProvider through Laravel package discovery.

To publish the configuration:

php artisan vendor:publish --tag=crawlerx-config

Quick example

use JOOservices\CrawlerX\Dto\RequestDto;
use JOOservices\CrawlerX\Enums\CrawlType;
use JOOservices\CrawlerX\Services\CrawlerXService;

$result = app(CrawlerXService::class)
    ->site('onejav')
    ->crawl(new RequestDto(
        url: 'https://onejav.com/2026/05/31',
        type: CrawlType::Listing,
        page: 1,
    ));

$payload = $result->toArray();

Detail crawls use the same service with CrawlType::Detail:

$item = app(CrawlerXService::class)
    ->site('onejav')
    ->crawl(new RequestDto(
        url: 'https://onejav.com/torrent/ymds282',
        type: CrawlType::Detail,
    ));

Supported sites

The default config registers:

• onejav => JOOservices\CrawlerX\Adapters\Onejav\OnejavCrawler
• 141jav => JOOservices\CrawlerX\Adapters\OneFourOneJav\OneFourOneJavCrawler
• ffjav => JOOservices\CrawlerX\Adapters\FfJav\FfJavCrawler

Sites are configured in config/crawlerx.php under sites.

Result DTOs

RequestDto accepts:

• url: target URL
• type: CrawlType::Listing or CrawlType::Detail
• page: page number, default 1
• options: request-specific HTTP options

ListDto contains:

• url
• page
• items, a list of ItemDto
• pagination, a PaginationDto

ItemDto contains:

• url
• externalId
• title
• meta, adapter-specific structured metadata

Development

composer lint
composer lint:all
composer lint:fix
composer test
composer test:coverage
composer check
composer ci

Start with:

• Documentation Hub
• Installation
• Quick Start
• Configuration
• Agent Instructions
• AI Skills Map

License

This package is licensed under the MIT License.