README

A PHP 8.5+ URL-driven crawl and parse library for JAV catalog sites. Give CrawlerX a supported URL and it detects the site and page type, fetches the page through the appropriate HTTP or browser strategy, and returns typed DTOs.

use JOOservices\CrawlerX\CrawlerX;

$item = CrawlerX::url('https://onejav.com/torrent/ymds282')->crawl();

echo $item->entityType;                         // movie
echo $item->meta['movie']['external_id'];       // ymds282
echo $item->meta['movie']['code'];              // YMDS-282

Note

Packagist v1.0.0 is the retired Laravel implementation. This framework-agnostic rebuild is released as 1.3.1 and provides the API documented here.

Features

One public builder API for listing, detail, search, gallery, performer-listing, and performer-detail pages
Multi-hop discovery: list items carry a nextCrawlType so sub-listings (e.g. XCITY) crawl to terminal detail items
Automatic site and crawl-type detection from the URL
Typed immutable results built on jooservices/dto v3
HTTP fetching through jooservices/client v4.4.1+
Adaptive fallback through curl-impersonate, Playwright, stealth browser modes, Puppeteer, and FlareSolverr
Browser cookie handoff, challenge-page detection, and JavBus age-verification handling
Manifest-driven capability declaration validated against adapter implementations
Pagination metadata and structured non-throwing errors
Network-free CI tests backed by captured HTML fixtures
Docker CLI for running the same public crawl flow against live URLs

Requirements

PHP ^8.5
PHP extensions: dom, libxml
Composer
A way to run PHP: your own PHP 8.5, the project's Docker (docker compose), or a shared runtime configured with CRAWLERX_PHP
A Playwright browser service and a FlareSolverr service reachable by URL when you crawl browser-likely sites (see Runtime configuration)

Nothing is hard-wired to Docker: every service address and the PHP runner come from .env. Docker is one optional way to provide them.

Installation

Current development checkout

Place this repository next to the consuming application, then register it as a Composer path repository:

composer config repositories.crawlerx path ../crawlerx
composer require jooservices/crawlerx:@dev

The path installation uses this checkout's actual requirements, including jooservices/client v4 and jooservices/dto v3.

Tagged release

Released as 1.3.1 — install from Packagist:

composer require jooservices/crawlerx:^1.3

The retired v1.0.0 is the previous Laravel implementation and does not provide this API.

For development inside this repository, build the PHP 8.5 tooling image and install its dependencies:

make install

Usage

Automatic detection

CrawlerX::url($url)->crawl() is the normal entry point. The URL determines the adapter and crawl type:

use JOOservices\CrawlerX\CrawlerX;
use JOOservices\CrawlerX\Dto\CrawlItemResultDto;
use JOOservices\CrawlerX\Dto\CrawlListResultDto;

$list = CrawlerX::url('https://onejav.com/new')->crawl();
$item = CrawlerX::url('https://onejav.com/torrent/ymds282')->crawl();

assert($list instanceof CrawlListResultDto);
assert($item instanceof CrawlItemResultDto);

Explicit site, type, and page

Override detection when a URL is ambiguous or when the caller already knows the desired operation:

use JOOservices\CrawlerX\CrawlerX;
use JOOservices\CrawlerX\Enums\CrawlType;

$result = CrawlerX::url('https://en.1pondo.tv/list/?o=n&page=2')
    ->site('onepondo')
    ->type(CrawlType::Listing)
    ->page(2)
    ->crawl();

Supported crawl types are Listing, Detail, Search, Gallery, PerformerListing, PerformerDetail, and PerformerSearch.

Result DTOs

crawl() returns only CrawlItemResultDto or CrawlListResultDto. Item roots contain url, entity_type, meta, and next_crawl_type; movie data lives under meta.movie, and performer data under meta.performer. Listings declare the same entity_type and contain one entity per items[] entry. CrawlerX does not create a track_id; correlation belongs to the consuming application.

Multi-hop discovery

Some sites (e.g. XCITY) reach performers through several listing hops. A list item with a non-null next_crawl_type is an intermediate hop: crawl its URL again with that type and keep going until the item is terminal (next_crawl_type === null, i.e. a detail page).

$pending = [
    ['url' => 'https://xxx.xcity.jp/idol/', 'type' => CrawlType::PerformerListing],
];

while ($pending !== []) {
    $current = array_shift($pending);

    $result = CrawlerX::url($current['url'])
        ->site('xcity')
        ->type($current['type'])
        ->crawl();

    foreach ($result->items as $item) {
        if ($item->nextCrawlType === null) {
            // Terminal detail item — no further hop needed.
            continue;
        }

        $pending[] = ['url' => $item->url, 'type' => CrawlType::from($item->nextCrawlType)];
    }
}

Search

Sites that expose search declare the SearchCapable / PerformerSearchCapable contract. Pass the search-results URL with the matching crawl type; the query term is carried on the request as metadata:

use JOOservices\CrawlerX\Enums\CrawlType;

$results = CrawlerX::url('https://www.javbus.com/search/STARS-456')
    ->site('javbus')
    ->type(CrawlType::Search)
    ->query('STARS-456')
    ->crawl();

Pagination

A listing or performer-listing result also provides pagination:

$list->pagination->currentPage;
$list->pagination->lastPage;
$list->pagination->nextPage;
$list->pagination->nextUrl;
$list->pagination->hasNextPage;

Non-throwing crawl

Use tryCrawl() when failures should be returned as data instead of thrown:

$outcome = CrawlerX::url($url)->tryCrawl();

if ($outcome->failed()) {
    echo $outcome->error?->code->value;
    echo $outcome->error?->message;
}

Error codes include unsupported_url, ambiguous_url, adapter_not_found, blocked, parse_failed, unknown, not_found, gone, rate_limited, timeout, challenge, network, auth_required, and ssrf_blocked.

Upgrading from 1.2

The CrawlErrorCode enum adds not_found, gone, rate_limited, timeout, challenge, network, auth_required, and ssrf_blocked. If your application matches on error codes, handle these values explicitly; consumers that map unknown values to a generic parser-drift error should update that mapping. In jvmeta, unrecognized values currently map to parse_drift until its mapping adopts these new cases. rate_limited, timeout, challenge, and network are retryable by default. The other new codes are terminal by default; in particular, do not retry ssrf_blocked without changing the browser target or its allowlist.

Fetch options

Each adapter has a default fetch profile. Override it per request when a specific runtime strategy is required:

use JOOservices\CrawlerX\CrawlerX;
use JOOservices\CrawlerX\Dto\CrawlOptionsDto;
use JOOservices\CrawlerX\Dto\FetchOptionsDto;
use JOOservices\CrawlerX\Enums\FetchMethod;

$item = CrawlerX::url('https://en.jable.tv/videos/fjin-091/')
    ->options(new CrawlOptionsDto(
        fetch: new FetchOptionsDto(
            method: FetchMethod::ChromeStealth,
            noFallback: true,
        ),
    ))
    ->crawl();

Available methods are http, curl_impersonate, playwright, playwright_stealth, chrome_stealth, puppeteer_stealth, and flaresolverr. Available profiles are http_only, browser_likely, and adaptive.

Supported sites

CrawlerX currently registers 25 adapters. Capabilities below come from each adapter's current manifest.

Site	Slug	Listing	Detail	Gallery	Performer listing	Performer detail
141Jav	`141jav`	Yes	Yes	—	—	—
1Pondo	`onepondo`	Yes	Yes	—	—	—
Aisex	`aisex`	—	—	—	Yes	Yes
Avfan	`avfan`	Yes	Yes	—	—	—
Avfan Profiles	`avfan_profiles`	—	—	—	Yes	Yes
Avjoho	`avjoho`	—	—	—	Yes	Yes
Caribbeancom	`caribbeancom`	Yes	Yes	—	—	—
DUGA	`duga`	Yes	Yes	—	—	—
EPORNER	`eporner`	—	—	Yes	—	—
FC2 Content Market	`fc2`	Yes	Yes	—	—	—
FFJav	`ffjav`	Yes	Yes	—	—	—
HEYZO	`heyzo`	Yes	Yes	—	—	—
Jable	`jable`	Yes	Yes	—	Yes	Yes
JAV Database	`javdatabase`	—	—	—	Yes	Yes
JavBTC	`javbtc`	Yes	Yes	—	—	—
JavBus	`javbus`	Yes	Yes	—	Yes	Yes
JavDB	`javdb`	Yes	Yes	—	—	—
JAVLibrary	`javlibrary`	Yes	Yes	—	Yes	Yes
JavPhotos	`javphotos`	Yes	—	Yes	—	—
Minnano AV	`minnanoav`	Yes	Yes	—	Yes	Yes
MissAV	`missav`	Yes	Yes	—	—	—
OneJav	`onejav`	Yes	Yes	—	—	—
Tokyo-Hot	`tokyohot`	Yes	Yes	—	—	—
Warashi	`warashi`	—	—	Yes	Yes	Yes
XCITY	`xcity`	Yes	Yes	—	Yes	Yes

DUGA, Tokyo-Hot, Caribbeancom, HEYZO, and JavBus use accepted adult landing routes when a root listing URL would otherwise return an age-verification page.

Live sites change independently of this package. A supported adapter means the URL shape and parser are implemented; availability can still be affected by site downtime, blocking, region restrictions, or DOM changes.

Live crawl

The live tools read PLAYWRIGHT_URL and FLARESOLVERR_URL from .env (copy .env.example; real environment variables win) and start nothing themselves. Point .env at any running services. To use this project's own Docker services instead, start them and use the Docker values listed in .env.example:

docker compose --profile fetch up -d --wait node flaresolverr

Then crawl any URL. The command executes the same public builder flow and prints the complete result DTO as JSON:

docker compose run --rm php php tools/live-check.php \
  'https://www.javbus.com/en/NAMH-074'

Pass builder options when explicit routing is required:

docker compose run --rm php php tools/live-check.php \
  'https://www.javbus.com/en/NAMH-074' \
  --site=javbus \
  --type=detail

That command is equivalent to:

CrawlerX::url('https://www.javbus.com/en/NAMH-074')
    ->site('javbus')
    ->type(CrawlType::Detail)
    ->crawl();

Common CLI mappings:

CLI argument	Public API equivalent
`--site=javbus`	`->site('javbus')`
`--type=detail`	`->type(CrawlType::Detail)`
`--page=2`	`->page(2)`
`--try`	`->tryCrawl()`
`--fetch-profile=browser_likely`	`FetchProfile::BrowserLikely`
`--fetch-method=flaresolverr --no-fallback`	FlareSolverr with fallback disabled
`--fetch-chain=playwright,chrome_stealth,flaresolverr`	Explicit ordered fetch chain
`--timeout=30`	`HttpOptionsDto(timeout: 30)`
`--header='Accept-Language: en-US'`	HTTP request header override

Show every CLI option:

docker compose run --rm php php tools/live-check.php --help

Run one canonical target for every registered adapter, or only one adapter:

make live-check
make live-check SITE=--site=javbus

Stop and remove the browser sidecars when finished:

docker compose --profile fetch down

The PHP container owns CrawlerX execution and parsing. The node and flaresolverr services are long-running fetch helpers used only when the selected strategy requires a browser or challenge solver.

The fetch services publish loopback host ports by default. Set PLAYWRIGHT_HOST_PORT (default 3000), FIXTURE_SITE_HOST_PORT (default 8080), or FLARESOLVERR_HOST_PORT (default 8191) to avoid a port already used by another stack. To disable host publishing, use the tracked docker-compose.no-ports.yml override when starting the lab, or pass it through COMPOSE_FILE when using a Make target:

docker compose -f docker-compose.yml -f docker-compose.no-ports.yml --profile fetch up -d --wait fixture-site node flaresolverr
COMPOSE_FILE=docker-compose.yml:docker-compose.no-ports.yml make fetch-up

The browser sidecar sends all browser egress through a loopback-only proxy and rejects private, loopback, link-local, and metadata addresses, including every redirect hop. It resolves and pins the destination address before connecting, and fails closed when DNS resolution fails. Set CRAWLERX_BROWSER_ALLOWED_HOSTS on the node service to restrict fetches further; use comma-separated exact hosts or *.example.org patterns. The allowlist does not override address blocking. Private-network access is enabled only by the dedicated docker-compose.fetch-lab.yml test overlay. Requests blocked by this policy return the non-retryable ssrf_blocked error and stop the fetch fallback chain.

The browser image is built with CRAWLERX_VERSION set to the matching CrawlerX package release (for example, 1.3.1); the sidecar reports that value in its health response. With Compose, set it when building:

CRAWLERX_VERSION=1.3.1 docker compose build node

Compose's local image name remains jooservices/crawlerx:browser. If you create a versioned local image tag, use the same version as the package release.

Runtime configuration

Everything is configured through environment variables. Copy .env.example to .env and set them; the live tools (tools/live-check.php, tools/source-matrix.php, make live-check, make source-matrix) load that file, and real environment variables win. The Docker Compose file reads the same variables and keeps the Docker sidecar addresses only as defaults. A missing browser service is reported as "Browser service is not configured".

Variable	Purpose
`PLAYWRIGHT_URL`	One Playwright container for this node
`FLARESOLVERR_URL`	One FlareSolverr container for this node
`CRAWLERX_PLAYWRIGHT_SCRIPT`	Path to `playwright-fetch.mjs`
`CRAWLERX_PUPPETEER_SCRIPT`	Path to `puppeteer-stealth-fetch.mjs`
`CRAWLERX_CURL_IMPERSONATE`	curl-impersonate executable

Testing

make runs PHP with CRAWLERX_PHP from .env when it is set (for example a shared runtime), otherwise in php:8.5-cli-bookworm through this project's Docker Compose.

make build
make install
make test
make ci

make test runs the deterministic PHPUnit suites. make ci runs lint, static analysis, both coverage suites, and the 85% coverage checks.

Command	Purpose
`make build`	Build the PHP 8.5 Docker image
`make install`	Build and run `composer install` inside Docker
`make shell`	Open an interactive shell in the PHP container
`make validate`	Run `composer validate --strict`
`make lint`	Run Pint, PHPCS, PHPStan, PHPMD, and PHP-CS-Fixer
`make test`	Run all PHPUnit suites without coverage
`make test-coverage`	Generate Unit and Feature Clover reports
`make audit`	Run Composer's dependency security audit
`make ci`	Run lint, coverage suites, and coverage enforcement
`make fetch-up`	Start and health-check Node and FlareSolverr
`make live-check`	Crawl one live target for every adapter
`make fetch-down`	Stop and remove live-fetch sidecars

Fixture-based tests

CI never performs live HTTP requests. Tests use ClientBuilder::fake() and captured HTML under tests/Fixtures/:

use JOOservices\Client\Client\ClientBuilder;
use JOOservices\CrawlerX\CrawlerX;
use JOOservices\CrawlerX\Tests\Support\FixtureResponder;

ClientBuilder::fake();
FixtureResponder::for('GET', 'https://onejav.com/torrent/ymds282')
    ->file('onejav/detail-sample-1.html');

$result = CrawlerX::url('https://onejav.com/torrent/ymds282')->crawl();

When the client is faked, the fetch plan is intentionally restricted to HTTP, so tests cannot launch Playwright, Puppeteer, or FlareSolverr accidentally.

Refresh the configured live fixtures from the host when a site's DOM changes:

make fixtures

Fixture refresh uses the URLs declared in adapter manifests. Review captured HTML before committing it and keep live network access out of CI.

Architecture

CrawlerX facade
    -> immutable CrawlRequestBuilder
    -> URL/site/type detection
    -> fetch-plan selection and fallback chain
    -> site adapter parser
    -> CrawlItemResultDto or CrawlListResultDto

Adapter behavior and supported page types are declared under src/Adapters/*/manifest.json. CrawlerXFactory::reset() is available for long-running workers that need to release the process-wide runtime instance.

jooservices / crawlerx

Package info

Security

Statistics