belisoful / prado-webextract

dev-main 2026-11-04 05:22 UTC

Requires

• php: >=8.2.0
• ext-dom: *
• ext-mbstring: *
• embed/embed: ^4.6
• fivefilters/readability.php: ^3.3 || ^4.0
• nyholm/psr7: ^1.8
• oscarotero/html-parser: ^0.1.7
• pradosoft/prado: ^4.4@dev
• symfony/http-client: ^6.4.17 || ^7.4 || ^8.0
• symfony/http-foundation: ^6.4.41 || ^7.4.13 || ^8.0.13

Requires (Dev)

• friendsofphp/php-cs-fixer: ^3.94
• phpdocumentor/shim: ^3
• phpstan/phpstan: ^2.3.3
• phpunit/phpunit: ^10

Conflicts

• masterminds/html5: <2.7.5

BSD-3-Clause

• Brad Anderson

• framework
• oembed
• extension
• scraper
• prado
• readability
• ssrf

Published on Packagist.org on 2026-11-04 05:22 UTC

README

Fetch web pages safely and reduce them to clean text and metadata, for PRADO.

Give it a web address and it keeps, in one table row per address:

• the article as plain text, without the navigation, advertising, and footer around it;
• the title, summary, author, site name, language, representative image, and publication date;
• keywords, the canonical address, feeds, and a provider's embed code for a video, when the page offers them.

It is built for addresses the public submits. Private, loopback, and link-local networks are refused, including behind redirects and in IPv6 forms, and every request has limits on size, time, and redirects.

The work is done by established libraries:

Job	Library
Fetching, and the private network checks	symfony/http-client
The article text	fivefilters/readability.php, a port of Firefox's Reader View
Metadata: Open Graph, Twitter cards, JSON-LD, oEmbed	embed/embed

Installation

composer require belisoful/prado-webextract

It requires PHP 8.2 or newer and PRADO 4.4.

Configuration

<modules>
	<module id="db" class="TDataSourceConfig">
		<database ConnectionString="mysql:host=localhost;dbname=site" Username="site" Password="..." />
	</module>
	<module id="webextract" class="TWebExtractor" ConnectionID="db" />
</modules>

The table (web_extracts by default) is created on first use.

Property	Default
TableName	web_extracts
AutoCreateTables	true
MaxAttempts	3	tries before an address fails for good
RetryDelay	3600	seconds before the first retry; each later one waits twice as long
LeaseTime	300	seconds a queue worker holds an address
StripParameters	tracking parameters	query parameters taken off an address (utm_*, fbclid, ...)
ProcessOnEndRequest	0	addresses fetched after each response, for hosts with no cron
Timeout	10	seconds to wait for a server
MaxDuration	30	seconds a whole fetch may take
MaxBytes	2097152	the largest page read
MaxRedirects	5
UserAgent	identifies the package
AllowPrivateNetworks	false	never for addresses the public submits
ReadMetadata	true	read metadata with Embed, which may ask a provider's oEmbed service

Use

Fetching takes seconds, so queue addresses as they arrive and fetch them from cron:

$extractor = $this->getApplication()->getModule('webextract');
$extractor->queueAll($urls);

<module id="cron" class="Prado\Util\Cron\TCronModule">
	<job Name="webextract" Schedule="* * * * *" Task="webextract->processQueue(10)" />
</module>

Then read what was learned:

foreach ($extractor->findAll($urls) as $url => $extract) {
	if ($extract->getIsExtracted()) {
		echo $extract->getTitle(), ' - ', $extract->getSiteName(), "\n", $extract->getDescription();
	}
}

extract($url) fetches one address immediately for a caller that can wait. An address is fetched once; extract($url, true) fetches it again.

A failure that could go differently later (a timeout, a 503) is retried; one that cannot (a 404, a private address, a page too large) fails at once. PDFs, images, and other files that are not text are marked unsupported without being downloaded.

Events: onExtracting (after reading, before storing, so a handler can change the extract, for example putting a video's transcript in its text), onExtracted, and onExtractFailed.

The text is plain text and safe to show escaped. The image, canonical, icon, author, provider, and feed addresses are kept only when they are http or https ones, but they are still the page's choice. The embed meta entry holds a provider's embed HTML, which is third-party markup: decide whether to trust the provider before showing it.

extract($url, true) fetches an address again. When that fails in a way that could go differently later (a timeout, a 503), the earlier extract is kept, with the failure in its getError().

The table works on SQLite, MySQL, and PostgreSQL, and is created on first use. TableName is a plain SQL name, optionally after a schema name: news.web_extracts.

Development

composer fulltest runs the full check: compile, code style, static analysis, unit tests. composer integration installs the package into a throwaway PRADO application and checks it. See AGENTS.md for the conventions this package holds to, and its Security section before changing how anything is fetched.