nguyenhiepvan/crawler

A simple, fast sitemap generator crawler

v2.2 2026-07-21 15:59 UTC

Requires

• php: ^8.4
• illuminate/console: ^10.0 || ^11.0 || ^12.0 || ^13.0
• illuminate/support: ^10.0 || ^11.0 || ^12.0 || ^13.0

Requires (Dev)

• orchestra/testbench: ^8.0 || ^9.0 || ^10.0 || ^11.0
• phpunit/phpunit: ^11.0

MIT

Published on Packagist.org on 2026-07-21 16:00 UTC

README

A fast, zero-dependency sitemap crawler and generator for PHP 8.4+.

Features

• PHP 8.4 Optimized: Built utilizing modern PHP 8.4 features like Property Hooks and Asymmetric Visibility.
• AI Ready: Capable of generating llms.txt and AI-optimized directory files. Can parse robots.txt for AI bots and respects <meta name="robots" content="noai"> tags.
• Zero dependencies: Extremely lightweight.
• Format Extensibility: Create XML sitemaps for Google, Markdown for LLMs, or write your own custom formatter.
• Backward Compatible: Existing CLI and procedural usage remains intact.

Requirements

• PHP 8.4 or higher
• cURL extension

Usage (Object-Oriented)

The library provides a clean OOP interface.

Standard XML Sitemap (For Google/Bing)

require 'vendor/autoload.php';

use SitemapGenerator\Crawler;
use SitemapGenerator\Format\XMLSitemap;

$crawler = new Crawler('https://example.com');
// Configure options using PHP 8.4 property assignments
$crawler->maxDepth = 2;
$crawler->enableFrequency = true;

// Instruct crawler to output standard XML
$crawler->setFormat(new XMLSitemap());

$crawler->generate('public/sitemap.xml');

AI-Optimized Directory (llms.txt for LLMs)

AI bots like GPTBot or Claude prefer clean text files to understand your site structure. You can easily generate an AI-friendly llms.txt:

require 'vendor/autoload.php';

use SitemapGenerator\Crawler;
use SitemapGenerator\Format\LLMSitemap;

$crawler = new Crawler('https://example.com');
// Instruct crawler to build an AI-optimized map
$crawler->setFormat(new LLMSitemap());

// Automatically exclude URLs blocked by 'noai' meta tags or robots.txt AI Bot rules
$crawler->excludeAILockedPages(true);

$crawler->generate('public/llms.txt');

Adding Media (Images & Videos) and GEO Data

You can add rich media or geographic tags to your sitemaps to optimize for Google Images, Google Videos, and Local Business SEO.

require 'vendor/autoload.php';

use SitemapGenerator\Crawler;
use SitemapGenerator\DTO\SitemapUrl;
use SitemapGenerator\DTO\SitemapImage;
use SitemapGenerator\DTO\SitemapVideo;

$crawler = new Crawler('https://example.com');
// Automatically extract basic images from <img> tags while crawling:
$crawler->indexImg = true;

// Or manually add a rich URL with Media and GEO
$url = new SitemapUrl('https://example.com/product');

$url->addImage(new SitemapImage(
    'https://example.com/product.jpg',
    'Product Title',
    'Product Caption'
));

$url->addVideo(new SitemapVideo(
    'https://example.com/thumb.jpg', // thumbnail
    'Product Video', // title
    'See the product in action', // description
    'https://example.com/video.mp4' // content URL
));

$url->setGeo('kml');

$crawler->add($url);

$crawler->generate('public/sitemap.xml');

Usage via Artisan (Laravel)

If you have installed this package inside a Laravel application, the Service Provider is automatically discovered, allowing you to use the artisan command directly.

Generate a sitemap for your current Laravel application (uses APP_URL):

php artisan sitemap:generate

Generate a sitemap for a specific external URL:

php artisan sitemap:generate https://another-site.com

Generate strictly for AI Agents (llms.txt):

php artisan sitemap:generate --format=llm

Usage (Legacy CLI)

You can still use the legacy script for simple CLI generation or CRON jobs. Configure the crawler by modifying the config file sitemap.config.php.

php sitemap.php file=/home/user/public_html/sitemap.xml site=http://www.mywebsite.com/

License

MIT License