schliesser/sitecrawler

TYPO3 sitemap crawler

v2.0.1 2023-03-07 19:35 UTC

Requires

• php: ^7.4 || ^8.0
• ext-simplexml: *
• typo3/cms-core: ^10.4 || ^11.5 || ^12.2

Requires (Dev)

• friendsofphp/php-cs-fixer: ^3.0
• php-parallel-lint/php-parallel-lint: ^1.2
• sbuerk/typo3-cmscomposerinstallers-testingframework-bridge: ^0.0.1
• typo3/cms-composer-installers: ^3.1.3 || 4.0.0-RC1 || ^5.0
• typo3/testing-framework: ^6 || ~7.0@dev

GPL-2.0-or-later

• André Buchmann

README

This extension provides a Symfony command to crawl any sitemap including all sub sitemaps. It gathers all available urls and then calls each url. This way you can warm up the TYPO3 page cache. Any standard sitemap can be crawled: TYPO3, Shopware, ...

You can provide custom request headers in json format. For basic auth you need to base64 encode user:password in the header.

Since version 1.1.0 the sitecrawler can read robots.txt files to fetch all defined sitemaps from it.

Examples

Composer based:

vendor/bin/typo3cms sitecrawler:crawl 'https://www.example.com/sitemap.xml'

# with custom request headers
vendor/bin/typo3cms sitecrawler:crawl 'https://www.example.com/sitemap.xml' '{"Authorization": "Basic dXNlcjpwYXNzd29yZA==", "Cache-Control": "no-cache"}'

Non composer:

typo3/sysext/core/bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml'

Development

• Clone project and cd into the extension folder
• Install dependencies for tests: composer install
• Run tests with: composer run test