CLI tool to crawl a website and check HTTP status code
This repository provides a tool to check the HTTP status code of every link on a given website.
This package can be installed via Composer:
composer global require spatie/http-status-check
This tool will scan all links on a given website:
http-status-check scan https://example.com
It outputs a line per link found:
When the crawling process is finished a summary will be shown.
By default the crawler uses 10 concurrent connections to speed up the crawling process. You can change that number by passing a different value to the
http-status-check scan https://example.com --concurrency=20
You can also write all urls that gave a non-2xx or non-3xx response to a file:
http-status-check scan https://example.com --output=log.txt
When the crawler finds a link to an external website it will by default crawl that link as well. If you don't want the crawler to crawl such external urls use the
http-status-check scan https://example.com --dont-crawl-external-links
By default, requests timeout after 10 seconds. You can change this by passing the number of seconds to the
http-status-check scan https://example.com --timeout=30
By default, the crawler will respect robots data. You can ignore them though with the
http-status-check scan https://example.com --ignore-robots
To run the tests, first make sure you have Node.js installed. Then start the included node based server in a separate terminal window:
cd tests/server ./start_server.sh
With the server running, you can start testing:
Please see CHANGELOG for more information on what has changed recently.
Please see CONTRIBUTING for details.
If you discover any security related issues, please email firstname.lastname@example.org instead of using the issue tracker.
You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.
Our address is: Spatie, Samberstraat 69D, 2060 Antwerp, Belgium.
We publish all received postcards on our company website.
Spatie is a webdesign agency based in Antwerp, Belgium. You'll find an overview of all our open source projects on our website.
Does your business depend on our contributions? Reach out and support us on Patreon. All pledges will be dedicated to allocating workforce on maintenance and new awesome stuff.
The MIT License (MIT). Please see License File for more information.