PHP
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
5 402 9
Web crawling and scraping library.
2 882 302