tomverran member since: Apr 15, 2014
tomverran's packages
-
-
PHP
tomverran/robots-txt-checker
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
PHP
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page