diggin/diggin-robotrules

spidercrawlerrobotsrobotstxt

Overall: 8 installs
This month: 0 installs
Today: 0 installs

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Maintainer: sasezaki
Canonical: https://github.com/diggin/Diggin_RobotRules
Source: https://github.com/diggin/Diggin_RobotRules/tree/master
Issues: https://github.com/diggin/Diggin_RobotRules/issues