Jump to letter: [
3ABCDEFGHIJKLMNOPQRSTUVWXYZ
]
perl-WWW-RobotRules - Database of robots.txt-derived permissions
- Description:
This module parses /robots.txt files as specified in "A Standard for Robot
Exclusion", at <http://www.robotstxt.org/wc/norobots.html>. Webmasters can
use the /robots.txt file to forbid conforming robots from accessing parts
of their web site.
Packages
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.