The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules

The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.

Version: 0.2.5
Depends: R (≥ 3.2.0)
Imports: Rcpp
LinkingTo: Rcpp
Suggests: covr, robotstxt, tinytest
Published: 2023-02-11
DOI: 10.32614/CRAN.package.spiderbar
Author: Bob Rudis (bob@rud.is) [aut, cre], SEOmoz, Inc [aut]
Maintainer: Bob Rudis <bob at rud.is>
BugReports: https://github.com/hrbrmstr/spiderbar/issues
License: MIT + file LICENSE
URL: https://github.com/hrbrmstr/spiderbar
NeedsCompilation: yes
In views: WebTechnologies
CRAN checks: spiderbar results

Documentation:

Reference manual: spiderbar.pdf

Downloads:

Package source: spiderbar_0.2.5.tar.gz
Windows binaries: r-devel: spiderbar_0.2.5.zip, r-release: spiderbar_0.2.5.zip, r-oldrel: spiderbar_0.2.5.zip
macOS binaries: r-release (arm64): spiderbar_0.2.5.tgz, r-oldrel (arm64): spiderbar_0.2.5.tgz, r-release (x86_64): spiderbar_0.2.5.tgz, r-oldrel (x86_64): spiderbar_0.2.5.tgz
Old sources: spiderbar archive

Reverse dependencies:

Reverse imports: robotstxt

Linking:

Please use the canonical form https://CRAN.R-project.org/package=spiderbar to link to this page.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.