The 'Robots Exclusion Protocol' <http://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
Version: | 0.2.1 |
Depends: | R (≥ 3.2.0) |
Imports: | Rcpp |
LinkingTo: | Rcpp |
Suggests: | testthat, covr, robotstxt |
Published: | 2017-11-17 |
Author: | Bob Rudis (bob@rud.is) [aut, cre], SEOmoz, Inc [aut] |
Maintainer: | Bob Rudis <bob at rud.is> |
BugReports: | https://github.com/hrbrmstr/spiderbar/issues |
License: | MIT + file LICENSE |
URL: | https://github.com/hrbrmstr/spiderbar |
NeedsCompilation: | yes |
SystemRequirements: | C++11 |
Materials: | README NEWS |
CRAN checks: | spiderbar results |
Reference manual: | spiderbar.pdf |
Package source: | spiderbar_0.2.1.tar.gz |
Windows binaries: | r-devel: spiderbar_0.2.1.zip, r-release: spiderbar_0.2.1.zip, r-oldrel: spiderbar_0.2.1.zip |
OS X El Capitan binaries: | r-release: spiderbar_0.2.1.tgz |
OS X Mavericks binaries: | r-oldrel: spiderbar_0.2.1.tgz |
Old sources: | spiderbar archive |
Reverse imports: | robotstxt |
Please use the canonical form https://CRAN.R-project.org/package=spiderbar to link to this page.