robotstxtr: Engine-Aware 'robots.txt' Policy with Google's Matcher Algorithm
Evaluates whether URLs may be crawled according to 'robots.txt' rules. A versioned engine-aware facade keeps neutral fetch evidence, per-engine status policy, robots product tokens, and matcher capabilities separate, returning explicit non-decisions when policy is undocumented or a matcher is unavailable. The available matcher is a bundled, upstream-test-validated snapshot of Google's open-source C++ 'robots.txt' matcher, preserving 'Googlebot' behavior. Fetches use a deterministic, conservative policy built on 'httr2', with fetch origins constructed by 'rurl'. All network access in tests, examples, and vignettes is mocked or served locally, so no internet connection is required to install, test, or check the package. This matcher derives from Google's 'robotstxt' project <https://github.com/google/robotstxt>, distributed under the Apache License 2.0; see the package NOTICE.
| Version: | 0.3.1 |
| Depends: | R (≥ 4.1.0) |
| Imports: | httr2 (≥ 1.2.0), rurl (≥ 3.0.1) |
| LinkingTo: | cpp11 |
| Suggests: | callr, httpuv, jsonlite, knitr, oysteR, rmarkdown, rosv, testthat (≥ 3.0.0) |
| Published: | 2026-10-24 |
| DOI: | 10.32614/CRAN.package.robotstxtr (may not be active yet) |
| Author: | Bart Turczynski 
[aut, cre, cph],
Google LLC [cph] (Bundled 'robots.txt' matcher: src/robots.cc,
src/robots.h, src/reporting_robots.cc, src/reporting_robots.h) |
| Maintainer: | Bart Turczynski <bartek at turczynski.pl> |
| BugReports: | https://gitlab.com/bart-turczynski/robotstxtr/-/issues |
| License: | MIT + file LICENSE |
| Copyright: | file inst/COPYRIGHTS robotstxtr copyright details |
| URL: | https://bart-turczynski.gitlab.io/robotstxtr/, https://gitlab.com/bart-turczynski/robotstxtr, https://bart-turczynski.r-universe.dev/robotstxtr |
| NeedsCompilation: | yes |
| SystemRequirements: | C++17 |
| Language: | en-US |
| Citation: | robotstxtr citation info |
| Materials: | README, NEWS |
| CRAN checks: | robotstxtr results |
Documentation:
| Reference manual: | robotstxtr.html , robotstxtr.pdf |
| Vignettes: |
Introduction to robotstxtr (source, R code) |
Downloads:
| Package source: | robotstxtr_0.3.1.tar.gz |
| Windows binaries: | r-devel: robotstxtr_0.3.1.zip, r-release: not available, r-oldrel: not available |
| macOS binaries: | r-release (arm64): robotstxtr_0.3.1.tgz, r-oldrel (arm64): robotstxtr_0.3.1.tgz, r-release (x86_64): robotstxtr_0.3.1.tgz, r-oldrel (x86_64): robotstxtr_0.3.1.tgz |
Linking:
Please use the canonical form https://CRAN.R-project.org/package=robotstxtr to link to this page.