Category: Developer tools

robots.txt tester

System ideaMissing a tool?

Check whether robots.txt allows or blocks a path

Paste robots.txt, enter a user-agent and path, and see whether crawling is allowed with the matched group and rule. Runs locally in your browser and does not upload data to LocalTo

ResultAllowedMatched group*Matched ruleAllow: /admin/public/

Everything on this page is processed in your browser. Nothing is uploaded.

What this tool does

Paste robots.txt, enter a user-agent and path, and see whether crawling is allowed with the matched group and rule. Runs locally in your browser and does not upload data to LocalTools. A robots.txt file can contain multiple user-agent groups and overlapping Allow and Disallow rules, which makes it easy to misjudge whether a specific path is actually blocked. This tool parses the pasted robots.txt text, selects the most specific matching user-agent group, and applies the most-specific-rule-wins precedence that Google's crawlers use: the matching rule with the longest path wins, and an equally specific Allow beats an equally specific Disallow. Runs locally in your browser and does not upload data to LocalTools.

How to use it

  1. Paste the input.
  2. Review the result and any issue details.
  3. Copy the result.

Privacy

This tool runs entirely in your browser. Your input is never uploaded, stored or shared — closing the tab removes it.

Frequently asked questions

How do I test whether robots.txt blocks a path?
Paste robots.txt, enter a user-agent and path, and see whether crawling is allowed with the matched group and rule. Runs locally in your browser and does not upload data to LocalTools.
Which rule wins when Allow and Disallow both match?
Paste robots.txt, enter a user-agent and path, and see whether crawling is allowed with the matched group and rule.
Can I see what the tool used for the result?
Yes. The page shows the result plus the matched items or issue details relevant to the tool.
Is my input uploaded?
No. The input, settings and result are processed locally in your browser.
What does this tool not support?
It does not fetch a live robots.txt file, crawl the site, change server files, generate robots meta tags, or model every bot-specific implementation; it only tests the pasted rules against one user-agent and path.