Category: Developer tools
Robots.txt Tester
Check whether a robots.txt file allows or disallows a path for a user-agent
Paste robots.txt content, a user-agent and a path to see whether crawling is allowed, and which rule and group matched. Runs entirely in your browser.
Everything on this page is processed in your browser. Nothing is uploaded.
What this tool does
A robots.txt file can contain multiple user-agent groups and overlapping Allow and Disallow rules, which makes it easy to misjudge whether a specific path is actually blocked. This tool parses the pasted robots.txt text, selects the most specific matching user-agent group, and applies the most-specific-rule-wins precedence that Google's crawlers use: the matching rule with the longest path wins, and an equally specific Allow beats an equally specific Disallow.
How to use it
- Paste the robots.txt content into the input box.
- Enter the user-agent to test, such as * or Googlebot.
- Enter the path to test and read whether it is allowed, along with the matched group and rule.
Privacy
This tool runs entirely in your browser. Your input is never uploaded, stored or shared — closing the tab removes it.
Frequently asked questions
- Is my robots.txt sent anywhere?
- No. Parsing and matching run entirely in JavaScript in your browser tab.
- What happens if no rule matches the path?
- The path is allowed by default, which mirrors how real crawlers treat robots.txt: only explicit Disallow rules restrict access.
- Which precedence rule does this use?
- Most-specific-rule-wins: among the rules that match a path, the one with the longest path string applies. If two rules of equal length disagree, the tool follows Google's documented behavior and prefers the less restrictive Allow rule.