Category: Developer tools

robots.txt Generator

System ideaMissing a tool?

Create a robots.txt file with shared rules and crawler presets

Generate robots.txt locally with User-agent, Allow, Disallow, Crawl-delay, one Sitemap and presets for search, social, AI and SEO crawlers.

robots.txt

Applies to the default group and any crawler set to Allow or Block below. Google ignores Crawl-delay; Bing, Yandex and most others honor it.

Restricted directories are blocked wherever access is Allow. A crawler you block entirely is already blocked, so they have no extra effect there.

Crawler-specific rules

"Same as default" makes a crawler follow the default group above. "Allow" or "Block" gives it its own User-agent block with the paths and crawl-delay set above.

Search engines

Social link previews

AI crawlers

SEO crawlers

robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Everything on this page is processed in your browser. Nothing is uploaded.

What this tool does

This robots.txt generator builds the file body in your browser. Configure the User-agent: * group, default allow or block behavior, Disallow paths, Allow paths, optional Crawl-delay, one Sitemap URL, one custom user-agent and 14 fixed crawler presets. The presets cover search, social, AI and SEO crawlers such as Googlebot, Bingbot, GPTBot, Google-Extended, CCBot and AhrefsBot. A crawler set to default is omitted and falls through to the wildcard group. The same Allow and Disallow paths are repeated in every generated group, so the tool cannot scope one path rule to just one crawler. You can copy the output or download robots.txt from a Blob object URL. The tool does not test, validate, simulate matching, fetch an existing robots.txt, publish to your site, submit to webmaster tools, validate paths, add multiple Sitemap lines or output robots meta tags, X-Robots-Tag, Noindex, Host or Clean-param directives. Crawl-delay is written exactly as typed and is ignored by Googlebot. Settings stay in the browser.

How to use it

  1. Choose the default access mode and enter any shared Allow or Disallow paths.
  2. Add Crawl-delay, one Sitemap URL, a custom user-agent or crawler preset overrides if needed.
  3. Review the generated robots.txt, then copy it or download the file for your site root.

Privacy

This tool runs entirely in your browser. Your input is never uploaded, stored or shared — closing the tab removes it.

Frequently asked questions

How do I define the User-agent?
The wildcard User-agent: * group is always generated. You can add one custom user-agent and choose allow, block or default for the fixed crawler presets.
How should I use the Disallow directive?
Enter one path per line. LocalTools trims empty lines and writes one Disallow line for each path, but it does not validate paths or simulate matching.
Can I add my Sitemap to robots.txt?
Yes, one Sitemap line can be appended at the end. The tool does not validate the sitemap URL and does not support multiple Sitemap lines.
How do I add the generated robots.txt file to my website?
Copy or download the file and place it at your own site root. LocalTools does not upload, publish or submit it for you.
Can I set a different Disallow for just one crawler?
No. Crawler presets can be allowed, blocked or omitted, but the same Allow and Disallow path lists are repeated in each generated group.