LocalToolsOffline browser tools collection

Category: Developer tools

Robots.txt Generator

Generate a robots.txt file with allow, disallow and sitemap rules

Set a user-agent, allow and disallow paths, an optional crawl-delay and a sitemap URL to generate a ready-to-use robots.txt file. Runs entirely in your browser.

robots.txt
robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Everything on this page is processed in your browser. Nothing is uploaded.

What this tool does

A robots.txt file tells search engine crawlers which parts of a site they may and may not request, and can point them at a sitemap. This generator builds a standards-shaped file from a user-agent, a list of allowed paths, a list of disallowed paths, an optional crawl-delay, and a sitemap URL.

How to use it

  1. Set the user-agent, or leave it as * to match every crawler.
  2. List allowed and disallowed paths, one per line, and add a sitemap URL.
  3. Copy the generated robots.txt into the root of your site.

Privacy

This tool runs entirely in your browser. Your input is never uploaded, stored or shared — closing the tab removes it.

Frequently asked questions

Is anything about my site sent anywhere?
No. The file is assembled from your inputs in JavaScript running in the page, so nothing is uploaded.
Where does robots.txt need to live?
At the root of your domain, such as https://example.com/robots.txt, so crawlers can find it before requesting other pages.
Do all crawlers honor crawl-delay?
No. Crawl-delay is a widely supported convention but not a formal standard, so some crawlers ignore it entirely.

Back to all tools