Rule groups
Each group outputs one User-agent block with Allow, Disallow, and optional Crawl-delay directives.
robots.txt generator
robots.txt generator
Configure crawler access rules visually, preview the generated robots.txt instantly, then copy or download the file. Everything runs locally in this browser.
Frontend onlyRealtime outputNo remote validation
Common templates
Rule groups
Sitemaps
Local generation only
The tool does not fetch your website, validate live robots.txt files, call a backend, or save settings in browser storage.
What gets generated
Sitemaps
Each sitemap is appended as a Sitemap directive after the crawler rules.
Warnings
Empty User-agent values, repeated allow/disallow paths, and invalid crawl delays are flagged before you copy or download.
FAQ
Does this validate my live site?
No. It only generates text from the settings on this page and does not request remote URLs.
Which rule wins when Allow and Disallow conflict?
Crawler implementations can differ, so the page flags exact duplicate paths and leaves the final policy decision to you.
Is the output saved?
No. It is held only in the current page memory until you copy, download, reset, or refresh.