Robots.txt Generator
Build a valid robots.txt with allow/disallow rules, multiple user-agents, sitemaps, and crawl-delay. Free.
100% client-side · no data leaves your browser
User-agent: *
About this tool
Create a correct robots.txt without memorizing the syntax. Add per-user-agent allow and disallow rules, list your sitemaps, and set an optional crawl-delay. The output updates live and can be copied or downloaded as robots.txt.
How to create a robots.txt file
Generate a search-engine robots.txt for your website.
- 1
Add rules
Set a user-agent and the paths to allow or disallow.
- 2
Add sitemaps
List your sitemap URLs and an optional crawl-delay.
- 3
Download
Copy or download robots.txt and place it at your site root.