Robots.txt Generator
Create robots.txt files for search engine crawlers
Rule 1
How to use
- Add user agents and rules
- Specify allowed/disallowed paths
- Generate and download robots.txt
Use cases
- SEO management
- Crawl control
- Website protection
- Search engine optimization
About the Robots.txt Generator
robots.txt is the file search engine and crawler bots read first when visiting your site. It declares which paths they may crawl and where to find your sitemap. This generator builds the file interactively — multiple user-agent blocks, allow/disallow rules, crawl delays — and outputs a valid, spec-compliant robots.txt you can drop straight into the root of your domain.
Features
- Multiple user-agent blocks
- Allow / Disallow path rules
- Crawl-delay directive
- Sitemap references
- One-click copy and download as robots.txt
How it works
- Add one or more User-agent rules (use * to apply to all bots).
- Under each, list paths to Allow or Disallow.
- Optionally add Crawl-delay and one or more Sitemap URLs.
- Copy or download the file and upload it to the root of your domain.
Frequently asked questions
Where does robots.txt live?
+
At the root of your domain — https://example.com/robots.txt. It must be served with text/plain content type.
Is robots.txt a security control?
+
No. It's a request, not an enforcement. Bad actors ignore it. Never use Disallow to 'hide' sensitive URLs — use authentication instead.
How do I block ChatGPT or other AI crawlers?
+
Add User-agent: GPTBot (and CCBot, ClaudeBot, Google-Extended, PerplexityBot, etc.) with Disallow: /. Note: this only blocks compliant bots.
Can I have multiple User-agent blocks?
+
Yes. Each block applies to the listed agent. Use User-agent: * as a catch-all last.
Does order matter?
+
For rules inside a single User-agent group, Google applies the most-specific rule. Across groups, the first matching User-agent block wins.