🤖 Robots.txt Generator
Build a correct robots.txt for your blog, store or portfolio. Add rule groups per crawler, block AI training bots if you want to, and list your sitemaps.
✅ Free · No sign-up
robots.txt preview
How to use the Robots.txt Generator
- Choose the default behaviour for all crawlers.
- Add groups for specific crawlers (for example Googlebot or GPTBot) with Allow and Disallow paths, or apply a preset.
- Add your sitemap URLs.
- Copy or download the file and upload it to the root of your domain as
/robots.txt.
Example
- Input
- Allow everything, block GPTBot, sitemap https://example.com/sitemap.xml
- Output
- User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Why creators use it
- Keep crawlers out of thank-you pages, carts and internal search results.
- Opt out of AI model training crawlers that respect robots.txt.
- Tell search engines where your sitemap lives.
Good to know
- robots.txt is a request, not a lock: well-behaved crawlers follow it, bad bots ignore it. Never use it to hide private content.
- Disallowing a page does not remove it from Google if other sites link to it; use a noindex meta tag for that (and do not block that page in robots.txt).
- Google ignores Crawl-delay; Bing and some others respect it.
Frequently asked questions
Where do I put robots.txt?
At the root of the host, e.g. https://example.com/robots.txt. Each subdomain needs its own file.
What do * and $ mean in paths?
* matches any sequence of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks URLs ending in .pdf. Google and Bing support both.
Will blocking AI crawlers remove my content from existing AI models?
No. It only asks crawlers that honour robots.txt not to collect pages from now on.