Keep model crawlers out while search crawlers carry on
Runs in your browser. Nothing you type is sent anywhere. Updated on 24 Oct 2026.This starts with a file that welcomes Googlebot and Bingbot and turns away the crawlers that collect pages for model training, each named with the token its operator publishes. The list says who runs each one and what it is for, so you can keep the ones you want. Be honest with yourself about what this achieves: it is a request, and only crawlers that choose to read it obey.
The page opens with the Block AI crawlers preset: search crawlers allowed and each AI crawler disallowed in its own group.
Open AI crawler tokens and who runs them to see who runs each crawler and what it collects.
Press the remove button on the group of any crawler you want to let in.
Replace the example address under Sitemap addresses with your own.
Download robots.txt and place it at the root of your domain.
This robots.txt generator builds the file rule by rule: a group per crawler, allow and disallow paths, a crawl delay where one is honoured, and sitemap lines. Start from a preset, or paste a file you already have and read what every line actually does in plain words.
The presets cover the jobs people come for. Allow everything, which is also what having no file at all means. Block everything, for a staging copy that must never compete with the live site. A typical site that keeps carts, admin pages and internal search results out. And a file that welcomes search crawlers while asking the model-training crawlers to stay out, each named with the token its operator publishes.
One thing worth knowing before you block anything: robots.txt stops a page being read, not listed. An address already in the index can linger as a bare link. For a page that has to disappear, allow the crawl and serve a noindex tag. Related tools: Sitemap Generator and Meta Tag Generator.
The rest of the article, and all 6 questions about Robots.txt Generator, are on the Robots.txt Generator page.
In the root of the domain, at https://example.com/robots.txt, served as plain text. It applies to that host and port only, so a subdomain needs its own file. Crawlers do not look anywhere else, and a file in a subfolder is ignored.
Two lines: User-agent: * then Disallow: /. That is the right file for a staging copy. On a live site it removes your traffic within days, so check which environment you are editing before you deploy it.
Only for crawlers that choose to read the file. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and PerplexityBot all publish tokens and state that they obey robots.txt. A scraper that ignores the file is unaffected, and robots.txt is not a security control.
No. Googlebot ignores Crawl-delay entirely and sets its own rate, which you can lower in Search Console. Bing, Yandex and Seznam do honour it. A value above about 30 seconds means a large site will never be crawled fully.
Still stuck? Ask about Robots.txt Generator and anybody who knows can answer. No account needed.
Same job, other tools: Random IP Address Generator – IPv4 addresses for logs and tests, drawn only from the ranges reserved for documentation · Random Name and Email Generator – Names, emails and phone numbers that are safe to use in a demo.
On the related tools: CSS Text Shadow Generator – Make a CSS text-shadow for glows, outlines and 3D lettering, with a live preview.
In the collection SEO Tools.