Build a robots.txt with presets, then check what every line really does
Runs in your browser. Nothing you type is sent anywhere. Updated on 24 Oct 2026.A robots.txt controls what a crawler may fetch, and one wrong line can hide an entire site. Build one from presets, then check what each rule really does against a URL you type in. The 2 directives that matter most, Allow and Disallow, are tested live.
Press a chip under Preset, such as Typical site or WordPress, to start from common rules.
In each group, type the User-agents and the paths under Disallow and Allow, one per line.
Press Add a group for a crawler that needs different rules.
Add your sitemap under Sitemap addresses, one per line.
Press Copy, or download robots.txt and upload it to the root of your site.
This robots.txt generator builds the file rule by rule: a group per crawler, allow and disallow paths, a crawl delay where one is honoured, and sitemap lines. Start from a preset, or paste a file you already have and read what every line actually does in plain words.
The presets cover the jobs people come for. Allow everything, which is also what having no file at all means. Block everything, for a staging copy that must never compete with the live site. A typical site that keeps carts, admin pages and internal search results out. And a file that welcomes search crawlers while asking the model-training crawlers to stay out, each named with the token its operator publishes.
The checker is the part most people need. It catches a path with no leading slash, a rule that sits above its User-agent line and therefore applies to nobody, a relative sitemap address, a Noindex line that Google stopped reading in 2019, and a Crawl-delay under a group that ignores it.
One thing worth knowing before you block anything: robots.txt stops a page being read, not listed. An address already in the index can linger as a bare link. For a page that has to disappear, allow the crawl and serve a noindex tag. Related tools: Sitemap Generator and Meta Tag Generator.
In the root of the domain, at https://example.com/robots.txt, served as plain text. It applies to that host and port only, so a subdomain needs its own file. Crawlers do not look anywhere else, and a file in a subfolder is ignored.
No. It stops the page being read. A blocked address can still be listed as a bare link with no description, because the crawler never fetches the page and so never sees a noindex tag. To remove a page from results, allow the crawl and add a robots meta tag.
Two lines: User-agent: * then Disallow: /. That is the right file for a staging copy. On a live site it removes your traffic within days, so check which environment you are editing before you deploy it.
Only for crawlers that choose to read the file. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and PerplexityBot all publish tokens and state that they obey robots.txt. A scraper that ignores the file is unaffected, and robots.txt is not a security control.
No. Googlebot ignores Crawl-delay entirely and sets its own rate, which you can lower in Search Console. Bing, Yandex and Seznam do honour it. A value above about 30 seconds means a large site will never be crawled fully.
A Sitemap line is not required, but it is the cheapest way to tell every crawler where your URL list is, including ones you never submit anything to. Use the full address, write one line per sitemap, and put them at the end of the file.
Still stuck? Ask about Robots.txt Generator and anybody who knows can answer. No account needed.
Same job, other tools: Meta Description Generator – Write a description that fits the snippet instead of being cut off.
On the related tools: CSS Text Shadow Generator – Make a CSS text-shadow for glows, outlines and 3D lettering, with a live preview · Cron Expression Generator – Generate a cron expression by choosing when the job should run.
Also works as: Robots.txt Tester.
In the collection SEO Tools.