The two-line file that closes a staging site to crawlers
Runs in your browser. Nothing you type is sent anywhere. Updated on 24 Oct 2026.A staging copy that gets crawled will compete with the real site and can outrank it. The fix is two lines, and this page writes them. It also explains the trap: blocking a page stops it being read, not listed, so an address already in the index can linger as a bare link. For a page that must disappear, allow the crawl and serve a noindex tag.
The page opens with the Staging site preset: User-agent * with Disallow: /.
Edit the Comment at the top of the file if you want a different warning for whoever deploys it.
Press Copy, or download robots.txt.
Upload it to the root of the staging site only, never to the live site.
This robots.txt generator builds the file rule by rule: a group per crawler, allow and disallow paths, a crawl delay where one is honoured, and sitemap lines. Start from a preset, or paste a file you already have and read what every line actually does in plain words.
The presets cover the jobs people come for. Allow everything, which is also what having no file at all means. Block everything, for a staging copy that must never compete with the live site. A typical site that keeps carts, admin pages and internal search results out. And a file that welcomes search crawlers while asking the model-training crawlers to stay out, each named with the token its operator publishes.
One thing worth knowing before you block anything: robots.txt stops a page being read, not listed. An address already in the index can linger as a bare link. For a page that has to disappear, allow the crawl and serve a noindex tag. Related tools: Sitemap Generator and Meta Tag Generator.
The rest of the article, and all 6 questions about Robots.txt Generator, are on the Robots.txt Generator page.
In the root of the domain, at https://example.com/robots.txt, served as plain text. It applies to that host and port only, so a subdomain needs its own file. Crawlers do not look anywhere else, and a file in a subfolder is ignored.
No. It stops the page being read. A blocked address can still be listed as a bare link with no description, because the crawler never fetches the page and so never sees a noindex tag. To remove a page from results, allow the crawl and add a robots meta tag.
Two lines: User-agent: * then Disallow: /. That is the right file for a staging copy. On a live site it removes your traffic within days, so check which environment you are editing before you deploy it.
Only for crawlers that choose to read the file. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and PerplexityBot all publish tokens and state that they obey robots.txt. A scraper that ignores the file is unaffected, and robots.txt is not a security control.
Still stuck? Ask about Robots.txt Generator and anybody who knows can answer. No account needed.
On the related tools: CSS Text Shadow Generator – Make a CSS text-shadow for glows, outlines and 3D lettering, with a live preview · Cron Expression Generator – Generate a cron expression by choosing when the job should run.
In the collection SEO Tools.