Check a live robots.txt for errors and test whether a URL is blocked for Googlebot
Checked from our server in India. Only the address you type is sent to us; nothing else leaves your device. Checked against sources on 16 Oct 2026.A robots.txt checker fetches the file a website actually serves and tells you what crawlers make of it. This one reads the live file from our server, follows up to 5 redirects as crawlers do, flags syntax errors and serving problems, shows which search and AI crawlers may fetch the home page, and tests any URL for any crawler.
Type the domain and press Check robots.txt.
Read the fetch result, the problems and the crawler table.
Type a path and choose a crawler to see whether it is allowed and which rule decided.
Every crawler asks for /robots.txt before anything else, and what it does next depends on the answer as much as on the file. This checker requests the file from our server the way a crawler does: over https (plain http only if https does not answer), following up to five redirects, reading up to 512 KB.
Then it applies RFC 9309, the robots exclusion standard. A 404 or other 4xx answer means there is no file and everything may be crawled. A 5xx answer, or no answer at all, means crawlers must assume everything is disallowed; Google stops crawling the site for the first 12 hours and then falls back to its last good copy for up to 30 days. A file served as HTML is usually an error page returned with status 200, and is flagged.
The file is read line by line with the same parser as the Robots.txt Generator on this site: lines without a colon, rules before any User-agent, relative sitemap addresses, the retired Noindex and Host directives, and Disallow: / for everyone are all reported with their line number.
Type a path or a full URL and choose a crawler. The checker picks the group that names that crawler (case-insensitive, several groups combined), or the * group when none does, then applies the longest matching rule, with Allow winning a tie, and * and $ as wildcards. It shows Allowed or Blocked and the exact rule that decided. The table lists Googlebot, Bingbot and the main AI crawlers, with whether each may fetch the home page.
Robots.txt is a request, not a lock; it does not hide a page from people or from crawlers that ignore it. The file is read from India; a site that serves different files by region is rare but possible.
Type the domain and press Check robots.txt, then type the page's path or full address under Test a URL and choose the crawler, such as Googlebot. The checker applies the rules of the live file and answers Allowed or Blocked, naming the exact Allow or Disallow line that decided it.
Nothing bad. A 404 or any other 4xx answer tells crawlers there are no restrictions, so they may crawl every page. A robots.txt is only needed when something should be kept out, or to list the sitemap. The checker reports a missing file as information, not as an error.
A 5xx answer, or a server that does not answer, is treated by crawlers as disallow everything, under RFC 9309. Google stops crawling the whole site for the first 12 hours, then uses its last good copy for up to 30 days. The checker shows this as a failure, because it stops search engines from crawling.
The longer one, counted in characters of the pattern. If they are the same length, Allow wins. So Allow: /folder/page beats Disallow: /folder/, and Allow: /folder ties with Disallow: /folder and wins. The checker shows the rule that decided each test, so you can see why.
Yes. The crawler table lists GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and other AI crawlers next to Googlebot and Bingbot, each with Allowed or Blocked for the home page and the user-agent group that applied. To write new rules, use the Robots.txt Generator.
The Robots.txt Tester reads a file you paste, which suits drafting changes. This checker fetches the file a site actually serves, so it also catches serving problems the text cannot show: server errors, redirects, an HTML page in place of the file, or a file too large to be read in full.
Rates and rules change; check the source before you rely on a figure.
Still stuck? Ask about Robots.txt Checker and anybody who knows can answer. No account needed.
Also works as: Robots.txt Validator.
Same job, other tools: TXT Record Lookup – Read the TXT records of a domain: SPF, site verification codes and other text.
On the related tools: IP to ASN Lookup – Map an IP address to the ASN that announces it, with the BGP prefix.