SwipeToLaunch
Add your product

Robots.txt tester

Test a path against robots.txt for any crawler, and see which AI bots are blocked.

How to use it

  1. Enter a site. We fetch its /robots.txt.
  2. Pick a crawler and path. Googlebot, Bingbot, an AI crawler, any crawler (*) or your own token, and the path you want to test.
  3. See the verdict. Allowed or blocked, plus the exact line that decided it. Change the path or crawler to re-test instantly.

How crawlers read robots.txt

Robots.txt is a plain-text file at the root of a host that tells crawlers which paths they may fetch. This tester applies the same rules Google documents, so the verdict matches what Googlebot will do:

  • One group per crawler. A crawler obeys the group whose User-agent matches its name. If none matches, it uses the * group. It never combines its own group with *.
  • The longest matching rule wins. Allow: /blog/public beats Disallow: /blog for /blog/public/post because it is more specific.
  • Allow wins a tie. When an allow and a disallow rule are the same length, the page may be crawled.
  • Wildcards. * matches any run of characters and $ anchors the end, so Disallow: /*.pdf$ blocks every PDF.
  • Case matters in paths: /Admin and /admin are different.

Blocking is not the same as noindex

Robots.txt stops crawling, not indexing. A blocked URL can still appear in search results if other sites link to it, just without a description. To keep a page out of Google, let it be crawled and add a noindex tag or X-Robots-Tag header. Check those with the HTTP header checker.

AI crawlers

AI companies publish separate user-agents for training and for live answers. Blocking GPTBot stops OpenAI training on your pages; blocking OAI-SearchBot would also keep you out of ChatGPT search. Google-Extended controls Gemini training without affecting Google Search. Decide per crawler: being cited in AI answers can send real traffic.

Common mistakes

  • Disallow: / left over from a staging site.
  • Blocking CSS and JavaScript, which stops Google rendering the page.
  • A robots.txt that returns a 5xx error: Google treats that as "block everything".

Write a clean file with the robots.txt generator, and point crawlers at your sitemap with a Sitemap: line.

What you get

  • Google's longest-match and Allow-wins-ties rules
  • * and $ wildcard support
  • Googlebot, Bingbot, AI crawlers or a custom token
  • Shows the deciding line
  • AI crawler block report for 13 bots

Questions

How do I check if a page is blocked by robots.txt?

Enter the site, choose the crawler (Googlebot for Google Search), type the path and run the test. The verdict shows whether it is allowed and which rule decided it.

Does Disallow in robots.txt remove a page from Google?

No. It stops Google crawling the page, but the URL can still be indexed from links. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header.

How do I block GPTBot and other AI crawlers?

Add a group per crawler, for example `User-agent: GPTBot` followed by `Disallow: /`. Each AI company documents its own tokens, and training and search crawlers are often separate.

What happens if robots.txt is missing?

If it returns 404, crawlers assume everything is allowed. If it returns a server error (5xx), Google pauses crawling the whole site until the file answers again.

Does Google support Crawl-delay?

No. Google ignores Crawl-delay. Bing and some other crawlers respect it.