SwipeToLaunch
Add your product

Robots.txt generator

Disallow rules, sitemaps, per-bot rules and an AI crawler block, ready to download.

How to use it

  1. Pick a default. Allow all for a normal site. Block all only for staging or private sites.
  2. List paths to keep out. Admin areas, internal search, cart and API routes, one per line.
  3. Add sitemaps and bot rules. Point crawlers at your sitemap, add rules for one bot, or block AI training crawlers.
  4. Download and upload. Save robots.txt to the root of your domain so it loads at /robots.txt.

What robots.txt does, and what it does not

robots.txt is a plain-text file at the root of your site that tells crawlers which URLs they may fetch. Well-behaved bots (Googlebot, Bingbot, most AI crawlers from large companies) read it before crawling. It is a request, not a lock: bad bots ignore it, and anyone can read the file.

It also does not remove pages from search. A blocked URL can still appear in Google if other sites link to it, just without a description. To keep a page out of results, let it be crawled and add a noindex robots meta tag with the meta tag generator.

How the rules work

  • User-agent starts a group. * means every crawler; a name like Bingbot targets one.
  • Disallow blocks a path prefix. /admin/ blocks everything under it. An empty Disallow: allows everything.
  • Allow makes an exception inside a blocked folder. The longest matching rule wins.
  • *` and $** are wildcards: /*.pdf$` blocks every URL ending in .pdf.
  • Crawl-delay asks for seconds between requests. Bing and Yandex use it; Google ignores it.
  • Sitemap lines can go anywhere and point crawlers at your XML sitemap.

A crawler follows only the most specific group that names it. If you add a group for Bingbot, Bingbot ignores the * group entirely, so repeat any shared rules there.

Blocking AI training crawlers

The AI toggle adds a group that disallows the main training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), CCBot (Common Crawl), PerplexityBot, Bytespider (ByteDance), Applebot-Extended and meta-externalagent (Meta). Google-Extended and Applebot-Extended are control tokens, so blocking them does not affect Google Search or Siri. Weigh it before you switch it on: blocking AI crawlers can also keep your product out of AI answers. If you want to be cited, an llms.txt file helps instead.

Common mistakes

  • Blocking CSS and JS folders, which stops Google rendering the page.
  • Leaving Disallow: / from a staging site on the live domain.
  • Using robots.txt to hide private content. Use authentication instead.

Test any URL against your finished file with the robots.txt tester.

What you get

  • Allow-all or block-all default
  • Disallow and allow path lists with wildcard support
  • Per-bot groups with their own crawl-delay
  • One toggle to block eight AI training crawlers
  • Download as robots.txt

Questions

How do I block GPTBot and other AI crawlers?

Add a group with User-agent: GPTBot and Disallow: / for each crawler. This tool's AI toggle writes that group for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended and meta-externalagent.

Where do I put robots.txt?

At the root of the host, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file. Crawlers will not look for it in a subfolder.

Does robots.txt stop a page from being indexed?

No. It stops crawling, not indexing. A blocked page can still show in search if it is linked from elsewhere. Use a noindex meta tag on a crawlable page to keep it out.

Does Google respect crawl-delay?

No. Googlebot ignores Crawl-delay and adjusts its own rate based on how your server responds. Bing and Yandex do honor it.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether your content is used to train and ground Gemini models. Googlebot and Search are unaffected.