# Robots.txt Generator

> Build a robots.txt file with disallow paths, sitemap links, crawl-delay, per-bot rules and a one-click block for AI training crawlers like GPTBot and ClaudeBot.

Use it free at https://swipetolaunch.com/tools/robots-txt-generator

## How to use it

1. **Pick a default.** Allow all for a normal site. Block all only for staging or private sites.
2. **List paths to keep out.** Admin areas, internal search, cart and API routes, one per line.
3. **Add sitemaps and bot rules.** Point crawlers at your sitemap, add rules for one bot, or block AI training crawlers.
4. **Download and upload.** Save robots.txt to the root of your domain so it loads at /robots.txt.

## What robots.txt does, and what it does not

robots.txt is a plain-text file at the root of your site that tells crawlers which URLs they may fetch. Well-behaved bots (Googlebot, Bingbot, most AI crawlers from large companies) read it before crawling. It is a request, not a lock: bad bots ignore it, and anyone can read the file.

It also does not remove pages from search. A blocked URL can still appear in Google if other sites link to it, just without a description. To keep a page out of results, let it be crawled and add a `noindex` robots meta tag with the [meta tag generator](/tools/meta-tag-generator).

## How the rules work

- **User-agent** starts a group. `*` means every crawler; a name like `Bingbot` targets one.
- **Disallow** blocks a path prefix. `/admin/` blocks everything under it. An empty `Disallow:` allows everything.
- **Allow** makes an exception inside a blocked folder. The longest matching rule wins.
- **`*` and `$`** are wildcards: `/*.pdf$` blocks every URL ending in .pdf.
- **Crawl-delay** asks for seconds between requests. Bing and Yandex use it; Google ignores it.
- **Sitemap** lines can go anywhere and point crawlers at your XML sitemap.

A crawler follows only the most specific group that names it. If you add a group for Bingbot, Bingbot ignores the `*` group entirely, so repeat any shared rules there.

## Blocking AI training crawlers

The AI toggle adds a group that disallows the main training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), CCBot (Common Crawl), PerplexityBot, Bytespider (ByteDance), Applebot-Extended and meta-externalagent (Meta). Google-Extended and Applebot-Extended are control tokens, so blocking them does not affect Google Search or Siri. Weigh it before you switch it on: blocking AI crawlers can also keep your product out of AI answers. If you want to be cited, an [llms.txt file](/tools/llms-txt-generator) helps instead.

## Common mistakes

- Blocking CSS and JS folders, which stops Google rendering the page.
- Leaving `Disallow: /` from a staging site on the live domain.
- Using robots.txt to hide private content. Use authentication instead.

Test any URL against your finished file with the [robots.txt tester](/tools/robots-txt-tester).

## What you get

- Allow-all or block-all default
- Disallow and allow path lists with wildcard support
- Per-bot groups with their own crawl-delay
- One toggle to block eight AI training crawlers
- Download as robots.txt

## Questions

### How do I block GPTBot and other AI crawlers?

Add a group with User-agent: GPTBot and Disallow: / for each crawler. This tool's AI toggle writes that group for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended and meta-externalagent.

### Where do I put robots.txt?

At the root of the host, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file. Crawlers will not look for it in a subfolder.

### Does robots.txt stop a page from being indexed?

No. It stops crawling, not indexing. A blocked page can still show in search if it is linked from elsewhere. Use a noindex meta tag on a crawlable page to keep it out.

### Does Google respect crawl-delay?

No. Googlebot ignores Crawl-delay and adjusts its own rate based on how your server responds. Bing and Yandex do honor it.

### Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether your content is used to train and ground Gemini models. Googlebot and Search are unaffected.