Tools / AI robots.txt generator
🚦 Free and unlimited

robots.txt generator for AI crawlers

Allow or block each AI crawler knowing exactly what it does, then copy the block to add to your robots.txt.

GPTBot and OAI-SearchBot: two decisions, not one

GPTBot collects pages to train OpenAI models. Blocking it has no effect on your presence inside ChatGPT. OAI-SearchBot feeds the ChatGPT search index: it decides whether your page can be retrieved, summarised and cited with a link. Blocking it removes you from ChatGPT answers. Plenty of sites copied a GPTBot block and threw in OAI-SearchBot "to be safe", cutting themselves out of the exact traffic channel they were trying to win. Decide line by line, not in bulk.

Presets
Opens everything that can cite you, closes the collectors that never do.
GPTBot OpenAI Training

Collects pages to train OpenAI models.

Blocking it does not remove you from ChatGPT answers. It is a decision about your data, not about your visibility.

OAI-SearchBot OpenAI Search and citation

Indexes pages for ChatGPT search.

This is the crawler that governs whether you appear and get linked inside ChatGPT answers. Blocking it removes you from citations.

OAI-AdsBot OpenAI Advertising

Checks the landing pages used by ChatGPT Ads.

Required if you plan to buy ads inside ChatGPT: without it your landing pages cannot be validated.

ChatGPT-User OpenAI User-triggered fetch

Fetches a page live because a user shared the link or asked a question about it.

Traffic triggered by a real person in real time. Blocking it means refusing a visit someone explicitly asked for.

ClaudeBot Anthropic Training

Collects pages for the Claude models.

Anthropic also runs Claude-User for live fetches and Claude-SearchBot for search: those are separate agents and deserve separate decisions.

PerplexityBot Perplexity Search and citation

Indexes pages for the sourced answers Perplexity returns.

Perplexity always shows its sources. Blocking it means giving up those citations and the traffic attached to them.

Google-Extended Google Training

A token controlling whether your content feeds Gemini and answer grounding.

It is not a crawler: it changes neither your indexing nor your ranking in Google Search.

CCBot Common Crawl Training

Builds a public corpus reused by many training runs.

No direct visibility upside: Common Crawl cites nobody. It is the first candidate for a block if you are protecting your content.

Bytespider ByteDance Training

Collects pages for ByteDance models.

Known for an aggressive crawl rate. Many sites block it for server load reasons as much as for rights reasons.

meta-externalagent Meta Training

Collects pages for Meta models and Meta AI.

Meta runs several agents. This one is the documented crawler used for training.

Applebot-Extended Apple Training

A token controlling whether your content feeds Apple Intelligence.

It does not touch Applebot, which powers Siri and Spotlight: your presence in Apple search is unaffected.

One per line. Useful for a customer area, a cart or a staging zone.
Appended at the end of the file. Leave empty to skip it.
Free, unlimited, no account

This tool runs entirely in your browser. Nothing you type here reaches our servers, nothing is stored, there is no account to create and no quota to respect. We can give it away because it costs us nothing to run.

What this tool does

A miscalibrated robots.txt is the fastest way to disappear from AI answers without noticing. The most common and most expensive confusion involves OpenAI: GPTBot and OAI-SearchBot do not do the same job, and blocking one instead of the other has opposite consequences. This tool shows what each crawler actually does before letting you decide.

  • Eleven AI crawlers covered, each with its real job: training, search and citation, user-triggered fetch, advertising.
  • Four presets, including one that keeps citations while refusing training collection.
  • Adds disallowed paths that apply to the crawlers you let through, protecting a customer area or a cart.
  • Appends a Sitemap line, and comments recalling what each crawler does.
  • Built locally: the file is produced in your browser and nothing is transmitted.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is for model training. What governs your presence in ChatGPT answers is OAI-SearchBot, plus ChatGPT-User for live fetches. You can refuse training and still be perfectly citable.
Do crawlers really honour robots.txt?
The documented agents from OpenAI, Anthropic, Google, Apple and Perplexity say they honour it, and in practice they do. robots.txt remains a convention, not a technical barrier. If you truly need to prevent access, block at the application firewall or CDN level and verify against the published IP ranges.
Where does the file go?
At the root of the domain, and only there: https://example.com/robots.txt. A file in a subdirectory is never read. Every subdomain has its own robots.txt.
Can I block only part of the site?
Yes. The disallowed paths field adds Disallow lines to every crawler you leave allowed: they keep access to the site but not to those directories. That is the right approach for a customer area or a checkout funnel.
How do I check my rules are actually applied?
Look at your server logs: they are what tell you which crawlers really come through and on what. That is exactly what our AI crawler log analyser does, and it also catches accidental blocks from your firewall or CDN.

Beyond the generator

These micro-tools produce a file. The Ready2GEO scanners go and look at what is actually happening on your site and inside AI answers.

📣 ChatGPT Ads Readiness

Checks whether your site is genuinely eligible and usable by the ChatGPT ads platform: crawler access, structure, commercial signals.

Test my site 🕷️ AI crawler log analyser

Paste your server logs and see which AI crawlers really visit, how often, and which pages they choke on.

Analyse my logs 🧭 Full GEO audit

The bundled audit aggregates every scanner into a single report, with prioritised fixes and a PDF export.

Run the audit

All Ready2GEO tools

🔖Preferred Source button generator 💡Context hints generator 🧮ChatGPT Ads budget calculator 🔗UTM URL builder 📄llms.txt generator { }JSON-LD schema generator 🧰See the full suite