AI Crawler Robots.txt Guide: Every Bot You Need to Allow (2026 Update)

Robots.txt is the first gate between your content and AI search engines. If the crawlers that feed ChatGPT Search, Perplexity, Google AI Overviews, and Claude cannot fetch your pages, no amount of structured data or factual density will earn you a citation.

AI search engines use dedicated crawlers separate from their training crawlers. In 2026, there are thirteen major AI crawlers actively scanning the web, up from just five in 2024. The key strategic shift: website owners are increasingly adopting a "block training, allow search" posture — explicitly blocking extractive training bots that return no traffic while permitting search citation bots that drive referral visits. This guide covers every AI crawler you need to know, the latest 2026 blocking data, crawl-to-refer ratios, and the exact robots.txt configuration to deploy.

The second panel widens the sample and turns the ratio into a decision rule. AI Crawler Check fetched the robots.txt of every domain in the Tranco top 10,000 on 5 October 2026 and parsed 4,962 valid files the way RFC 9309 says a crawler must: 19.2% block at least one training crawler while only 10.8% block at least one search crawler. The detail that matters for a GEO programme is the conditional one — only about half of GPTBot blockers keep OpenAI search open (347 of 710 still allow OAI-SearchBot) — which means the "block training, allow search" split is correct but far from automatic. It also records the accidents: 398 sites answered with an HTML page instead of robots.txt rules, and 3.6% still ship a blanket User-agent: * / Disallow: /.

Sources: BrandVector, "AI Crawlers List (2026): User Agents and Who Blocks Them" (817 top sites, 8 October 2026) · AI Crawler Check, "Who Blocks AI Crawlers? A Study of the Top 10,000 Sites' robots.txt" (4,962 valid files, fetched 5 October 2026). The two panels use different denominators — BrandVector reports share of 817 sites that name the token and disallow the whole site; AI Crawler Check reports share of 4,962 parsed files — so compare each column within itself, never across the row.

The mixed-use crawler collision: Googlebot, Applebot and BingBot

Cloudflare has stated plainly that a multi-purpose crawler is resolved by the most restrictive matching rule, and it names three of them explicitly: Googlebot, Applebot and BingBot. Each of these both indexes for search and feeds AI surfaces. So a site that ticks "block training" to keep model scrapers out can take its search crawler down with it — the strictest rule wins. This is not a theoretical edge case; it is the single most consequential side effect of the September policy and the reason this section exists.

Source: Cloudflare, "Accountable mixed-use crawlers" policy update (effective 15 September 2026); Scalevise analysis of the Search/Training/Agent split and Bot Preference Sync. Cloudflare states that Apple, Google and Microsoft have committed to honour the Disallow AI Training directive within a stated timeframe.

The failure mode is organisational before it is technical: an engineer changes one setting, and marketing discovers it weeks later when impressions have already fallen. Because the Agent category is off by default, an ad-carrying page behind the new default will not be reached by on-demand retrieval at all — and Search crawlers are unaffected, so the page still ranks while quietly losing the citations and clicks that come from live fetching. The diagnostic order that matters: check the Cloudflare dashboard category settings, then the robots.txt actually served, then server logs for the specific user-agents. Do not infer access from a green checkmark in a third-party tool.

One architectural signal is worth acting on regardless of whether you use Cloudflare. Cloudflare now recommends that providers use separate crawlers for separate purposes, and it has begun classifying content usage in three levels — immediate (real-time interaction, no storage), reference (indexing with links and excerpts), and full (summarisation and beyond) — publishable as a robots.txt preference. It has also signalled that bots which ignore declared usage terms may lose verified-bot status. The direction is unmistakable, and it is the same direction as the billing change: the fetch is no longer the unit of value, the use is. Operators running one crawler for training, indexing and agents are being pushed toward the same separation publishers are now being asked to configure.

The 13 AI crawlers you need to know in 2026

Each AI platform runs at least one crawler. Some run multiple: a training crawler that builds foundation models, a search crawler that powers real-time citations, and a user proxy that fetches pages on demand during conversations. The distinction matters — many sites block the training crawler and accidentally block the search crawler too. In Q2 2026, 87.51% of AI crawls are for training or mixed purposes, returning no direct value to publishers. Only 11.91% are for search or user action that can send referral traffic (Digital Applied, 2026).

Sources: OpenAI platform documentation (2026), Anthropic crawler docs (2026), Google Search Central (2026), Perplexity documentation (2026), Digital Applied AI Crawler Statistics (2026), TechnologyChecker robots.txt analysis (2026).

AI crawler traffic share shifts (Q2 2025 → Q2 2026)

The AI crawler landscape has fragmented significantly over the past year. Googlebot's dominance has been cut in half as new AI crawlers scale up. This shift has major implications for robots.txt strategy — the old approach of "just allow Googlebot" is no longer sufficient for GEO.

Source: TechnologyChecker robots.txt analysis across Cloudflare's network (2026), Digital Applied AI Crawler & Bot Traffic Statistics (2026).

Crawl-to-refer ratios: the metric driving blocking decisions

The crawl-to-refer ratio — how many pages a crawler fetches for each referral visit it sends back — has become the key metric for surgical blocking decisions in 2026. The most rigorous measurement to date comes from SEOmator's GEO Data Report 2026 (July 21, 2026), which cross-checked Cloudflare Radar's global network against its own panel of 500+ sites. The two independent datasets agreed closely — Anthropic measured 2,237:1 on Cloudflare versus 2,363:1 on the SEOmator panel — confirming the signal is real rather than a sampling artifact.

The single most important finding: extraction ratios are elastic, not fixed. Every major extractor improved dramatically the moment it shipped a consumer product that links out. Anthropic fell from 56,969:1 in January 2026 to 2,363:1 by July — a 24× improvement in seven months. OpenAI collapsed from 1,264:1 to 179:1 over the same period as ChatGPT Search matured, and OpenAI referrals crossed 1% of all referral traffic (1.05% globally) ahead of schedule. The practical implication for robots.txt policy: a block decision made in January 2026 is almost certainly wrong by August. Re-audit quarterly.

Source: SEOmator, "GEO Data Report 2026: Which AI Crawlers & LLM Bots Take the Most and Give the Least?" (July 21, 2026) — Cloudflare Radar rolling 28-day window ending July 21, 2026, cross-checked against SEOmator Agent Analytics across 500+ sites (Jan 1 – Jul 21, 2026).

The elasticity trend: why quarterly re-audits matter

Tracking the same operators month by month shows how fast this metric moves. Every ratio below is from the SEOmator panel, so the numbers are directly comparable across columns:

Source: SEOmator Agent Analytics panel, 500+ sites (July column covers Jul 1–21, 2026). Mistral's reversal reflects a large training crawl launched mid-2026 against a consumer surface that sends almost no clickable traffic back.

Two operators moved in opposite directions, and both moves are instructive. Anthropic and OpenAI improved by 24× and 7× respectively — not out of goodwill, but because they shipped cited-source surfaces that link out. Mistral went the other way, from a near-polite 22:1 in January to 6,021:1 in July, after launching a heavy training crawl behind a consumer product with no referral mechanism. Meta-ExternalAgent, now 6.35% of all crawler traffic, has never had a referral product at all.

"Extraction ratios are elastic, not fixed. The biggest extractors improved dramatically the moment they shipped consumer products that link out."

"Google AI Overviews does not use a separate crawler. It reuses the regular Googlebot index. So if your site is in Google's index, you are already eligible for AI Overviews citations — provided your content matches the query."

"More than half of all internet traffic is now bot-driven, and the fastest-growing slice is AI crawlers built to feed generative engines. Webmasters who treat robots.txt as a static afterthought are ceding citation visibility they will never get back."

"Authority decides whether you are in the candidate set; structure decides whether you are extracted. A clean robots.txt that lets the search crawlers in is the first line of structure — without it, no amount of E-E-A-T earns a citation."

The minimum-viable robots.txt for GEO in 2026

The block below implements the "block training, allow search" posture — the consensus recommendation for 2026. It explicitly allows every AI search and citation crawler while blocking training-only crawlers. Paste it at the top of your robots.txt, above any User-agent: * block.

# ── AI citation crawlers: explicitly allow ──
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

# ── Training-only crawlers: block ──
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# ── Highest extraction, no referral product (July 2026 data) ──
User-agent: MistralAI-User
Disallow: /

# ── Default rules ──
User-agent: *
Disallow: /private/
Disallow: /admin/
Allow: /

Sitemap: https://geoaura.world/sitemap.xml

The three-layer AI visibility system

In 2026, robots.txt is no longer the only file governing AI interactions. A three-layer system has emerged as the industry standard for managing AI access:

Robots.txt remains the access gatekeeper — if the crawler cannot reach your pages, nothing else matters. llms.txt provides a structured AI-readable map of your site's most important content. ai.txt declares your site's policy on AI training usage, content licensing, and citation permissions. Together, these three files form a complete AI visibility framework.

Why "block everything by default" backfires for GEO

A common SEO-school recommendation is to start robots.txt with User-agent: * / Disallow: / then selectively allow bots. This approach breaks GEO because most AI search crawlers fall back to the * rule when their specific block is missing or malformed. With AI crawler 403 forbidden rates at 8.56% (more than double the 3.63% rate in Q2 2025), accidental blocking is a growing and costly problem.

The TechnologyChecker 2026 analysis of Cloudflare's network found that the "block training, allow search" configuration is rapidly becoming the consensus standard. The key insight: AI crawlers fetch full page text paragraph-by-paragraph, looking for facts, claims, and quotable statements — they do not just index metadata and links like traditional search bots. A correctly configured robots.txt is the prerequisite for this extraction to happen at all.

Three rules for writing crawler blocks

1. 1.
   Place specific agents before the wildcard.

   Robots.txt parsers match top-down. If User-agent: * comes first with a Disallow, the AI bot may inherit that block before reaching its own entry.

2. 2.
   Each crawler needs its own block.

   You cannot list multiple crawlers under one User-agent directive. Each AI crawler requires its own User-agent line followed by Allow or Disallow.

3. 3.
   Validate before deploying.

   Run your robots.txt through Google Search Console's robots.txt Tester and a third-party validator. A single misplaced character can silently block an entire crawler. Meta robots tags are not consistently respected by AI crawlers — robots.txt is the primary reliable gatekeeper.

The GPTBot caveat: a 2026 update

A critical nuance emerged in 2026: GPTBot serves dual purposes. It crawls pages for OpenAI model training and for search indexing. Blocking GPTBot while allowing OAI-SearchBot may reduce SearchGPT visibility because OpenAI may route a portion of search queries through GPTBot's index. The economics have shifted decisively in OpenAI's favour: its crawl-to-refer ratio fell from 1,264:1 in January 2026 to 179:1 by July, and ChatGPT referrals crossed 1% of all referral traffic — a threshold analysts had not expected until late 2026. Cloudflare Radar put OpenAI at 392 pages crawled per referral in the seven days to 6 October 2026, with Anthropic at 473, Microsoft at 41 and Google at 4.9 — while Perplexity measured 3,700:1 over the same window. OpenAI is now one of the better-behaved operators that still returns meaningful traffic, which weakens the case for blocking it at all. The conservative recommendation for maximum GEO impact: test with GPTBot blocked and monitor your ChatGPT Search citation rate for 30 days. If it falls, restore access — the training-side benefit of blocking is worth less than search-side visibility.

Pay per crawl, per answer, and the 402 default

Pricing access is the third posture between allow and block, and 2026 moved it from experiment to documented product. Cloudflare's timeline is the cleanest record: pay per crawl opened in private beta on 1 July 2025 using HTTP 402 Payment Required; on 1 July 2026 the company shifted from per-crawl charging toward per-answer payment, naming Ceramic.ai and You.com as pay-per-query partners; on 2 October 2026 it shipped a Web Search API at $0.25, $5.00 and $7.00 per 1,000 requests with providers bound to verified-bot compliance and source attribution.

One interaction surprises almost everyone who tries it: a firewall block overrides the charge. Cloudflare states that if an AI crawler is blocked through WAF or Bot Management, those rules override pay per crawl — the blocked crawler gets no access and no invoice. A site that wants to monetise AI access and also runs a block-by-default rule will silently collect nothing. Before setting a price, read your own traffic composition: a 30-day edge count on one site (143,601 AI requests, 5 September – 4 October 2026) found 91.3% from Meta-ExternalAgent, a training crawler that returns no citation, against 5.2% from OpenAI's three agents and 0.4% from PerplexityBot. The volume that costs you bandwidth and the volume that sends you readers are almost entirely different traffic.

The 90-minute access audit (four layers, in order)

Most "we are blocked" and "we are definitely allowed" claims fail at a layer nobody checked. Run these four in sequence; each one can silently invalidate the layer above it.

1. 1.
   Serve-level: does robots.txt return rules at all?

   Fetch /robots.txt as a plain client. A 4xx lets crawlers access everything; a 5xx means they must assume complete disallow — a broken file is worse than no file. Check for an HTML challenge page returned with a 200: 398 of the Tranco top 10,000 did exactly this and parsed as no rules.

2. 2.
   File-level: does the search crawler have its own block?

   Confirm each search token is named explicitly and is not caught by the wildcard. Then confirm the reverse: that your training block did not also name a mixed-use crawler (Googlebot, Applebot, BingBot), because the most restrictive rule wins.

3. 3.
   Edge-level: does the WAF return 200 to that exact user-agent?

   Re-request the homepage with each crawler's user-agent string. Expect a 3.8% baseline miss rate for search crawlers and 13% for training crawlers even on sites whose robots.txt allows everything — higher on commerce sites. A 403 here is invisible in Search Console and invisible in robots.txt validators.

4. 4.
   Render-level: is the answer in the HTML?

   OAI-SearchBot does not execute JavaScript. Strip the scripts and read the page: if the passage a searcher would quote only exists after hydration, the first three layers pass and the citation still does not happen.

Only after all four pass is "the problem is content" a defensible conclusion. Before then it is a guess — and the retrieval-first evidence says a page that is never fetched cannot be fixed by anything written into it.

The silent killer: CDN bot-management rules

A correct robots.txt is necessary but not sufficient. Nico Digital's July 2026 audit flags a growing failure mode: CDN and WAF bot-management rules silently block AI crawlers by user-agent, excluding pages from OpenAI's index even when robots.txt explicitly allows OAI-SearchBot and GPTBot. The crawler passes the robots.txt gate, then gets a 403 at the edge — and your content never enters the retrieval pool.

October 2026 produced the first controlled measurement of how large that gap is. The Houston AI Visibility Index (Marketing & AI, homepages fetched 2 October 2026, WAF follow-up 3 October) took 400 sites that allow every AI crawler in robots.txt and re-fetched each homepage twice: once with an ordinary browser user-agent, once with each AI crawler's user-agent. 13% turned away at least one training crawler (GPTBot 9.8%, ClaudeBot 10%) and 3.8% turned away at least one search crawler (OAI-SearchBot 3.0%, PerplexityBot 3.3%, Claude-SearchBot 2.7%), almost always with a 403 from bot protection. Combined with robots.txt, roughly 5% of independent sites block AI search crawlers — not the 1.6% a robots.txt-only read would report. A separate 384-site convenience panel published on 3 October found bot protection challenging a declared crawler on 16% of homepages, rising to 27.3% of e-commerce sites. The honest limit: both studies sent user-agents from their own networks, not from the crawlers' verified IP ranges, so some firewalls that only block impostors will inflate the number. Server logs remain the only per-site confirmation.

The underrated AEO target: Bing Copilot + IndexNow

Because ChatGPT Search retrieves from the Bing index, Bing Copilot is the most underrated AI citation surface of 2026. Bing's lower SEO competition and native IndexNow integration — which pushes new and updated URLs to Bing within hours — make it the fastest way to get fresh content into the pool that powers both Bing Copilot and ChatGPT Search. Submitting via IndexNow after every update (as this site does on publish) shortens the citation lag from weeks to days.

Verifying crawler access

After deploying, verify that each AI search crawler can actually fetch your pages. Three checks cover 95% of issues:

• ▸ Server logs — grep your access logs for the user-agent strings above. If you see fetches, the bot reached you. Bytespider is known for aggressive crawl rates — monitor your log volume.
• ▸ robots.txt Tester — Google Search Console lets you test any user-agent against your live robots.txt. Test "OAI-SearchBot", "ChatGPT-User", "PerplexityBot", and "ClaudeBot" explicitly.
• ▸ Direct citation check — ask ChatGPT Search and Perplexity a question your site should answer. If neither cites you after 4 weeks of correct robots.txt, the problem is content quality, not access.

Frequently asked questions