Blocking AI Crawlers: What It Changes in AI Answers

Blocking AI crawlers changes one input to an AI answer. If a site disallows GPTBot or ClaudeBot in robots.txt, those compliant crawlers stop fetching its pages. Each blocked crawler loses access to the page. The assistant loses that source unless another crawl path reaches the URL. The effect on an answer depends on which assistant answers, which index or live fetch path it uses, and whether another copy of the same fact exists on a crawled page.

The mechanics diverge. Google-Extended controls how Google uses content for Gemini and Vertex AI grounding, while Googlebot still feeds Search. Microsoft’s Bingbot feeds the Copilot search layer. OpenAI’s GPTBot is one crawler; ChatGPT can also retrieve pages through user-triggered browsing and partner indexes. PerplexityBot crawls for Perplexity’s answer index.

Crawl access is a gate to the answer layer

The crawler-by-crawler breakdown above points to one conclusion. Google-Extended, Bingbot, GPTBot, and PerplexityBot each serve different pipelines, and each pipeline sits between a page fetch and an AI reply. Confusing those two layers is the most common error in discussions about blocking AI crawlers.

Layer one is access. A robots.txt rule tells a crawler whether it can fetch a URL. That rule is binary. Allow or disallow. The crawler obeys or ignores. When a site blocks GPTBot, it removes one path for OpenAI’s crawler to retrieve the page. That is all it does.

Layer two is the answer. When a buyer asks an assistant for a recommendation, the assistant assembles a shortlist from retrieval sources. Those sources include third-party indexes, partner data feeds, cached copies, and user-triggered browsing sessions. The business’s own pages are one input among many. Often they are not the input that decides the mention.

The same pattern shows up repeatedly in AI answers. Most recommendations trace back to retrieval layers rather than the brand’s own pages. The business gets named because a third-party source, an index, or a partner feed carried the relevant signal. That finding appears in the analysis of why AI assistants recommend the same businesses.

This distinction matters for blocking decisions. A robots.txt directive can change what a crawler fetches from a domain. It cannot change what a retrieval layer already holds. It cannot change what a partner index summarizes. It cannot change what a user-triggered browsing session sees when the buyer visits the site directly. Blocking a crawler removes one signal from one layer. The answer layer draws from many signals across many sources.

Within OpenAI, the separation is explicit. GPTBot and OAI-SearchBot are separate agents with separate roles and separate user agents. That separation is the next piece.

GPTBot and OAI-SearchBot are separate agents

OpenAI’s documentation lists GPTBot and OAI-SearchBot as separate user agents with independent configuration. GPTBot governs whether a site’s content is used to train OpenAI models. OAI-SearchBot governs whether a page is included in ChatGPT search results. A robots.txt rule that disallows GPTBot stops future training use. It does not stop OAI-SearchBot from fetching the page for ChatGPT search. The two decisions are separate because the two agents are separate.

The same split appears across vendors. Anthropic documents ClaudeBot for model training and Claude-SearchBot for search. Perplexity documents PerplexityBot for its search index. Google documents Google-Extended as a control for Gemini and Vertex AI training, while Googlebot feeds the regular Google index that supplies AI Overviews. In each case, the training crawler and the retrieval crawler are different user agents with different consequences.

This matters because an AI answer is assembled from an index at query time. Training data shapes model weights. Retrieval data shapes which pages the model can read and cite for a specific question. Blocking GPTBot removes a page from future training data. The page remains available to OAI-SearchBot, so ChatGPT search can still retrieve it and cite it. Blocking Google-Extended removes the page from Gemini and Vertex AI training. Googlebot still indexes the page, so AI Overviews can still use it. The citation half-life of a page depends on which of those two paths is closed.

The practical result is a split control panel. A site can block training while leaving search retrieval open. A site can block search retrieval while leaving training open. A site can block both. Each choice changes a different pipeline. The output of the retrieval pipeline is measurable by checking whether an assistant cites the page in answers. That measurement shows the effect of the choice.

The distinction becomes concrete in the Cloudflare default block that began September 15, 2026.

The Cloudflare default block that began September 15, 2026

Cloudflare sorts bots into Search, Agent, and Training. Search crawlers index pages for search results. Agent crawlers fetch pages to complete a user request, such as booking a table or comparing prices. Training crawlers collect content for model development. The taxonomy matters because each bot type receives different treatment under Cloudflare’s defaults.

On September 15, 2026, Cloudflare began blocking Agent and Training crawlers by default on ad-supported pages. The default applies to new domains and new sites on existing accounts. Untouched free-tier accounts also fall under it. A site owner can change the setting. The initial state, however, is blocked.

The same announcement moved Cloudflare’s publisher payment model from pay-per-crawl to pay-per-use. Under pay-per-crawl, a publisher received payment when a bot fetched a page. Under pay-per-use, payment is tied to content used in an answer. That shift aligns compensation with the output of an AI assistant rather than the crawl event. It also gives publishers a reason to allow Agent crawlers while still blocking Training crawlers, since one produces answer-time value and the other builds a model.

The consequence is an access decision at the edge layer. Cloudflare’s rule runs before the origin server and can override what robots.txt appears to allow. A site can have a permissive robots.txt and still block Agent crawlers for every visitor behind that Cloudflare configuration. The reverse also holds: a restrictive robots.txt does not guarantee a block if the edge layer is configured to allow the bot. Policy now lives in two places, and the edge layer wins for requests that pass through it.

For AI answers, this changes the pool of retrievable sources. If Agent crawlers are blocked on a domain, assistants that rely on live retrieval lose access to that content for user queries. A cached index or a prior crawl can keep the domain in the answer, and those citations decay at different rates. The AI citation half-life describes how long a source remains visible in assistant answers after the underlying crawl stops. A default block can accelerate that decay for every ad-supported site that never adjusts the setting.

The adoption data and the llms.txt dead end show whether publishers are changing the default, and whether a proposed standard has replaced the need for edge-level decisions.

Adoption data and the llms.txt dead end

The default is not uniform. Across top sites, roughly 30% block training bots while allowing search bots. About 9.9% block GPTBot. About 13.7% block at least one AI crawler. Those numbers describe a minority practice. No settled norm exists. A publisher that blocks GPTBot still leaves OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended to decide separately. Each bot carries a distinct user agent and stated purpose. Enforcement behavior differs.

llms.txt was proposed as a single file to declare permissions and preferences. The measured outcome is weaker. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended largely ignore llms.txt. Google states it ignores the file outright. A curated llms.txt can document intent for humans and for tools that choose to read it. It does not replace robots.txt rules or server-side controls. When an assistant resolves an entity, it retrieves structured third-party records. Those records include profiles and knowledge graph entries that exist outside the publisher’s own file structure. That retrieval step shapes which businesses enter the shortlist.

This split matters for diagnosis. Blocking a training crawler changes what a model can learn from a site. Allowing a search crawler changes what an assistant can retrieve at answer time. The two decisions use different controls and produce different effects. A site with a clean llms.txt and no server-level rules is still exposed to whichever crawler ignores the file. A site with strict bot rules and no third-party entity record remains absent from the retrieval layer.

This is measurable by inspecting assistant answers for mentions and citations, then recording position. The adoption data show how publishers set edge rules. The llms.txt data show that a proposed standard has not replaced those rules. The next question is cost. Two recent analyses disagree on what blocking costs.

Two analyses disagree on what blocking costs

One published analysis of AI crawler blocking and citation outcomes found that sites that blocked AI crawlers did not see fewer citations in AI answers. That result sounds like blocking is free. A separate analysis reached a different conclusion: blanket blocks behave close to a delisting. The disagreement is real, and both results follow from different measurement targets.

Name-level presence and page-level citation depend on different retrieval paths. An assistant can name a business because third-party pages carry that name, including directories and review sites. Prior model associations also carry names. If the model learned the business during training, a block on the business’s own domain does not erase that association. This is why blocking can leave mentions intact.

Page-level citation requires live fetch access. When an assistant retrieves a page to quote or summarize, a blanket block stops the fetch. The answer then cites another source that allowed access. That mechanism explains the delisting-like result. The cost shows up in citations to the blocked domain. The business name can survive. Citation half-life matters here: a page cited today can lose citation share as retrieval shifts to accessible sources.

Neither analysis establishes causation. Domains that block AI crawlers differ from domains that do not. They differ by size and content type. Legal posture and server stack vary. Existing search visibility varies. The samples do not control for those differences. A correlation between blocking and citation counts can reflect the kind of site that blocks. A correlation between blocking and stable mentions can reflect strong third-party coverage. Mentions and citations are measured separately for this reason. The separation prevents one metric from hiding the other.

The next section covers where a block bites: freshness, local prompts, live retrieval.

Where a block bites: freshness, local prompts, live retrieval

Search-backed assistants do not answer from a static snapshot. They read the current search index and the current text of pages they can fetch. When a robots.txt rule or firewall blocks an AI crawler, the assistant can still see the business name in a directory, a review site, or a news article. It cannot read the business’s own page. That page cannot be quoted or used to resolve a specific question. The business keeps getting named in the abstract. The details that win the shortlist stay outside the answer.

Recency raises the cost. aeod.app found that about 72% of cited pages were updated within the past year, while 42% of pages were published in that window. The gap matters. Assistants favor pages that reflect current hours, current service areas, current pricing signals, and current proof. A blocked page can be fresh on the server and invisible to retrieval. The assistant falls back to older third-party text. That text can be accurate enough to name the business and stale enough to misstate what it does.

Location and service-area questions are the most exposed. A question like “who handles emergency plumbing in Tempe” depends on local evidence: city pages, service-area pages, address markup, and recent reviews tied to that place. If the crawler cannot fetch those pages, the assistant has to infer location from other sources. Those sources often lack the service boundary or the neighborhood detail. The result is a name without the qualifying context. See ai visibility by location for how location signals interact with assistant answers.

Follow-up turns add another retrieval pass. A buyer asks for a provider, then adds a constraint: “open on Sunday” or “under $200.” Each follow-up triggers another retrieval pass. If the page that answers that constraint is blocked, the assistant substitutes a source it can read. The substitution can change the shortlist.

The practical takeaway: separate training blocks from retrieval blocks, check the edge layer, and measure citations rather than assuming a block is free. The decision is per-crawler, per-pipeline, and per-page.