AI crawlers: which bots to allow, and what each one does

The two jobs, and why the distinction matters

A training crawler collects text that may inform a future model. The effect on you is diffuse and delayed, and you get no link.

A retrieval crawler fetches a page because a user asked something now. If the assistant uses it, you typically get named and linked. That is the traffic and brand exposure people mean when they say they want AI visibility.

The practical consequence: opting out of training and opting out of citation are separate decisions, and a single blanket rule collapses them into one. Plenty of sites have quietly removed themselves from AI answers while intending only to protect their content from training.

The bots, by job

Two things worth knowing. Google-Extended does not affect ordinary Google Search indexing, so using it does not hurt classic rankings. And robots.txt is a voluntary convention — well-behaved crawlers honour it, and it is not a security control.

Three sensible policies

Pick the one that matches your actual position rather than copying someone else's file.

• Visibility-first — allow retrieval and training. Best when discovery matters more than content control, which is most businesses trying to become known.
• Cite me, don't train on me — allow OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot; disallow GPTBot, Google-Extended and CCBot. The common choice for publishers.
• Closed — disallow the lot. Coherent for paywalled or licensed content, and it does mean accepting invisibility in AI answers.

Our own file, as a worked example

Zylx runs the visibility-first policy, deliberately, and the reasoning is written into the file itself: we are establishing a category, so being retrieved and cited matters more to us than keeping marketing pages out of training corpora. Retrieval bots are named explicitly rather than left to a wildcard, so the policy is a decision rather than an accident. /api/ is never crawlable.

The reason to name bots explicitly even when allowing them: a future edit to the wildcard rule cannot silently change your AI-visibility posture. The named rules make the intent auditable.

You can read the live policy at zylx.ai/robots.txt. If your own file has a blanket disallow that predates the retrieval-versus-training split, it is worth a look — that split is recent enough that plenty of files were written before it existed.

Being allowed is necessary, not sufficient

Letting a crawler in gets you eligible. Whether an assistant then uses your page depends on whether the passage it needs is findable and quotable: a direct answer under the heading that asks the question, self-contained enough that pulling it out of the page does not strip its meaning.

Zylx assesses AI crawlability as part of its site crawl and scores pages for answer readiness, returning the specific blocks a page is missing. That covers eligibility and structure. It does not tell you whether a given assistant actually cited you — see the honest limits of measurement.

Frequently asked questions