llms.txt and AI crawlers are not the same thing, and only one of them decides whether your site gets cited. llms.txt is a small Markdown file that points AI tools to your most useful pages. It does not get your site cited in ChatGPT, Claude, Perplexity, or Google AI Overviews. What decides citations is crawler access: if the search bot for that AI service cannot fetch your page, it cannot cite it. So in llms.txt and AI crawlers, allow the search crawlers, block only the training crawlers you do not want, and treat llms.txt as cheap housekeeping, not as a visibility lever.
llms.txt and AI crawlers: what llms.txt actually is
For llms.txt and AI crawlers, the first thing to know is that llms.txt is a proposal, not a standard. Jeremy Howard of Answer.AI published it on 3 September 2024 at the llms.txt proposal, and it was revised to version 2 on 10 August 2026. No standards body has adopted it.
In llms.txt and AI crawlers, the file sits at yoursite.com/llms.txt. It is written in Markdown, and it gives a language model a short summary of a site and a curated list of links to its most useful pages. The format has only one required part: an H1 with the name of the project or site. After that come an optional short summary in a blockquote, optional notes, and H2 sections, each holding a list of links with a one-line note on each page.
It was built for a specific job in llms.txt and AI crawlers: helping an agent use a website while it works. The proposal itself says llms.txt files are used most heavily for software documentation, where a coding agent follows them to find API references and tutorials. That is a real use case. A developer who points a coding tool at your docs can use the file. A search engine deciding which site to cite is a different job, and the file was not built for that.
The AI labs publish llms.txt files for their own developer documentation, including OpenAI, Anthropic, and Google Gemini. That shows they offer the file to coding tools. It does not prove their search products read your file. The proposal page also lists tools that generate the file automatically, including Mintlify, GitBook, Yoast SEO, AIOSEO, and Wix.
llms.txt and AI crawlers: what the evidence says
The honest answer on llms.txt and AI crawlers is that llms.txt has no measured effect on AI citations, and the largest measurement so far points the other way.
Ahrefs published a server-log study on 15 June 2026, by Louise Linehan, called “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”. It looked at 137,210 domains in Ahrefs Web Analytics that had traffic in May 2026. About 28% of those domains published an llms.txt file, and 97% of those files received zero requests in May 2026. Nothing fetched them at all. The crawlers that fetch pages for AI search answers made only 233 of the requests the study counted, about 1.1% of the requests that arrived.
Ahrefs is clear about the limits: fetching a file does not prove a model read it or used it, so every figure in the study is a ceiling on actual use, not a measure of it. Its customer base also skews technical and SEO-aware, so the 28% adoption figure is an upper bound, not a picture of the average site.
A second study, by SE Ranking, looked at nearly 300,000 domains and found no detectable link between having an llms.txt file and being cited in AI answers. That study is reported by other publishers, and the finding matches the Ahrefs logs: the file is rarely fetched, and fetching it is not the same as citing from it.
On llms.txt and AI crawlers, Google says the same thing in plain words. In its guide to optimizing for generative AI features, under “Mythbusting generative AI search”, Google writes: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” The guide adds that creating an llms.txt file “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
Google’s John Mueller said in June 2025 that no AI system currently uses llms.txt, and Google’s Gary Illyes said in July 2025 that Google does not support llms.txt and is not planning to. As of 13 November 2026, no major AI search company has said publicly that it uses llms.txt to choose which sources to cite. Perplexity is the engine practitioners most often report reading the file, but there is no official confirmation from Perplexity, and the Ahrefs logs show its search bot barely requested the file.
Why Google AI Overviews do not need llms.txt
For llms.txt and AI crawlers on Google, the gate to AI Overviews and AI Mode is ordinary Search eligibility, not any AI-specific file. A page can appear in AI Overviews only if it is indexed and eligible to be shown in Google Search with a snippet. That is the whole gate, and it is why the same content that ranks well is the content most likely to be cited.
Blocking Google-Extended does not remove you from AI Overviews. Google-Extended is the token for training Google’s AI models. AI Overviews draw on the normal Google index through Googlebot, not through Google-Extended, so opting out of training does not opt you out of AI answers. The reverse is also true: allowing Google-Extended does not get you into AI Overviews.
If your content does not appear in Google’s AI answers, the fix is usually the content itself, not a new file. For more on that, see our guide on AI Overviews and our guide to AEO vs GEO vs SEO.
llms.txt and AI crawlers: the crawlers that actually decide citations
In llms.txt and AI crawlers, there is not one “AI crawler”. Each company runs separate bots for training, for search, and for fetching a page when a user asks about it. A robots.txt rule aimed at one of them has no effect on the others. That split is the most common mistake in this whole topic: a site owner blocks “AI bots” with one rule and accidentally takes the site out of ChatGPT search, when they only meant to opt out of training.
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
OpenAI documents three agents in its crawler documentation, and each setting is independent of the others.
- GPTBot crawls content that may be used to train OpenAI’s foundation models. Disallowing GPTBot tells OpenAI that the content should not be used for training. Allowing GPTBot does not get you cited in ChatGPT, and blocking it does not remove you from ChatGPT search.
- OAI-SearchBot is the crawler for ChatGPT search. It surfaces websites in ChatGPT’s search results. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.
- ChatGPT-User fetches a page when a user asks ChatGPT or a Custom GPT a question. It is not used for automatic crawling, and it is not used to decide whether content may appear in Search. Because the fetch is started by a user, OpenAI says robots.txt rules may not apply to it.
Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic documents three bots on its crawler page, updated 4 November 2026.
- ClaudeBot collects web content that could contribute to training Anthropic’s models. Restricting it signals that the site’s future content should be excluded from training datasets.
- Claude-SearchBot navigates the web to improve search result quality for Claude users. Disabling it prevents Anthropic from indexing your content for search, which may reduce your visibility in Claude search results.
- Claude-User fetches a page when a person asks Claude a question. Anthropic says site owners can control it with robots.txt, and disabling it may reduce visibility for user-directed web search.
Anthropic also said on 30 October 2026 that it has begun recognizing a machine-readable signal that lets a site say “no AI training”, based on a draft from the IETF’s aipref working group. That signal is separate from robots.txt, and Anthropic says ClaudeBot rules named directly still take priority over it. This is still rolling out, so do not rely on it alone yet.
Perplexity: PerplexityBot and Perplexity-User
Perplexity documents two agents in its crawler documentation, updated 12 November 2026.
- PerplexityBot surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. To appear in Perplexity search results, Perplexity recommends allowing PerplexityBot in robots.txt.
- Perplexity-User fetches a page when a user asks Perplexity a question, so it can answer and link to the page. It is not used for web crawling or for training. Because a user requested the fetch, Perplexity says this fetcher generally ignores robots.txt rules.
Perplexity was also the subject of a public dispute in August 2025, when Cloudflare reported undeclared crawlers tied to Perplexity reaching content that had blocked its declared bot. Perplexity denied it. That dispute is a reminder that robots.txt is a request, not a lock: a bot that ignores it can only be stopped at the firewall, by IP range.
Google: Googlebot and Google-Extended
- Googlebot crawls for the Search index, which feeds AI Overviews and AI Mode. Do not block it.
- Google-Extended controls whether Google may use your content to train its AI models. Blocking it opts you out of training only. It does not remove you from Search, AI Overviews, or AI Mode.
Other training crawlers exist, such as Common Crawl’s CCBot and ByteDance’s Bytespider, but the four operators above decide most citations. If you block a training crawler, block it by its exact token, not with a blanket rule that also catches the search bots.
llms.txt and AI crawlers: how to control them in robots.txt
In llms.txt and AI crawlers, robots.txt is the file that actually controls AI access to your site. It lives at yoursite.com/robots.txt, and it is checked before any other page. A single wrong line there can take you out of an AI answer, so check it before you add anything else.
I checked ClickRank.ai on 13 November 2026. Our robots.txt blocks MJ12bot, AhrefsBot, and SemrushBot, and it blocks /wp-admin/ and /tag/ for all bots. It has no rule at all for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, or Google-Extended. That means every AI crawler is allowed on ClickRank.ai by default, for search and for training. That was a choice by omission, not a decision, and it is the setup most sites are already on without knowing it.
Setup 1: Allow AI search, block AI training (recommended for most sites)
This is the setup most sites should use. It keeps you citable in ChatGPT, Claude, and Perplexity, while telling the training crawlers not to use your content for training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Do not add a User-agent: * block with Disallow: / above these lines. A blanket block catches the search bots too, and that is how most sites accidentally disappear from AI answers.
Setup 2: Allow everything
If you do not mind your content being used for training, you do not need to add anything. AI crawlers are allowed by default when robots.txt has no rule about them. This is the simplest setup, and it is the right one for most documentation sites, where being read by coding agents is the point.
Setup 3: Block everything, and know what you lose
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
If you use this setup, your site will not be shown in ChatGPT search answers, and it will not be indexed for Claude search or surfaced in Perplexity results. Do not use it unless you have decided that staying out of AI answers is worth that loss. And remember the user-triggered fetchers: ChatGPT-User and Perplexity-User generally ignore robots.txt because a user asked for the page, so a robots.txt block does not guarantee that a page can never be fetched on a user’s request. A firewall rule on the published IP ranges is the only hard control for those.
llms.txt and AI crawlers: how to add an llms.txt file anyway, in 10 minutes
If you still want the file, in llms.txt and AI crawlers it costs 10 minutes and it does no harm. It is most useful for a documentation site, where coding agents read it, and least useful for a blog that wants AI citations, where it does almost nothing.
- Create a file called llms.txt at the root of your site, at yoursite.com/llms.txt.
- Start with one line: an H1 with your site name. That is the only required part.
- Add a short blockquote summary: what the site is, in one or two sentences.
- Add H2 sections with a short list of links to your most useful pages, each with a one-line note. Link to 10 to 20 pages at most, not every page you have. A file that lists everything is no better than a sitemap.
- Test it: ask a coding agent a question about your content, giving it only your llms.txt as a starting point, and see whether it finds the right page.
Here is the shape, from the proposal itself:
# ClickRank
> ClickRank is an SEO and AI search blog that compares AI tools and explains how Google and AI search work.
## Best guides
– [AEO vs GEO vs SEO](https://www.clickrank.ai/aeo-vs-geo-vs-seo/): What AEO and GEO mean, and how they differ from SEO
– [Why content does not appear in Google AI Overviews](https://www.clickrank.ai/content-doesnt-appear-in-google-ai-overviews/): The checks to run when a page is missing from AI answers
## Optional
– [E-E-A-T](https://www.clickrank.ai/e-e-a-t-a-google-ranking-factor/): Experience, expertise, authoritativeness, and trust
If you run WordPress, you do not need to write this by hand. Yoast SEO and AIOSEO both generate and maintain an llms.txt file, and Wix generates one for every Wix site. Check the generated file before you trust it: it should list your best pages, not your login page and your tag archives. I opened ClickRank.ai/llms.txt on 13 November 2026. Rank Math generates it for us, and it lists post after post, hundreds of lines long, not a short curated list of best pages. I would not leave a file like that as a guide for an agent. Open the generated file and cut it to your best pages.
llms.txt and AI crawlers: how to check your own site in 10 minutes
- Open yoursite.com/robots.txt in a browser. Search it for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and Google-Extended. If any search bot is disallowed, that is your first fix. I did this check on ClickRank.ai on 13 November 2026. It took one minute, and it found no AI bot rule at all.
- Open your server logs, or your Cloudflare or hosting analytics, and search the user-agent field for OAI-SearchBot, PerplexityBot, Claude-SearchBot, and GPTBot. A search bot that never appears has either never found your site or is being blocked somewhere, including by a firewall rule that returns a 403.
- Open Google Search Console and check the Generative AI performance report. It shows impressions only, not clicks, and it covers AI Overviews and AI Mode on Google Search. If a page has no AI impressions, check first whether it is indexed and eligible for a snippet, because that is the gate.
- Ask ChatGPT and Perplexity a question your best page answers, and see whether your site is cited. If it is not, do not add an llms.txt file and wait. Check the robots.txt step first, then check whether the page itself is the kind of page an AI answer would cite: original, specific, and backed by E-E-A-T.
llms.txt and AI crawlers: what not to do
- Do not block all AI bots with one User-agent: * rule. That blocks the search bots that cite you, not just the training bots.
- Do not pay a service that promises llms.txt and AI crawlers will get you cited in ChatGPT or Perplexity. No company has said the file does that, and the largest log study found the file is almost never fetched.
- Do not expect llms.txt to fix thin content. A file that points to a weak page still points to a weak page. The content has to be worth citing first.
- Do not block Googlebot to stop AI training. Googlebot is the Search index. Blocking it removes you from Search and from AI Overviews. Use Google-Extended for training instead.
- Do not treat a fetch as a citation. A bot downloading your llms.txt file, or your page, does not mean it used a word of it in an answer.
You have just read how llms.txt and AI crawlers work. Now check your own site the same way. ClickRank reads your Google Search Console data and shows you what to fix in on-page SEO, from titles and meta descriptions to headings and internal links. Try ClickRank and run your first on-page SEO check today.
Do llms.txt and AI crawlers help with AI search rankings?
No, not as far as any public evidence shows. Google says its Search ignores the file, and the Ahrefs log study found 97% of published files received zero requests in May 2026. It will neither harm nor help your visibility in Google Search, in Google own words.
How do llms.txt and AI crawlers work together?
In llms.txt and AI crawlers, they do different jobs. llms.txt is a map: it tells an agent which pages are worth reading. AI crawlers are the visitors: they fetch your pages, and the search crawlers among them decide what can be cited. A map does not help if the visitor is blocked at the door, and a visitor that is allowed in does not need the map to find a good page.
Should I block GPTBot?
In llms.txt and AI crawlers, block GPTBot only if you do not want your content used to train OpenAI models. Blocking GPTBot does not remove you from ChatGPT search, because that is OAI-SearchBot job, and the two settings are independent. If you block GPTBot, do it by its exact token, and leave OAI-SearchBot allowed.
Should I block ClaudeBot?
The same rule applies. ClaudeBot is Anthropic training crawler. Blocking it signals that your content should not be used for training. Claude-SearchBot is the search crawler, so leave that one allowed if you want to be indexed for Claude search.
Will Perplexity cite my site if I allow PerplexityBot?
For llms.txt and AI crawlers, allowing PerplexityBot is a requirement, not a guarantee. Perplexity recommends allowing it so your site can appear in its search results, and it says the bot is not used for training. Whether your page is actually cited depends on whether it answers the question well and whether Perplexity system picks it.
Are llms.txt and AI crawlers worth adding at all?
For a documentation site, yes, it is 10 minutes of work and coding agents do read it. For a blog that wants AI citations, it is the last thing to add, not the first. Fix robots.txt, fix the content, and check that the page is indexed first. Then add the file if you want it, and do not expect it to move your citations.
