Most conversations about AI crawlability focus on one file: llms.txt. But if you look at what Anthropic, Vercel, and LangGraph actually ship – and what GEO crawler research found AI agents fetching most – the file that matters more is its companion: llms-full.txt.
Here’s the practical reality: llms.txt is the map. llms-full.txt is the territory. And in 2026, the agents that matter for citation traffic are fetching the territory.
The Full File Family You Probably Don’t Know About
The original llms.txt proposal – published by Jeremy Howard in September 2024 – defined one file. Implementers built the rest. The complete family as of mid-2026 is four files, but most sites only need two:
| File | What’s in it | When to use |
|---|---|---|
/llms.txt | Curated index – H1, summary, link sections | Always. The orientation layer. |
/llms-full.txt | Full content of every linked page, concatenated as Markdown | When you want a model to deep-ingest your docs in a single fetch |
/llms-ctx.txt | Pre-expanded context without URLs | FastHTML-style implementations |
/llms-ctx-full.txt | Pre-expanded context with URLs preserved | Same, but URL-aware |
The pattern that works – and the one Anthropic, Vercel, and LangGraph all run – is the index + export pair: llms.txt for orientation, llms-full.txt for deep ingestion.
Why llms-full.txt Gets Crawled More
GEO researchers analyzing AI crawler behavior – including work cited by Profound – have noted that agents from Microsoft, OpenAI, and others tend to fetch llms-full.txt more frequently than llms.txt when both are present. The working explanation is structural: when a file contains the full content, it removes one retrieval step. An agent that fetches llms-full.txt gets everything it needs in a single HTTP request instead of fetching the index, parsing the links, then fetching each linked page individually. This is consistent with how developer documentation platforms like Mintlify describe the behavior of IDE agents operating under tight latency budgets.
For IDE agents (Cursor, Continue, Cline) and MCP integrations, this is even more pronounced. These tools are operating under tight context windows and latency budgets. A single fetch that returns a clean Markdown blob of your entire docs is structurally preferable to a multi-step crawl.
The implication: if you’ve shipped llms.txt but not llms-full.txt, you’ve done half the job.
How to Build llms-full.txt
The construction logic is simple: take every URL in your llms.txt, fetch each page, strip HTML to Markdown, and concatenate. In practice, most sites do this in their build pipeline.
Here’s the minimal Node.js pattern:
const fs = require('fs');
const fetch = require('node-fetch');
const TurndownService = require('turndown');
const turndown = new TurndownService();
async function buildLlmsFullTxt(llmsIndexPath, outputPath) {
const index = fs.readFileSync(llmsIndexPath, 'utf8');
const urlRegex = /\[.*?\]\((https?:\/\/[^\)]+)\)/g;
const urls = [...index.matchAll(urlRegex)].map(m => m[1]);
let output = '';
for (const url of urls) {
const res = await fetch(url);
const html = await res.text();
const markdown = turndown.turndown(html);
output += \n\n---\n# Source: \n\n;
}
fs.writeFileSync(outputPath, output);
console.log(Built llms-full.txt: pages, chars);
}
buildLlmsFullTxt('./public/llms.txt', './public/llms-full.txt');One constraint to manage: keep llms-full.txt under roughly 200,000 tokens (about 150K words, around 700KB). That’s the threshold where most models can ingest the file in a single context window. If your docs are larger, segment by product or language the way Supabase does – llms-full-api.txt, llms-full-guides.txt – and list the segmented files in your main llms.txt.
The 2026 robots.txt Stack That Completes the Picture
Shipping llms.txt and llms-full.txt is the visibility layer. The access-control layer is robots.txt – and it changed significantly in Q2 2026.
The key development: Anthropic split its crawler into two separate user-agents. ClaudeBot is the training scraper (high bandwidth, no citation value – block it). Claude-Web is the live-retrieval agent that fetches pages to answer Claude.ai user queries in real time (allow it, because it drives citation traffic). Brands that blanket-block “all Anthropic crawlers” lose Claude citations entirely.
Meta also shipped two active training scrapers in March 2026 – FacebookBot and Meta-ExternalAgent – at GPTBot-level crawl volume. Most sites have no rules for them yet.
Here’s the 2026 template:
# BLOCK: Training scrapers - high bandwidth, zero referral value
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
# OPT OUT: Google Gemini training (keeps Search indexing intact)
User-agent: Google-Extended
Disallow: /
# ALLOW: Live-retrieval agents - drive citation traffic
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /One important caveat on robots.txt enforcement: aggressive training scrapers often ignore the file or spoof their user-agents. The robots.txt rules signal intent and work for compliant bots; a WAF rule at the edge is the only deterministic block for non-compliant crawlers.
The Honest State of the Technology
The SERanking study of 300,000 domains (November 2025) found no measurable correlation between having llms.txt and being cited by ChatGPT, Claude, Gemini, or Perplexity. Google’s John Mueller compared the file to the deprecated keywords meta tag – something site owners declare but that search systems derive from the content itself.
None of that means you shouldn’t ship both files. The cost is low, the optionality is real, and the IDE-agent ecosystem (Cursor, Continue, Cline) does actively use llms.txt. But the robots.txt work is the lever that moves outcomes today. The llms.txt + llms-full.txt pair is infrastructure investment – you want to be correct when major LLM providers start honoring it, and building the build pipeline now costs far less than retrofitting it later.
The practical sequence for a site that hasn’t done this yet:
- Update robots.txt first. Add the Q2 2026 user-agent rules above. This takes twenty minutes and immediately affects how training scrapers treat your content.
- Ship llms.txt. Curated index, 20-50 priority pages, one-sentence description per link, sections in priority order.
- Build llms-full.txt. Concatenated Markdown of every linked page, under 200K tokens. Run it in your build pipeline so it stays current.
- Verify both files are served correctly.
curl -I https://yoursite.com/llms.txtshould return 200 withContent-Type: text/plain. A 404 on either file is the most common implementation error. - Add an access-log check. Once per month, grep your logs for requests to
/llms.txtand/llms-full.txtby user-agent. You want to see live-retrieval agents (Claude-Web, OAI-SearchBot, PerplexityBot) in the results – not just training scrapers.
The goal isn’t to optimize for a standard that isn’t fully adopted yet. It’s to build the infrastructure correctly now, while the field is still forming, so that adoption changes work in your favor rather than requiring catch-up.
Related Reading
- How AI Engines Actually Cite Your Content: Grounding and GEO Guide
- AEO Intent Classification: The Four-Query Framework for AI Engines
- How to Connect Any Tool to Claude with MCP: Complete Setup Guide
- Notion MCP Setup with Claude: Complete Config + Undocumented Errors
- How to Connect Claude Code to Postgres via MCP (The Right Way)
- The AI Operator’s Stack: Running a Multi-Brand Content Machine
- How I Made a $400 Laptop More AI-First Than a Copilot+ PC
Frequently Asked Questions
What is the difference between llms.txt and llms-full.txt?
llms.txt is a curated index — an H1, a summary, and link sections that orient an AI agent to your site. llms-full.txt is the full content of every linked page concatenated as Markdown, so an agent can deep-ingest your documentation in a single fetch. The index is the map; the full file is the territory.
Why do AI agents crawl llms-full.txt more often than llms.txt?
Fetching llms-full.txt removes a retrieval step: the agent gets everything in one HTTP request instead of fetching the index, parsing links, and fetching each page individually. For IDE agents like Cursor, Continue, and Cline operating under tight latency and context budgets, a single clean Markdown blob is structurally preferable to a multi-step crawl.
How big should llms-full.txt be?
Keep it under roughly 200,000 tokens (about 150K words, around 700KB) so most models can ingest it in a single context window. If your docs are larger, segment by product or language — for example llms-full-api.txt and llms-full-guides.txt — and list the segmented files in your main llms.txt.
Does having llms.txt actually improve AI citations?
Not measurably on its own. A November 2025 SERanking study of 300,000 domains found no correlation between having llms.txt and being cited by ChatGPT, Claude, Gemini, or Perplexity, and Google’s John Mueller compared it to the deprecated keywords meta tag. The lever that moves outcomes today is robots.txt configuration; llms.txt and llms-full.txt are low-cost infrastructure for when adoption grows.
Which AI crawlers should I allow in robots.txt in 2026?
Allow live-retrieval agents that drive citation traffic — Claude-Web, OAI-SearchBot, ChatGPT-User, anthropic-ai, and PerplexityBot. Block high-bandwidth training scrapers with no referral value such as GPTBot, CCBot, ClaudeBot, FacebookBot, and Meta-ExternalAgent, and opt out of Google-Extended to skip Gemini training while keeping Search indexing intact.
