If you publish an llms.txt file this week, no major model is going to fetch it tonight. That is the honest 2026 read on the spec — and yet the file is still worth shipping for narrow, specific reasons. This guide covers the 4-element specification published at llmstxt.org, the robots.txt pairing that actually controls AI crawler behavior right now, and a server-log filter you can run to verify whether anyone is reading the file you just shipped.
What llms.txt actually is (and what it isn’t)
llms.txt is a Markdown file served at the site root — /llms.txt — proposed by Jeremy Howard of Answer.AI on September 3, 2024. The spec at llmstxt.org defines four elements: a required H1 with the project or site name; a blockquote summary; zero or more Markdown content sections (no headings); and zero or more H2-delimited file-list sections containing annotated Markdown links to deeper content. That is the entire specification. There is no header convention, no schema requirement, no robots-style allow/deny syntax.
What llms.txt is not: it is not a substitute for robots.txt, it is not an access-control mechanism, and as of May 2026 it is not consumed at inference time by ChatGPT, Claude, Gemini, Perplexity, or Copilot in any documented production system. Server-log audits across multiple independent practitioners show GPTBot, ClaudeBot, and Google-Extended do not request /llms.txt in meaningful volume during routine crawls.
The realistic 2026 use case is developer tooling. AI coding assistants and IDE agents — Cursor, GitHub Copilot, Claude Code, and similar tools — retrieve docs in real time, and a curated llms.txt cuts token waste by pointing them at canonical Markdown sources instead of HTML-rendered pages bloated with nav and tracking. Companies like Anthropic, Stripe, Cursor, Cloudflare, Vercel, Mintlify, Supabase, and LangGraph ship llms.txt for that reason.
The 4-element template — a working example
Here is a real, valid llms.txt for a hypothetical SaaS docs site. Copy this structure, change the project name, and you have a shippable file in under 30 minutes:
# Acme Analytics
> Acme Analytics is a self-hosted product analytics platform for SaaS teams. This file points AI assistants and IDE agents at canonical Markdown documentation, not the rendered HTML.
Authoritative Markdown sources for product, API, and SDK documentation. Use the `.md` variant of any docs page (append `.md` to the URL) for a clean, agent-friendly version.
## Getting Started
- [Quickstart](https://acme.example/docs/quickstart.md): 10-minute setup, install through first event.
- [Concepts](https://acme.example/docs/concepts.md): events, properties, identities, sessions — definitions and examples.
## API Reference
- [REST API Reference](https://acme.example/docs/api/rest.md): every endpoint, request/response schema, rate limits.
- [Webhook Reference](https://acme.example/docs/api/webhooks.md): payload contracts and retry behavior.
## SDKs
- [JavaScript SDK](https://acme.example/docs/sdk/js.md): browser and Node, including server-side rendering notes.
- [Python SDK](https://acme.example/docs/sdk/python.md): server-side ingestion patterns.
## Optional
- [Changelog](https://acme.example/docs/changelog.md): version history, breaking changes flagged inline.
Two practitioner notes. First, the spec uses an “Optional” H2 as a soft signal — links under that heading can be skipped by aggressive token budgets. Second, the file is most useful when every linked URL has a parallel .md Markdown version. If your site is pure HTML, llms.txt without paired Markdown does little.
The robots.txt pairing — this is what actually controls AI bots today
The lever that meaningfully controls AI crawler behavior in 2026 is robots.txt with user-agent–specific rules. Anthropic publishes official documentation for three bots — ClaudeBot for training, Claude-User for user-initiated fetches, and Claude-SearchBot for search indexing — and confirms all three honor robots.txt. OpenAI runs GPTBot (training) and OAI-SearchBot (live ChatGPT search). Google’s AI training opt-out is the Google-Extended user-agent. Perplexity uses PerplexityBot.
The two-bucket pattern most practitioner sites should ship: block training-only crawlers, allow search and user-initiated retrieval so your content can still be cited in answers.
# Allow AI search and user-fetch traffic (citations, attribution)
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Block training-only crawlers
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Standard search crawler — leave open
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xml
One operational caveat: robots.txt is policy, not enforcement. Anthropic, OpenAI, and Google have all publicly committed their named bots to compliance, but unnamed scrapers and residential-IP harvesters routinely ignore it. For sites with sensitive content, pair robots.txt with WAF or Cloudflare bot-management rules at the edge.
Structured data still does more heavy lifting than llms.txt
If your goal is AI citation rather than IDE-agent retrieval, structured data on the page itself moves the needle more than llms.txt. The minimum stack for any article you want cited: Article schema with named author and publisher, FAQPage schema on any post that answers a discrete question, and speakable markup on the answer paragraphs. These get parsed during normal HTML fetches by every major AI crawler — no separate file required.
How to verify your llms.txt is actually being read
Ship the file, then run this server-log filter weekly for 30 days. On any standard access-log format (nginx, Apache, or a Cloudflare log push), grep for requests to /llms.txt and break them down by user-agent:
grep "GET /llms.txt" /var/log/nginx/access.log \
| awk -F\" '{print $6}' \
| sort | uniq -c | sort -rn
What you will almost certainly see in May 2026: a steady trickle of human curl requests, the occasional IDE agent fetch tagged with a Cursor or VS Code user-agent, and effectively zero hits from GPTBot, ClaudeBot, or Google-Extended. That null result is itself the measurement — it tells you llms.txt is a developer-experience asset right now, not an AI-citation asset, and your investment should match that reality.
The recommended 2026 rollout
For most sites, the right sequence is: ship the robots.txt user-agent rules above first, because those are enforceable today and shape every AI crawler interaction. Add structured data to every article that competes for AI citation. Then publish llms.txt — under 30 minutes of work — for the IDE-agent and dev-tooling upside, with no expectation of immediate search lift. When OpenAI, Anthropic, or Google publicly confirm production llms.txt consumption, you are already in position.
Leave a Reply