The definitive resource for GEO (Generative Engine Optimization), AEO (Answer Engine Optimization), LLMs.txt, and ranking in AI-powered search — Perplexity, ChatGPT, Claude, Google AI Overviews.
This slug is a duplicate of the ranking desk. Do not treat numbers on this URL as current.
Use the live page: Claude AI Pricing (September 2026). Seats and API rates are verified there against claude.com/pricing and the official API table. This URL is noindexed and canonicalized to that slug.
Current flagship API list (as of 8 September 2026, restated from the hub): Haiku 4.5 $1/$5, Sonnet 5 $2/$10, Opus 5 $5/$25, Fable 5.1 $10/$50. Seats are not API credits.
In 2026, the digital marketing landscape has fundamentally shifted. For over two decades, agencies fought for the “blue link” on traditional Search Engine Results Pages (SERPs). Today, that battlefield has been eclipsed by an AI-synthesized ecosystem. With over 25% of traditional searches triggering AI Overviews, and millions of users migrating to conversational platforms like ChatGPT, Gemini, and Perplexity, the primary goal of search marketing has moved from earning a click to being cited, mentioned, or recommended by an AI model.
If your agency is still pitching traditional SEO to clients without a comprehensive Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) strategy, you are optimizing for a web that no longer exists.
High rankings on a traditional SERP no longer guarantee traffic. Users increasingly complete their research, compare products, and make buying decisions directly within AI interfaces without ever clicking through to a source website. This “zero-click” reality means agencies must optimize for AI authority rather than just keyword density. If a generative engine does not view your client as a trusted entity, they simply will not exist in the answers provided to the end-user.
While often used interchangeably, Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO) represent two sides of the same critical coin:
Because traditional organic traffic is diminishing for top-of-funnel queries, forward-thinking agencies have abandoned traffic-only reporting. In 2026, performance is measured by AI-specific KPIs:
Companies that are winning in 2026 have moved beyond “writing for search engines” to “writing for AI models.” This requires a convergence of PR, content marketing, and technical SEO.
First, agencies must prioritize E-E-A-T (Experience, Expertise, Authoritativeness, and Trust). Because AI models aggregate information from across the web—including forums, reviews, social media, and news—agencies must manage an “omnichannel” presence. A brand’s narrative must be consistent everywhere the AI might look.
Second, structured data and schema markup are no longer optional. Modern GEO requires engineering content for machine “extractability.” By ensuring that data is neatly organized and technically transparent, agencies reduce the risk of AI hallucinations and increase the likelihood that a model will trust and cite the provided information.
Generative Engine Optimization does not replace traditional SEO; it builds upon it. Traditional SEO creates the foundational web presence, while GEO ensures that presence is visible and authoritative across the fragmented AI search landscape. In the “Search Everywhere” era, research no longer starts and ends on a single search engine. Agencies that adapt to this reality will dominate the next decade of digital marketing.
Tygart Media Insights: Preparing agencies and tech leaders for the future of search, artificial intelligence, and digital authority.
Related on Tygart Media: SEO vs GEO vs AEO · GEO tactics · AI citation monitoring.
Related on Tygart Media: GEO tactics · SEO vs GEO vs AEO · AEO content optimizer skill.
Most conversations about AI crawlability focus on one file: llms.txt. But if you look at what Anthropic, Vercel, and LangGraph actually ship – and what GEO crawler research found AI agents fetching most – the file that matters more is its companion: llms-full.txt.
Here’s the practical reality: llms.txt is the map. llms-full.txt is the territory. And in 2026, the agents that matter for citation traffic are fetching the territory.
The original llms.txt proposal – published by Jeremy Howard in September 2024 – defined one file. Implementers built the rest. The complete family as of mid-2026 is four files, but most sites only need two:
| File | What’s in it | When to use |
|---|---|---|
/llms.txt | Curated index – H1, summary, link sections | Always. The orientation layer. |
/llms-full.txt | Full content of every linked page, concatenated as Markdown | When you want a model to deep-ingest your docs in a single fetch |
/llms-ctx.txt | Pre-expanded context without URLs | FastHTML-style implementations |
/llms-ctx-full.txt | Pre-expanded context with URLs preserved | Same, but URL-aware |
The pattern that works – and the one Anthropic, Vercel, and LangGraph all run – is the index + export pair: llms.txt for orientation, llms-full.txt for deep ingestion.
GEO researchers analyzing AI crawler behavior – including work cited by Profound – have noted that agents from Microsoft, OpenAI, and others tend to fetch llms-full.txt more frequently than llms.txt when both are present. The working explanation is structural: when a file contains the full content, it removes one retrieval step. An agent that fetches llms-full.txt gets everything it needs in a single HTTP request instead of fetching the index, parsing the links, then fetching each linked page individually. This is consistent with how developer documentation platforms like Mintlify describe the behavior of IDE agents operating under tight latency budgets.
For IDE agents (Cursor, Continue, Cline) and MCP integrations, this is even more pronounced. These tools are operating under tight context windows and latency budgets. A single fetch that returns a clean Markdown blob of your entire docs is structurally preferable to a multi-step crawl.
The implication: if you’ve shipped llms.txt but not llms-full.txt, you’ve done half the job.
The construction logic is simple: take every URL in your llms.txt, fetch each page, strip HTML to Markdown, and concatenate. In practice, most sites do this in their build pipeline.
Here’s the minimal Node.js pattern:
const fs = require('fs');
const fetch = require('node-fetch');
const TurndownService = require('turndown');
const turndown = new TurndownService();
async function buildLlmsFullTxt(llmsIndexPath, outputPath) {
const index = fs.readFileSync(llmsIndexPath, 'utf8');
const urlRegex = /\[.*?\]\((https?:\/\/[^\)]+)\)/g;
const urls = [...index.matchAll(urlRegex)].map(m => m[1]);
let output = '';
for (const url of urls) {
const res = await fetch(url);
const html = await res.text();
const markdown = turndown.turndown(html);
output += \n\n---\n# Source: \n\n;
}
fs.writeFileSync(outputPath, output);
console.log(Built llms-full.txt: pages, chars);
}
buildLlmsFullTxt('./public/llms.txt', './public/llms-full.txt');One constraint to manage: keep llms-full.txt under roughly 200,000 tokens (about 150K words, around 700KB). That’s the threshold where most models can ingest the file in a single context window. If your docs are larger, segment by product or language the way Supabase does – llms-full-api.txt, llms-full-guides.txt – and list the segmented files in your main llms.txt.
Shipping llms.txt and llms-full.txt is the visibility layer. The access-control layer is robots.txt – and it changed significantly in Q2 2026.
The key development: Anthropic split its crawler into two separate user-agents. ClaudeBot is the training scraper (high bandwidth, no citation value – block it). Claude-Web is the live-retrieval agent that fetches pages to answer Claude.ai user queries in real time (allow it, because it drives citation traffic). Brands that blanket-block “all Anthropic crawlers” lose Claude citations entirely.
Meta also shipped two active training scrapers in March 2026 – FacebookBot and Meta-ExternalAgent – at GPTBot-level crawl volume. Most sites have no rules for them yet.
Here’s the 2026 template:
# BLOCK: Training scrapers - high bandwidth, zero referral value
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
# OPT OUT: Google Gemini training (keeps Search indexing intact)
User-agent: Google-Extended
Disallow: /
# ALLOW: Live-retrieval agents - drive citation traffic
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /One important caveat on robots.txt enforcement: aggressive training scrapers often ignore the file or spoof their user-agents. The robots.txt rules signal intent and work for compliant bots; a WAF rule at the edge is the only deterministic block for non-compliant crawlers.
The SERanking study of 300,000 domains (November 2025) found no measurable correlation between having llms.txt and being cited by ChatGPT, Claude, Gemini, or Perplexity. Google’s John Mueller compared the file to the deprecated keywords meta tag – something site owners declare but that search systems derive from the content itself.
None of that means you shouldn’t ship both files. The cost is low, the optionality is real, and the IDE-agent ecosystem (Cursor, Continue, Cline) does actively use llms.txt. But the robots.txt work is the lever that moves outcomes today. The llms.txt + llms-full.txt pair is infrastructure investment – you want to be correct when major LLM providers start honoring it, and building the build pipeline now costs far less than retrofitting it later.
The practical sequence for a site that hasn’t done this yet:
curl -I https://yoursite.com/llms.txt should return 200 with Content-Type: text/plain. A 404 on either file is the most common implementation error./llms.txt and /llms-full.txt by user-agent. You want to see live-retrieval agents (Claude-Web, OAI-SearchBot, PerplexityBot) in the results – not just training scrapers.The goal isn’t to optimize for a standard that isn’t fully adopted yet. It’s to build the infrastructure correctly now, while the field is still forming, so that adoption changes work in your favor rather than requiring catch-up.
llms.txt is a curated index — an H1, a summary, and link sections that orient an AI agent to your site. llms-full.txt is the full content of every linked page concatenated as Markdown, so an agent can deep-ingest your documentation in a single fetch. The index is the map; the full file is the territory.
Fetching llms-full.txt removes a retrieval step: the agent gets everything in one HTTP request instead of fetching the index, parsing links, and fetching each page individually. For IDE agents like Cursor, Continue, and Cline operating under tight latency and context budgets, a single clean Markdown blob is structurally preferable to a multi-step crawl.
Keep it under roughly 200,000 tokens (about 150K words, around 700KB) so most models can ingest it in a single context window. If your docs are larger, segment by product or language — for example llms-full-api.txt and llms-full-guides.txt — and list the segmented files in your main llms.txt.
Not measurably on its own. A November 2025 SERanking study of 300,000 domains found no correlation between having llms.txt and being cited by ChatGPT, Claude, Gemini, or Perplexity, and Google’s John Mueller compared it to the deprecated keywords meta tag. The lever that moves outcomes today is robots.txt configuration; llms.txt and llms-full.txt are low-cost infrastructure for when adoption grows.
Allow live-retrieval agents that drive citation traffic — Claude-Web, OAI-SearchBot, ChatGPT-User, anthropic-ai, and PerplexityBot. Block high-bandwidth training scrapers with no referral value such as GPTBot, CCBot, ClaudeBot, FacebookBot, and Meta-ExternalAgent, and opt out of Google-Extended to skip Gemini training while keeping Search indexing intact.
Last verified: June 2026.
Most “GEO” advice is recycled SEO with the word “AI” pasted on top. This guide is different. It describes what actually happens when Microsoft Copilot, Bing’s AI answers, and Google’s AI Overviews build a response and decide whose page to cite — based on running content sites that get cited tens of thousands of times a month. The short version: AI engines do not cite the page that ranks #1 for a head term. They cite the page that most directly answers the specific sub-question the model is grounding on. That distinction changes everything about what you should write.
When you ask Copilot or Bing’s AI a question, the model does not answer from memory. It runs a retrieval step called grounding: it rewrites your question into one or more search queries, fetches a handful of live web results, reads them, and composes an answer with inline citations pointing back at the pages it used. Google’s AI Overviews work the same way with a technique it calls “query fan-out” — one user question becomes many narrower synthetic queries.
Two things follow directly from this mechanism:
This is why a niche, specific page routinely out-cites a high-authority generalist. The generalist ranks; the specialist gets quoted.
Across real citation data, the pages that get pulled into AI answers cluster into three shapes. None of them are “ultimate guide to X”.
When someone asks an AI assistant “how do I fix [specific error]” or “what’s the exact command to do X”, the model needs a page that contains the literal command, the literal config, or the literal error string. Generic advice cannot be cited because there is nothing concrete to quote. A page that says:
curl "https://www.bing.com/indexnow?url=https://example.com/new-page/&key=YOUR_KEY"
# 200 = received (not "indexed"), 422 = URL/key mismatch, 429 = too many submits…is citation gold, because the model can extract that block verbatim and the user can act on it. The error-code annotations matter: questions about failures (“IndexNow 422”, “why am I getting 429”) are high-intent and low-competition, and a page that names the exact codes owns them.
“Which is better, X or Y” is one of the most common shapes of AI query, and comparison content is structurally easy to cite because it maps cleanly to a decision. If you maintain honest, current head-to-head pages, you become the default source the model reaches for when a user is choosing between tools. This is exactly why we keep dedicated comparison pages like Claude Code vs Cursor and Claude Code vs Codex — they answer a decision the model is constantly being asked to make, and a table of differences is trivially quotable.
For anything that changes — pricing, model versions, API limits, feature availability — grounding strongly favors recency. The model would rather cite a page dated this month than an “authoritative” page from two years ago that might be wrong. A visible “Last verified” date and a real publish/update timestamp are not decoration; they are a relevance signal the retrieval layer reads.
The losing move is chasing broad head terms. “Best AI coding assistant” is saturated, generic, and rarely the literal grounding query. The winning move is to own the long, specific, operational and comparison strings that the fan-out actually generates.
Grounding can only cite pages the engine knows about. The bottleneck for new content is crawl latency — and IndexNow collapses it. IndexNow is an open protocol (backed by Microsoft Bing and Yandex) that lets you push a URL to the index the instant you publish, instead of waiting for a crawler to wander by.
Setup is two steps:
{key}.txt containing exactly that key. Example: https://example.com/daa44a2c....txt. This proves you own the host.curl "https://api.indexnow.org/indexnow?url=https://example.com/new-page/&key=YOUR_KEY"curl -X POST "https://api.indexnow.org/indexnow" \
-H "Content-Type: application/json" \
-d '{"host":"example.com","key":"YOUR_KEY","urlList":["https://example.com/a/","https://example.com/b/"]}'A 200 means the endpoint received your URL (not that it is indexed yet). Submitting to api.indexnow.org shares the ping with all participating engines, so you do not need to hit Bing and Yandex separately. Most WordPress SEO plugins (Rank Math, Yoast, SEOPress) have IndexNow built in — turn it on and it fires automatically on every publish and update. The practical payoff: pages can enter Bing’s crawl queue within hours, which means they are eligible to be grounded and cited the same day, not next week.
One caveat worth stating plainly: IndexNow accelerates indexing, which is a precondition for citation. It does not force a citation. You still need the page to be the best answer to the sub-question. But for fresh, time-sensitive content, same-day indexing is often the difference between getting cited while the topic is hot and showing up after the conversation has moved on.
For a long time AI citations were invisible — you could see referral clicks in analytics but not the citations themselves (most AI answers are zero-click). That changed. As of February 2026, Bing Webmaster Tools ships an AI Performance report (public preview) that shows when your pages are cited across Microsoft Copilot, Bing’s AI answers, and partner surfaces. It is the first direct, free window into AI citation behavior, and you should be reading it weekly.
The four metrics that matter:
Two limitations to keep in mind so you read the data honestly: the report does not show click data (you see citations, not visits from them), and it aggregates Copilot with Bing summaries, so you cannot isolate one surface from the other. For Google’s AI Overviews there is still no equivalent citation dashboard — the closest proxy is watching impressions and referral patterns in GA4 and Search Console, plus spot-checking your target queries by hand.
The workflow that works: pull the grounding-queries list, find the patterns, and feed them straight back into your content plan. If you are getting cited for “claude mcp setup” variants, that is a signal to deepen pages like the Claude MCP setup guide and adjacent operational walkthroughs, not to chase a new head term.
Everything above reduces to a build pattern. For any page you want AI engines to cite:
If you want the larger system this fits into — the full toolchain for operating as an AI-first publisher, from MCP servers to publishing pipelines — start with the AI operator’s stack.
Not reliably. AI engines run their own grounding retrieval and cite the page that most directly answers the specific decomposed sub-question, which is often a niche, operational page rather than the head-term winner. Ranking helps your page be discoverable, but the citation goes to whichever passage best answers the exact grounding query.
Grounding is the retrieval step where an AI assistant rewrites your question into search queries, fetches live web pages, reads them, and builds an answer with inline citations to those pages. It is why current, specific pages can get cited even by a model whose training data predates them.
No. IndexNow guarantees fast indexing, which is a precondition for being cited. The page still has to be the best, most specific answer to the sub-question the model is grounding on. Think of IndexNow as removing the crawl-latency excuse, not as buying a citation.
Use the AI Performance report in Bing Webmaster Tools (public preview since February 2026). It shows total citations, average cited pages per day, sample grounding queries, and citation counts by URL across Microsoft Copilot and Bing AI answers. It does not yet show click-through from those citations, and there is no equivalent dashboard for Google AI Overviews.
No. Citation extraction works on visible, well-structured text — question-shaped headings, short factual answers, and a plain visible FAQ. Schema can help search features generally, but it is not required for AI grounding to read and quote your page.
Three shapes dominate: operational pages with real commands, configs, and error fixes; comparison pages that resolve a “X vs Y” decision; and fresh, dated pages on fast-moving topics like pricing and model versions. Broad head-term content tends to get skipped because it rarely matches the literal grounding query and offers nothing concrete to quote.
Related on Tygart Media: citation economy · AI search funnel · citation monitoring.
Related on Tygart Media: GEO tactics · AI search funnel · SEO vs GEO vs AEO.
ChatGPT Search cites 15% of the pages it retrieves. The other 85% get pulled into the model’s context window, evaluated, and silently discarded — no visibility, no referral, no trace. If you are doing GEO work and your pages keep getting retrieved but never quoted, you are losing at the second filter, not the first.
This is the 2026 implementation guide for surviving both filters: getting retrieved by ChatGPT Search, then getting cited once you are there.
ChatGPT Search runs a three-stage pipeline. Each stage kills most candidates.
Most sites optimized for Google have never logged into Bing Webmaster Tools. Fix that first. Three checks before anything else:
If pages are missing, submit a sitemap via Bing Webmaster Tools and request URL inspection on any priority page. Bing typically reflects changes within 24–72 hours, faster than Google.
The single most-skipped step in GEO work. Add this block to your robots.txt:
# Allow ChatGPT Search to retrieve and cite this site
User-agent: OAI-SearchBot
Allow: /
# Optional: allow on-demand browsing for ChatGPT users
User-agent: ChatGPT-User
Allow: /
# Optional: block training crawler if you want retrieval without training
User-agent: GPTBot
Disallow: /OpenAI publishes these three user agents and treats each independently. You can allow OAI-SearchBot for ChatGPT Search visibility and still disallow GPTBot from using your content for model training. The settings do not conflict. OpenAI’s systems typically recognize robots.txt changes within 24 hours.
Retrieval is necessary but not sufficient. Once your page is in the candidate set, the model decides whether to quote it. Pages that get quoted share a structural pattern.
ChatGPT cites sources that answer the question fully. Partial answers lose to complete ones. Lead each page with a clean direct-answer paragraph: question implied or stated, answer in the next sentence, supporting detail after. This is the same pattern that wins featured snippets, which is not a coincidence — answer engines and snippet engines reward the same structure.
An AirOps study of 548,534 pages found pages with JSON-LD markup posted a 38.5% citation rate versus 32.0% without it. Article, FAQPage, and HowTo schema are the highest-leverage types. Add them.
Pages between 500 and 2,000 words performed best in the same AirOps study. Pages longer than 5,000 words were cited less often than pages under 500. The mechanism is mechanical: long pages overflow the retrieval context window, and the model defaults to shorter, denser sources it can quote in full.
Content updated within 30 days received 3.2x more citations than older material. The fix is not faked freshness — it is genuine updates: a new stat, a new case, a corrected claim. Update the date when you update the content, not before.
Structure gets you cited once. Authority gets you cited repeatedly. AirOps found sites with over 32,000 referring domains are 3.5x more likely to be cited by ChatGPT than sites with fewer than 200. You do not need 32,000 — you need to be in the upper band of your topical neighborhood.
ChatGPT’s citation pattern leans heavily on Wikipedia (roughly 48% of top citations in multiple studies) and large news/media properties. The practitioner read on that: ChatGPT favors sources with multi-source third-party validation. Build the kind of citations on the open web that Wikipedia editors accept — peer-reviewed studies, primary sources, named author attribution, transparent methodology.
You cannot manage what you do not measure. The minimum tracking stack for 2026:
Ranking in ChatGPT in 2026 is not mysterious. It is a four-gate funnel: Bing index → OAI-SearchBot crawl access → retrieval into the candidate set → citation selection. Most sites fail at gate one (not indexed in Bing) or gate two (OAI-SearchBot blocked or not addressed). Sites that clear those two gates and write pages that answer the question fully, with schema and a 500–2,000-word range, will land in the 15% that get quoted.
Treat ChatGPT Search like a separate search engine that happens to share an index with Bing. Optimize for the index. Allow the crawler. Write the page. The rest follows.
Related on Tygart Media: AI citation monitoring · AI search funnel · how AI engines cite.
You shipped an llms.txt file. You curated the links, you paired it with robots.txt, you validated the format. Now answer the only question that matters: is anything actually requesting it? Most site owners never check — and the data from 2026 suggests the honest answer, for most domains, is “almost nothing.” This is the verification step that turns llms.txt from an act of faith into a measurable signal. Here is how to read your own server logs and find out exactly what is fetching the file you published.
The uncomfortable finding of the last year is that publishing llms.txt and benefiting from llms.txt are two different things. In OtterlyAI’s 90-day crawler study, only 0.1% of AI crawler requests touched /llms.txt at all — 84 requests out of 62,100 total AI bot visits — and the file received far fewer visits than the average content page (OtterlyAI GEO study). As of Q1 2026, no major AI company — OpenAI, Google, Anthropic, Meta, or Mistral — has publicly committed to reading or acting on llms.txt in production systems, though GPTBot does fetch the file occasionally (AEO Engine).
That does not make the file worthless. It makes measurement the whole game. If you cannot tell whether a crawler ever requested the file, you cannot tell whether your time was wasted, whether a platform quietly started honoring it, or whether your file is returning a silent 404. Verification is the difference between strategy and superstition.
Every fetch of your llms.txt file leaves a row in your access log. The job is to isolate requests to that path, then filter by the user-agents that belong to AI systems. On any server with standard combined-format Apache or Nginx logs, this one-liner does the first pass:
grep -E "/llms(-full)?\.txt" /var/log/nginx/access.log | \
grep -E -i "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Google-Extended|Google-CloudVertexBot|Amazonbot|CCBot|Applebot|meta-externalagent|MistralAI-User|bingbot"The first grep narrows to requests for llms.txt or llms-full.txt. The second filters to the known AI crawler user-agent strings documented across 2026 reference work (No Hacks AI User-Agent Landscape 2026; Momentic crawler list). Each surviving line tells you three things: which bot, what time, and the HTTP status code it received.
That status code is the part people skip. A 200 means the bot got your file. A 404 means you have been congratulating yourself over a file the crawler never actually reached — a misconfigured path, a redirect loop, or a build step that drops the file on deploy. A 301 or 302 means it is being redirected, and not every crawler follows redirects for this path. Read the status column before you read anything else.
One grep tells you whether the file is reachable. To know whether anything is changing, you need the same query run on a schedule and counted by bot. Extend the pipeline to a count:
grep -E "/llms(-full)?\.txt" /var/log/nginx/access.log* | \
grep -E -i -o "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|bingbot|Amazonbot|CCBot|Applebot" | \
sort | uniq -c | sort -rnThis produces a leaderboard of which AI user-agents requested your llms.txt across all retained logs. Capture that number on the first of each month and you have a cadence series. The signal you are watching for is not the absolute count — it will be small — but the direction: a bot that appears for the first time, a bot whose hit count jumps, or a bot that goes silent. Those inflection points are the leading indicators that a platform has changed how it treats the file.
| What you see in the log | What it means | Action |
|---|---|---|
No requests to /llms.txt at all | File may be unreachable, or simply not yet fetched — both are common | Request the URL yourself; confirm a clean 200 before assuming neglect |
200 from GPTBot, low frequency | Consistent with reported behavior — GPTBot fetches occasionally | Log the cadence; treat as baseline, not a ranking signal |
404 or 301 on the path | Crawler is not getting the file you think you published | Fix the path/redirect today — this is a silent failure |
| A new bot appears month-over-month | A platform may have started fetching the file | Note the date; correlate with any citation or referral changes |
The llms.txt hit count means little in isolation. Compare it against how often the same bots fetch your actual content pages. If GPTBot pulls forty content URLs a day and never touches llms.txt, the file is not part of how that crawler discovers you — your content’s own structure and internal linking are doing the work. The practical monitoring approach documented for 2026 is exactly this: a server-log dashboard built against the major user-agents, watching cadence and path-preference shifts month over month (Digital Applied 30-day log study). The same study notes distinct personalities worth knowing — GPTBot crawls more aggressively than most assume, ClaudeBot is more patient than its volume suggests, and PerplexityBot is quieter than its share-of-voice would predict.
If your logs show the file is reachable and occasionally fetched, you are in the normal range for 2026 — keep the file current and keep measuring. If they show a 404, you found a real bug that no amount of curation would have fixed. And if they show a brand-new bot starting to request the path, you have spotted a platform behavior change before the blog posts catch up to it. That last case is the entire payoff: the practitioners who read their own logs will know the standard started mattering weeks before the ones who only read about it. Verification is not the boring final step of an llms.txt rollout. On a standard that nobody has formally committed to honoring yet, it is the only step that produces evidence instead of hope.
Related on Tygart Media: AI crawler experiment · GEO tactics · Bing Webmaster AI tab.
Most generative engine optimization advice still reads like SEO advice with new vocabulary. Add statistics. Build entities. Earn mentions. All true, all incomplete. The mechanic that determines whether ChatGPT, Perplexity, or Google AI Overviews quote your page in an answer is not the page. It is the chunk — the 200- to 500-character passage the retrieval layer pulled out of your page, scored against the user’s prompt, and handed to the language model as evidence.
If your paragraphs do not survive that extraction step intact, the rest of your GEO program is academic. This is the implementation gap most content teams have not closed yet, and it is the highest-leverage shift you can make in Q2 2026.
When a user asks Perplexity or ChatGPT a question, the system runs a process best described as query fan-out and chunked retrieval-augmented generation (RAG). The prompt is decomposed into sub-queries. Each sub-query is sent to a search index (Bing for ChatGPT, a proprietary index plus partner search for Perplexity, Google’s own corpus for AI Overviews). Top-ranking pages are fetched, broken into chunks, and re-scored against the original prompt for semantic match, factual density, source authority, and recency.
The model then composes its answer from the three to seven highest-scoring chunks across all retrieved pages. The visible citations are the source pages those winning chunks came from. Your page can rank well in the underlying search index and still produce no chunks that score high enough to enter the answer. That is the silent failure mode in GEO right now: traffic-tier visibility, zero citation share.
The optimization target is a paragraph that reads as a self-contained answer when removed from the page around it. No pronouns referring back to a previous heading. No “as we discussed above.” No buried lede. The first sentence is the claim. The second through fifth sentences supply the supporting fact, the qualifier, and the source if one is needed.
Concretely, here is the same answer written two ways. The first will not survive extraction. The second will.
Will not chunk well:
As we covered earlier in this post, the answer depends on what you are trying to measure. It is more nuanced than most people assume. There are several factors at play, including the ones we mentioned in the introduction.
Will chunk well:
LLMs.txt is a plain-text file at the root of a domain that points AI crawlers to the most authoritative Markdown versions of a site’s documentation. The file format was proposed by Jeremy Howard in September 2024 and has seen adoption signals from major AI vendors through 2025 and into 2026. A minimal valid file is twelve lines and takes under ten minutes to deploy.
The second version has a definition, a provenance fact, an adoption signal, and a deployment qualifier — four extractable units in three sentences. A retrieval system scoring chunks for “what is llms.txt” will rank this passage higher than a longer paragraph that buries the same facts under hedging language.
These rules come from observing what actually appears in Perplexity citations, ChatGPT browsing answers, and AI Overview extractions across hundreds of cited passages. They are mechanical. Apply them in revision passes, not at first draft.
1. One claim per paragraph. Multi-claim paragraphs lose to single-claim paragraphs because the retriever cannot score them as cleanly against a specific sub-query. If you have three claims, write three paragraphs.
2. Front-load the noun and the verb. The first eight words of the paragraph determine semantic match. “Generative engine optimization is…” beats “When thinking about how to approach modern search, generative engine optimization is…” every time.
3. Resolve every pronoun within the paragraph. If a paragraph says “it” or “this” without naming the antecedent inside the same paragraph, the chunk reads as orphaned to the retriever and gets discounted.
4. Keep paragraphs between forty and one hundred twenty words. Shorter paragraphs lack the factual density that scores well. Longer paragraphs get truncated mid-thought, which destroys the chunk. The forty-to-one-twenty band is where modern retrievers operate cleanly.
5. Put the source inline. “Princeton research published in 2023 found a 30 to 40 percent visibility lift from adding statistics and citations” outperforms the same fact with a footnote, because the retriever sees the authority signal in the same chunk as the claim.
For any page already ranking in the top twenty for a target query, run this three-step pass before chasing new content.
Step one: Print the article. Cover all headings. Read each paragraph in isolation. Mark any paragraph that does not answer a specific question on its own. That mark is your rewrite list.
Step two: For each marked paragraph, identify the implicit question it is trying to answer. Rewrite the first sentence to state the answer. Move supporting context into sentences two through four. Cut anything past sentence five into a new paragraph.
Step three: Add one inline source per claim that involves a number, a date, or a contested fact. Inline means “according to Anthropic’s official documentation,” not a hyperlinked footnote at the end of a sentence.
A site with eighty published pages can complete this pass in four to six weeks at one editor’s pace. The lift typically shows in AI referral traffic in GA4 — under Acquisition, Traffic Acquisition, with a manual segment for sessions where the source contains “chatgpt,” “perplexity,” “claude,” “copilot,” or “gemini” — within three to five weeks of the changes going live, because retrieval indexes refresh on independent cycles from Google’s main crawl.
New content takes weeks to be indexed by the underlying search layer and additional weeks before the retrieval scoring stabilizes. Rewritten paragraphs on already-indexed pages start scoring against retrieval queries the next time the page is recrawled, typically within days. The compound effect of converting forty already-ranking pages into chunk-optimized pages is larger and faster than the effect of publishing forty new pages.
This is the GEO discipline that separates teams who say they are doing generative engine optimization from teams whose names appear in actual AI answers. The unit of work is the paragraph. The test is whether the paragraph survives extraction. Everything else — entity binding, schema, llms.txt, brand co-occurrence — sits on top of that foundation.
Related on Tygart Media: RAG / source-worthy content · GEO tactics · SEO vs GEO vs AEO.
What is the ideal chunk length for GEO?
Modern retrievers extract chunks in the 200 to 500 character range, which corresponds to paragraphs of roughly 40 to 120 words. Paragraphs in this band give retrievers enough context to score factual density without losing the chunk to mid-paragraph truncation.
How is chunk-first GEO different from entity optimization?
Entity optimization tells the AI system who you are. Chunk-first writing tells the AI system what to quote. The two operate on different surfaces and are complementary. Entity work without chunk-survivable paragraphs leaves you recognized but unquoted.
Do headings matter for chunk extraction?
Headings help retrievers segment the document and improve the score of the paragraph immediately below the heading. The heading-then-clear-paragraph pattern is the strongest GEO structure currently observable in AI Overview citations.
How do I measure whether my chunks are getting cited?
Track AI referral sessions in GA4 with a segment filtering for source contains chatgpt, perplexity, claude, copilot, or gemini. Pair that with prompt-set testing in tools that query multiple LLMs with your target queries and parse the cited URLs from the responses.
Will Google penalize chunk-optimized writing?
Chunk-optimized paragraphs read as cleanly written, source-attributed prose. The same structural rules that help retrieval scoring also help featured snippet capture and traditional on-page SEO. There is no documented penalty signal and the structure is consistent with Google’s own quality rater guidelines on clear, useful writing.
The LLMs.txt file was supposed to be the AI-era equivalent of robots.txt — a clean, declarative way to hand large language models a curated map of your most valuable content. Three years after Jeremy Howard proposed the spec, the data is in. And the data is not what implementation evangelists have been promising.
This is a case study teardown of the three largest independent measurement efforts on LLMs.txt adoption and citation impact, the one documented recovery case where it did move the needle, and the structural lesson every practitioner should pull from the divergence.
A widely circulated dataset of nearly 300,000 domains — analyzed across multiple AI search citation benchmarks and reported by Search Engine Journal — found no statistically significant relationship between implementing LLMs.txt and how often AI engines cite a brand. Both standard statistical analysis and machine-learning models showed no effect. Removing LLMs.txt as a feature actually improved citation prediction accuracy in one model run, meaning the file’s presence was less than noise.
Adoption sits at roughly 10.13% of domains in that dataset, distributed evenly across traffic tiers. Translation: it is neither standard practice nor a differentiator.
A separate bot-traffic audit reported by adoption researchers found that out of 62,100-plus AI bot visits over a 90-day window, only 84 requests targeted the /llms.txt path. Across half a billion LLM bot traffic events analyzed in another dataset — filtering for the agents that actually drive citations (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) — the share of requests touching /llms.txt was statistically negligible.
As of Q1 2026, no major AI company — OpenAI, Google, Anthropic, Meta, or Mistral — has publicly committed to reading or acting on LLMs.txt in production systems. The file is a community proposal, not a supported standard. AI language models learn what to trust from the web as it existed during training. Citation behavior reflects which sources appeared consistently in training corpora, which were cited by other credible sources, and which had claims independently corroborated. A crawl-directive file published after training cannot retroactively change any of that.
Compare that to a documented recovery case reported by SEO Algorithm Recovery and corroborated by independent AI Overviews tracking: a Dallas retailer lost 72% of organic traffic to AI Overviews. Their agency deployed schema markup and restructured 150 pages around answer-first formatting. Traffic recovered to 118% of pre-AI Overview levels in 120 days, with $1.4M in revenue growth attributed to the recovered organic channel.
No LLMs.txt was involved. The intervention stack was schema markup, content restructuring for AI-extractable answers, and entity disambiguation in headings. Schema markup alone has been reported to recover 45%-plus of lost AI Overview traffic in case-study compilations across the recovery agency space.
The contrast is the case study. LLMs.txt is a static directive file that AI crawlers do not currently read at scale. Schema markup is a structured-data layer that AI systems already parse to construct answer panels and citation surfaces. One is aspirational. The other is operational.
The structural pattern under every documented AI-search recovery in 2026 is the same: answer-first content directly under each H2, structured data on the entity being described, tables for comparison data, and explicit source attribution inline. Sites earning AI citations report traffic gains. Brands with strong authority signals benefit from the halo effect. Companies adapting these specific structural interventions early — not the file directives — are the ones reporting growth exceeding pre-AI Overview levels.
The skeptical case is not “skip LLMs.txt entirely.” It is “do not let it absorb hours that should go to schema and content restructuring.” A minimum-viable LLMs.txt is ten lines and takes ten minutes to ship:
# Your Brand Name
> One-sentence description of what your site is and who it serves.
## Core Pages
- [About](https://yoursite.com/about): Who you are, in one paragraph.
- [Products](https://yoursite.com/products): What you sell, structured.
- [Pricing](https://yoursite.com/pricing): Numbers, plans, comparison.
## Documentation
- [Getting Started](https://yoursite.com/docs/start): The 5-step onboarding.
- [API Reference](https://yoursite.com/docs/api): Full method index.
Ship it. Stop tuning it. Then spend the rest of the week on schema and answer-first H2 restructuring, which is where the recovery cases are actually being won.
When two independent measurement methodologies across 300,000-plus domains agree that an optimization has no measurable effect on the outcome it is sold to improve, the rational move is to stop selling it as a primary intervention. Treat LLMs.txt as future-proofing insurance with a ten-minute implementation cost. Treat schema, entity binding, and answer-first content structure as the actual lever. The recovery cases that crossed pre-AI Overview revenue did the second set of things. The Search Engine Land-reported audit where 8 of 9 sites saw no measurable change after implementation did the first.
Related on Tygart Media: llms.txt URL curation · llms.txt 2026 spec · verify llms.txt in logs.
Independent studies across approximately 300,000 domains have found no statistically significant relationship between LLMs.txt presence and AI citation frequency. Major AI vendors have not publicly committed to reading the file in production. Implement it as low-cost future-proofing, not as a primary citation strategy.
Documented recovery cases share a consistent intervention pattern: schema markup deployment, content restructuring with answer-first formatting directly under each H2, entity disambiguation, and inline source attribution. One published case showed 118% recovery of pre-AI Overview traffic in 120 days using this stack.
Ten lines: an H1 with your brand name, a blockquote with one-sentence site description, and grouped H2 sections listing your core pages and documentation with one-line summaries. Ship it once, do not over-tune it.
The user agents that drive AI citations include GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended. These are the crawlers whose access determines whether your content surfaces in AI answer panels.
Three reasons: it is genuinely cheap to ship, it signals to clients that you are paying attention to AI search, and there is a non-zero chance AI vendors adopt it in the future. None of those reasons justify it being your primary AI-search intervention in 2026.
Sources: Search Engine Journal’s coverage of the 300,000-domain LLMs.txt citation study; SEO Algorithm Recovery’s documented AI Overviews recovery case study; published bot traffic audits from Authority Tech and Generix Marketing on LLMs.txt request rates; recovery-stack analysis aggregated from BlankBoard Studio, Stackmatix, and Mersel AI’s 2026 AI Overviews recovery compilations.