Tag: AI Search

  • IndexNow Speed Test: How Fast Do Bing, (2026)

    IndexNow Speed Test: How Fast Do Bing, (2026)

    IndexNow promises instant content discovery. But how fast is it really? We ran a controlled speed test — 40 articles published simultaneously to tygartmedia.com with IndexNow pings fired on every one — then measured exactly how long it took Bing, GPTBot, Google, and every other crawler to show up. The timestamps tell a story that IndexNow’s marketing materials do not.

    This is the second article in Tygart Media’s AI Search Intelligence series, based on proprietary server log data from our 40-article Microsoft Copilot content experiment conducted on June 22, 2026. Every timestamp and crawl interval cited here comes directly from our server access logs.

    What Is IndexNow and Why Speed Matters

    Three cards for Google cautious, Bing speed, OpenAI aggressive crawl styles
    What IndexNow is and why speed matters.

    IndexNow is an open-source protocol that lets websites notify participating search engines the moment content is published or updated. Instead of waiting for a crawler to discover your new page organically — which can take days or weeks — IndexNow sends a direct ping saying “this URL has new content, come get it.”

    Microsoft developed IndexNow and Bing is its primary participant. Yandex, Naver, Seznam, and several other engines also participate. Google does not. As of early 2026, over 60 million websites use IndexNow, and 22% of clicked Bing URLs come from IndexNow submissions, according to Bing’s published data.

    For publishers, the speed question is not academic. If you are publishing time-sensitive content — news, product launches, competitive analysis — the difference between a 3-hour crawl delay and a 3-day crawl delay determines whether your content gets indexed before or after your competitors. And in the AI era, the question extends beyond traditional indexing: how fast do AI crawlers like GPTBot find your new content?

    Our Test Setup: 40 Articles, One Timestamp

    On June 22, 2026, we published 40 original articles about Microsoft Copilot to tygartmedia.com. The site runs WordPress with RankMath SEO on a Google Cloud Platform Compute Engine instance. RankMath handles IndexNow submissions automatically on publish.

    Every article was published within a short window, and IndexNow pings were fired for each URL. We then monitored our raw server access logs for every subsequent crawler visit, recording the user-agent string, timestamp, and requested URL for each hit.

    This gave us a clean dataset: 40 identical test cases (same site, same publish time, same IndexNow submission) with crawler-by-crawler arrival times we could compare head-to-head.

    Head-to-Head Results: Who Arrived First?

    Topic platform fit visual for first-party AI citation measurement
    Head-to-head: who arrived first.

    Bing: 3 to 6 Hours via IndexNow

    Bingbot was the first traditional search engine crawler to reach our content, arriving within 3 to 6 hours of IndexNow submission. The pattern was remarkably consistent across all 40 articles — most fell within a tight 4-hour window from publication to first crawl.

    This is fast by search engine standards but not instant. IndexNow does not trigger immediate crawling. It places your URL into Bing’s priority crawl queue, and Bing processes that queue on its own schedule. For our batch of 40 articles, that schedule produced a 3-to-6-hour window with high consistency.

    For context, without IndexNow, new content on a site with our domain authority profile might wait 24 to 72 hours for Bing to discover it through sitemap parsing or link following. IndexNow compressed that to under 6 hours — a meaningful improvement for any publishing operation.

    GPTBot: Faster Than Bing

    Here is the result that surprised us most: GPTBot arrived at our content faster than Bingbot in many cases, despite GPTBot not being an official IndexNow participant.

    GPTBot is OpenAI’s crawler. It does not receive IndexNow pings directly. Yet it consistently reached our newly published articles before Bing’s own crawler had finished processing the IndexNow queue. At 11:00 UTC on June 22, GPTBot executed a 1,123-request structural crawl in a single hour, hitting not just article URLs but every tag, feed, and REST API endpoint on the site (Tygart Media server log analysis, June 2026).

    How does GPTBot discover content faster than IndexNow delivers it to Bing? The most likely explanation is that GPTBot monitors RSS feeds, sitemaps, or other real-time content signals independently. WordPress sites broadcast new content through multiple channels — RSS feeds update instantly, XML sitemaps regenerate on publish, and REST API endpoints reflect new posts immediately. GPTBot appears to be monitoring one or more of these channels with higher polling frequency than Bing’s IndexNow processing queue.

    The implication for publishers is significant: even if you do not use IndexNow, GPTBot is likely to find your new content quickly through other discovery mechanisms. But IndexNow remains essential for Bing-ecosystem discovery, which feeds Microsoft Copilot’s citation pipeline.

    YandexBot: 30 Seconds Behind Bing

    YandexBot arrived at each article approximately 30 seconds after Bingbot, with remarkable consistency across the full batch. Yandex participates in the IndexNow protocol, and this timing suggests Yandex processes IndexNow submissions from the same shared queue but with a slight processing delay relative to Bing (Tygart Media server log analysis, June 2026).

    The 30-second shadow is too consistent to be coincidental. It points to either a shared IndexNow notification infrastructure where Yandex processes submissions fractionally behind Bing, or to Yandex monitoring Bing’s crawl activity directly. Either way, publishers who submit to IndexNow get both Bing and Yandex coverage from a single ping.

    Googlebot: Effectively Absent

    Googlebot recorded only 1 hit on our Copilot content in the initial crawl window (Tygart Media server log analysis, June 2026). One hit. Across 40 articles. While Bing had crawled every article within 6 hours and GPTBot had mapped the entire site architecture.

    Google does not participate in IndexNow. Google has stated publicly that it relies on its own crawl scheduling, which considers factors like site crawl budget, historical update frequency, and sitemap change signals. For a batch of 40 new articles on a topic the site had not previously covered, Google’s algorithms apparently did not prioritize rapid discovery.

    This is not a criticism of Google’s approach — its crawl scheduling optimizes for different goals than real-time discovery. But for publishers who need content indexed quickly, the data is unambiguous: IndexNow-participating engines discover content in hours. Google discovers it on its own timeline.

    The IndexNow Technical Gotcha We Discovered

    During our experiment, we identified a technical issue that could affect other publishers: the IndexNow key file was returning a 404 at the standard verification paths where search engines expect to find it.

    IndexNow requires a verification key file at your site root (e.g., yourdomain.com/{key}.txt). Search engines check this file to confirm you authorized the IndexNow submission. In our case, the key file was not accessible at the expected root-level path, which should have caused verification failures.

    RankMath SEO’s fallback mechanism saved us — it handles IndexNow key verification through an alternative method that does not require the physical key file to exist at the root URL. But publishers using manual IndexNow implementations, or other SEO plugins without this fallback, should verify their key file is accessible by navigating directly to the expected URL.

    If your IndexNow submissions seem to be ignored by Bing, check the key file first. A 404 on the verification file silently kills the entire pipeline — Bing will not crawl the submitted URLs without successful verification.

    What the Speed Test Means for Your Publishing Strategy

    Desk with laptop, checklist notebook, and billing card ready before creating an Anthropic API key
    What the speed test means for publishing strategy.

    For Bing and Copilot Visibility

    IndexNow is the fastest path to Bing’s index, and Bing’s index feeds Microsoft Copilot’s citation system. Our 40-article experiment earned 3 confirmed Copilot citation referrals within 48 hours, and that pipeline started with IndexNow getting our content into Bing’s index within hours of publication.

    If you are publishing content that you want Copilot to cite, IndexNow is not optional — it is the first link in the citation chain.

    For AI Crawler Discovery

    GPTBot does not use IndexNow, but it finds new content fast anyway — faster than Bing in our test. This means your site’s real-time content signals (RSS feeds, sitemaps, REST API endpoints) are the discovery mechanism for OpenAI’s crawler ecosystem. Keep these endpoints clean, accessible, and unblocked in your robots.txt if you want AI systems to discover your content quickly.

    For Google

    Google’s crawl scheduling operates independently of IndexNow. If rapid Google indexing is important to you, continue submitting sitemaps through Google Search Console and requesting indexing for priority pages through the URL Inspection tool. Do not rely on IndexNow for Google discovery — the protocol has no effect on Google’s crawl behavior based on our data.

    For Multi-Engine Strategy

    The practical recommendation is to run both systems in parallel: IndexNow for Bing, Yandex, and the downstream AI systems that rely on Bing’s index, plus Google Search Console for Google’s independent crawl pipeline. Most WordPress SEO plugins handle IndexNow automatically, so the incremental effort is near zero.

    The Speed Hierarchy: From Fastest to Slowest

    Based on our server log data from the 40-article experiment, here is the definitive crawl speed ranking for newly published, IndexNow-submitted content (Tygart Media server log analysis, June 2026):

    1. GPTBot — fastest overall; arrived before IndexNow results in many cases; 1,123-request structural crawl in one hour
    2. ChatGPT-User — 3,404 hits over 48 hours; activates when real users query ChatGPT about relevant topics
    3. Bingbot — 3 to 6 hours via IndexNow; consistent, predictable timing
    4. YandexBot — ~30 seconds behind Bingbot; piggybacks on IndexNow shared infrastructure
    5. OAI-SearchBot — 3 hits total; minimal presence; appears highly selective
    6. AzureAI-SearchBot — 3 hits total; minimal presence
    7. Googlebot — 1 hit in initial window; operates on its own schedule independent of IndexNow

    The gap between the top of this list and the bottom is not hours — it is the difference between same-day discovery and multi-day (or longer) discovery. For publishers who need content discovered quickly, the AI crawlers and IndexNow-participating engines are delivering results that Google’s independent crawl schedule simply does not match.

    A Note on Methodology and Reproducibility

    Every crawl timestamp and interval cited in this article comes from raw server access logs on Tygart Media’s Google Cloud Platform Compute Engine instance, analyzed in June 2026. Crawler identification was performed by user-agent string matching, with IP range verification against OpenAI’s and Microsoft’s published crawler IP ranges for additional confirmation.

    The 40-article batch was published simultaneously to control for timing variables. All articles were submitted via IndexNow through RankMath SEO’s automatic submission feature. No manual crawl requests were submitted through Google Search Console, Bing Webmaster Tools, or any other interface — we wanted to measure organic and IndexNow-driven discovery only.

    This experiment is reproducible. Any publisher running a WordPress site with IndexNow enabled can monitor their server access logs after a batch publish and observe the same crawler patterns. The specific timing intervals may vary based on domain authority, server location, and crawl budget allocation, but the relative ordering — GPTBot fastest, Bing via IndexNow in hours, Google on its own schedule — should hold across most publishing environments.

    For the complete dataset including all crawler hit counts and the full methodology, see our anchor article: We Published 40 Articles and Watched Every AI Crawler in Real Time.

    Frequently Asked Questions

    How fast does IndexNow actually get content crawled by Bing?

    In our controlled test of 40 simultaneously published articles, IndexNow submissions resulted in first Bingbot crawls within 3 to 6 hours, with most articles falling in a consistent 4-hour window. This is significantly faster than the 24-to-72-hour organic discovery timeline for sites without IndexNow, but it is not instant — Bing queues IndexNow submissions and processes them on its own crawl schedule (Tygart Media server log analysis, June 2026).

    Does GPTBot use IndexNow to discover content?

    No. GPTBot is not an IndexNow participant, yet it arrived at our content faster than Bingbot in many cases. GPTBot appears to monitor RSS feeds, XML sitemaps, or REST API endpoints independently, giving it a faster discovery pipeline than Bing’s IndexNow processing queue. In our experiment, GPTBot executed a 1,123-request structural crawl at 11:00 UTC, mapping the entire site architecture within a single hour (Tygart Media server log analysis, June 2026).

    Does Google support IndexNow?

    No. Google does not participate in the IndexNow protocol as of June 2026. In our experiment, Googlebot recorded only 1 hit on our 40-article batch while Bingbot and GPTBot had fully crawled the content. Google relies on its own crawl scheduling algorithms and recommends using Google Search Console’s sitemap submission and URL Inspection tool for prioritized crawling (Tygart Media server log analysis, June 2026).

    Why was YandexBot always 30 seconds behind Bingbot?

    YandexBot, as an IndexNow participant, appears to process submissions from a shared notification infrastructure with a slight delay relative to Bing. The consistent 30-second gap across all 40 articles suggests either a shared queue processed fractionally behind Bing or direct monitoring of Bing’s crawl activity. The practical result is that a single IndexNow ping delivers both Bing and Yandex crawls almost simultaneously (Tygart Media server log analysis, June 2026).

    What should publishers do if IndexNow submissions are being ignored by Bing?

    Check your IndexNow key file first. The key file must be accessible at your domain root (e.g., yourdomain.com/{key}.txt). In our experiment, the key file was returning a 404 at standard paths, which would have silently killed the pipeline. Our RankMath SEO plugin’s fallback mechanism handled verification, but publishers using manual implementations should navigate directly to their key file URL to confirm it returns a 200 response (Tygart Media server log analysis, June 2026).

  • llms-full.txt vs llms.txt: Why AI Agents Crawl It More (2026)

    llms-full.txt vs llms.txt: Why AI Agents Crawl It More (2026)

    Last verified: October 11, 2026. AI crawler user-agents and llms.txt conventions shift quickly — always confirm bot names and file conventions against the current provider docs and your own server logs before relying on them.

    Most conversations about AI crawlability focus on one file: llms.txt. But if you look at what Anthropic, Vercel, and LangGraph actually ship – and what GEO crawler research found AI agents fetching most – the file that matters more is its companion: llms-full.txt.

    Here’s the practical reality: llms.txt is the map. llms-full.txt is the territory. And in 2026, the agents that matter for citation traffic are fetching the territory.

    The Full File Family You Probably Don’t Know About

    The original llms.txt proposal – published by Jeremy Howard in September 2024 – defined one file. Implementers built the rest. The complete family as of mid-2026 is four files, but most sites only need two:

    FileWhat’s in itWhen to use
    /llms.txtCurated index – H1, summary, link sectionsAlways. The orientation layer.
    /llms-full.txtFull content of every linked page, concatenated as MarkdownWhen you want a model to deep-ingest your docs in a single fetch
    /llms-ctx.txtPre-expanded context without URLsFastHTML-style implementations
    /llms-ctx-full.txtPre-expanded context with URLs preservedSame, but URL-aware

    The pattern that works – and the one Anthropic, Vercel, and LangGraph all run – is the index + export pair: llms.txt for orientation, llms-full.txt for deep ingestion.

    Why llms-full.txt Gets Crawled More

    Four ranked rows of AI crawler fleets reading publisher content
    Why llms-full.txt gets crawled more.

    GEO researchers analyzing AI crawler behavior – including work cited by Profound – have noted that agents from Microsoft, OpenAI, and others tend to fetch llms-full.txt more frequently than llms.txt when both are present. The working explanation is structural: when a file contains the full content, it removes one retrieval step. An agent that fetches llms-full.txt gets everything it needs in a single HTTP request instead of fetching the index, parsing the links, then fetching each linked page individually. This is consistent with how developer documentation platforms like Mintlify describe the behavior of IDE agents operating under tight latency budgets.

    For IDE agents (Cursor, Continue, Cline) and MCP integrations, this is even more pronounced. These tools are operating under tight context windows and latency budgets. A single fetch that returns a clean Markdown blob of your entire docs is structurally preferable to a multi-step crawl.

    The implication: if you’ve shipped llms.txt but not llms-full.txt, you’ve done half the job.

    How to Build llms-full.txt

    Desk with laptop, checklist notebook, and billing card ready before creating an Anthropic API key
    How to build llms-full.txt.

    The construction logic is simple: take every URL in your llms.txt, fetch each page, strip HTML to Markdown, and concatenate. In practice, most sites do this in their build pipeline.

    Here’s the minimal Node.js pattern:

    const fs = require('fs');
const fetch = require('node-fetch');
const TurndownService = require('turndown');
const turndown = new TurndownService();

async function buildLlmsFullTxt(llmsIndexPath, outputPath) {
  const index = fs.readFileSync(llmsIndexPath, 'utf8');
  const urlRegex = /\[.*?\]\((https?:\/\/[^\)]+)\)/g;
  const urls = [...index.matchAll(urlRegex)].map(m => m[1]);

  let output = '';
  for (const url of urls) {
    const res = await fetch(url);
    const html = await res.text();
    const markdown = turndown.turndown(html);
    output += \n\n---\n# Source: \n\n;
  }

  fs.writeFileSync(outputPath, output);
  console.log(Built llms-full.txt:  pages,  chars);
}

buildLlmsFullTxt('./public/llms.txt', './public/llms-full.txt');

    One constraint to manage: keep llms-full.txt under roughly 200,000 tokens (about 150K words, around 700KB). That’s the threshold where most models can ingest the file in a single context window. If your docs are larger, segment by product or language the way Supabase does – llms-full-api.txt, llms-full-guides.txt – and list the segmented files in your main llms.txt.

    The 2026 robots.txt Stack That Completes the Picture

    Three cards for Google cautious, Bing speed, OpenAI aggressive crawl styles
    The 2026 robots.txt stack that completes the picture.

    Shipping llms.txt and llms-full.txt is the visibility layer. The access-control layer is robots.txt – and it changed significantly in Q2 2026.

    The key development: Anthropic split its crawler into two separate user-agents. ClaudeBot is the training scraper (high bandwidth, no citation value – block it). Claude-Web is the live-retrieval agent that fetches pages to answer Claude.ai user queries in real time (allow it, because it drives citation traffic). Brands that blanket-block “all Anthropic crawlers” lose Claude citations entirely.

    Meta also shipped two active training scrapers in March 2026 – FacebookBot and Meta-ExternalAgent – at GPTBot-level crawl volume. Most sites have no rules for them yet.

    Here’s the 2026 template:

    # BLOCK: Training scrapers - high bandwidth, zero referral value
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# OPT OUT: Google Gemini training (keeps Search indexing intact)
User-agent: Google-Extended
Disallow: /

# ALLOW: Live-retrieval agents - drive citation traffic
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

    One important caveat on robots.txt enforcement: aggressive training scrapers often ignore the file or spoof their user-agents. The robots.txt rules signal intent and work for compliant bots; a WAF rule at the edge is the only deterministic block for non-compliant crawlers.

    The Honest State of the Technology

    The SERanking study of 300,000 domains (November 2025) found no measurable correlation between having llms.txt and being cited by ChatGPT, Claude, Gemini, or Perplexity. Google’s John Mueller compared the file to the deprecated keywords meta tag – something site owners declare but that search systems derive from the content itself.

    None of that means you shouldn’t ship both files. The cost is low, the optionality is real, and the IDE-agent ecosystem (Cursor, Continue, Cline) does actively use llms.txt. But the robots.txt work is the lever that moves outcomes today. The llms.txt + llms-full.txt pair is infrastructure investment – you want to be correct when major LLM providers start honoring it, and building the build pipeline now costs far less than retrofitting it later.

    The practical sequence for a site that hasn’t done this yet:

    1. Update robots.txt first. Add the Q2 2026 user-agent rules above. This takes twenty minutes and immediately affects how training scrapers treat your content.
    2. Ship llms.txt. Curated index, 20-50 priority pages, one-sentence description per link, sections in priority order.
    3. Build llms-full.txt. Concatenated Markdown of every linked page, under 200K tokens. Run it in your build pipeline so it stays current.
    4. Verify both files are served correctly. curl -I https://yoursite.com/llms.txt should return 200 with Content-Type: text/plain. A 404 on either file is the most common implementation error.
    5. Add an access-log check. Once per month, grep your logs for requests to /llms.txt and /llms-full.txt by user-agent. You want to see live-retrieval agents (Claude-Web, OAI-SearchBot, PerplexityBot) in the results – not just training scrapers.

    The goal isn’t to optimize for a standard that isn’t fully adopted yet. It’s to build the infrastructure correctly now, while the field is still forming, so that adoption changes work in your favor rather than requiring catch-up.

    Related Reading

    Frequently Asked Questions

    What is the difference between llms.txt and llms-full.txt?

    llms.txt is a curated index — an H1, a summary, and link sections that orient an AI agent to your site. llms-full.txt is the full content of every linked page concatenated as Markdown, so an agent can deep-ingest your documentation in a single fetch. The index is the map; the full file is the territory.

    Why do AI agents crawl llms-full.txt more often than llms.txt?

    Fetching llms-full.txt removes a retrieval step: the agent gets everything in one HTTP request instead of fetching the index, parsing links, and fetching each page individually. For IDE agents like Cursor, Continue, and Cline operating under tight latency and context budgets, a single clean Markdown blob is structurally preferable to a multi-step crawl.

    How big should llms-full.txt be?

    Keep it under roughly 200,000 tokens (about 150K words, around 700KB) so most models can ingest it in a single context window. If your docs are larger, segment by product or language — for example llms-full-api.txt and llms-full-guides.txt — and list the segmented files in your main llms.txt.

    Does having llms.txt actually improve AI citations?

    Not measurably on its own. A November 2025 SERanking study of 300,000 domains found no correlation between having llms.txt and being cited by ChatGPT, Claude, Gemini, or Perplexity, and Google’s John Mueller compared it to the deprecated keywords meta tag. The lever that moves outcomes today is robots.txt configuration; llms.txt and llms-full.txt are low-cost infrastructure for when adoption grows.

    Which AI crawlers should I allow in robots.txt in 2026?

    Allow live-retrieval agents that drive citation traffic — Claude-Web, OAI-SearchBot, ChatGPT-User, anthropic-ai, and PerplexityBot. Block high-bandwidth training scrapers with no referral value such as GPTBot, CCBot, ClaudeBot, FacebookBot, and Meta-ExternalAgent, and opt out of Google-Extended to skip Gemini training while keeping Search indexing intact.

  • How AI Engines Actually Cite Your Content: Grounding and GEO Guide

    How AI Engines Actually Cite Your Content: Grounding and GEO Guide

    Last verified: June 2026.

    Most “GEO” advice is recycled SEO with the word “AI” pasted on top. This guide is different. It describes what actually happens when Microsoft Copilot, Bing’s AI answers, and Google’s AI Overviews build a response and decide whose page to cite — based on running content sites that get cited tens of thousands of times a month. The short version: AI engines do not cite the page that ranks #1 for a head term. They cite the page that most directly answers the specific sub-question the model is grounding on. That distinction changes everything about what you should write.

    How grounding actually works (the part nobody explains)

    Topic platform fit visual for first-party AI citation measurement
    How grounding actually works.

    When you ask Copilot or Bing’s AI a question, the model does not answer from memory. It runs a retrieval step called grounding: it rewrites your question into one or more search queries, fetches a handful of live web results, reads them, and composes an answer with inline citations pointing back at the pages it used. Google’s AI Overviews work the same way with a technique it calls “query fan-out” — one user question becomes many narrower synthetic queries.

    Two things follow directly from this mechanism:

    • The model is not searching for your keyword. It is searching for the answer to a decomposed sub-question. A user who asks “what’s the best way to instantly index a new page” triggers grounding queries like “IndexNow API endpoint”, “submit URL to Bing programmatically”, and “IndexNow key file location”. The page that wins is the one that answers those narrow strings, not the one optimized for “indexing tips”.
    • Citations are extracted at the passage level, not the page level. The model lifts the specific sentence or table that answers the sub-question. If your answer is buried under 600 words of preamble, it loses to a page that states the fact in the first line under a matching heading.

    This is why a niche, specific page routinely out-cites a high-authority generalist. The generalist ranks; the specialist gets quoted.

    Why operational and comparison pages win over head terms

    Across real citation data, the pages that get pulled into AI answers cluster into three shapes. None of them are “ultimate guide to X”.

    1. Operational pages with real commands, configs, and error messages

    When someone asks an AI assistant “how do I fix [specific error]” or “what’s the exact command to do X”, the model needs a page that contains the literal command, the literal config, or the literal error string. Generic advice cannot be cited because there is nothing concrete to quote. A page that says:

    curl "https://www.bing.com/indexnow?url=https://example.com/new-page/&key=YOUR_KEY"
# 200 = received (not "indexed"), 422 = URL/key mismatch, 429 = too many submits

    …is citation gold, because the model can extract that block verbatim and the user can act on it. The error-code annotations matter: questions about failures (“IndexNow 422”, “why am I getting 429”) are high-intent and low-competition, and a page that names the exact codes owns them.

    2. Comparison pages (“X vs Y”)

    “Which is better, X or Y” is one of the most common shapes of AI query, and comparison content is structurally easy to cite because it maps cleanly to a decision. If you maintain honest, current head-to-head pages, you become the default source the model reaches for when a user is choosing between tools. This is exactly why we keep dedicated comparison pages like Claude Code vs Cursor and Claude Code vs Codex — they answer a decision the model is constantly being asked to make, and a table of differences is trivially quotable.

    3. Fresh, dated pages on fast-moving topics

    For anything that changes — pricing, model versions, API limits, feature availability — grounding strongly favors recency. The model would rather cite a page dated this month than an “authoritative” page from two years ago that might be wrong. A visible “Last verified” date and a real publish/update timestamp are not decoration; they are a relevance signal the retrieval layer reads.

    The losing move is chasing broad head terms. “Best AI coding assistant” is saturated, generic, and rarely the literal grounding query. The winning move is to own the long, specific, operational and comparison strings that the fan-out actually generates.

    IndexNow: how to get cited the same day you publish

    Four-stage funnel: citation, click, engage, convert
    IndexNow — cited the same day you publish.

    Grounding can only cite pages the engine knows about. The bottleneck for new content is crawl latency — and IndexNow collapses it. IndexNow is an open protocol (backed by Microsoft Bing and Yandex) that lets you push a URL to the index the instant you publish, instead of waiting for a crawler to wander by.

    Setup is two steps:

    1. Host a key file. Generate a key of 8-128 hex characters and place it at your site root as a UTF-8 text file named {key}.txt containing exactly that key. Example: https://example.com/daa44a2c....txt. This proves you own the host.
    2. Ping on publish. Single URL via GET: 
      curl "https://api.indexnow.org/indexnow?url=https://example.com/new-page/&key=YOUR_KEY"
      Or batch up to 10,000 URLs in one POST: 
      curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json" \
  -d '{"host":"example.com","key":"YOUR_KEY","urlList":["https://example.com/a/","https://example.com/b/"]}'

    A 200 means the endpoint received your URL (not that it is indexed yet). Submitting to api.indexnow.org shares the ping with all participating engines, so you do not need to hit Bing and Yandex separately. Most WordPress SEO plugins (Rank Math, Yoast, SEOPress) have IndexNow built in — turn it on and it fires automatically on every publish and update. The practical payoff: pages can enter Bing’s crawl queue within hours, which means they are eligible to be grounded and cited the same day, not next week.

    One caveat worth stating plainly: IndexNow accelerates indexing, which is a precondition for citation. It does not force a citation. You still need the page to be the best answer to the sub-question. But for fresh, time-sensitive content, same-day indexing is often the difference between getting cited while the topic is hot and showing up after the conversation has moved on.

    How to actually measure your AI citations

    For a long time AI citations were invisible — you could see referral clicks in analytics but not the citations themselves (most AI answers are zero-click). That changed. As of February 2026, Bing Webmaster Tools ships an AI Performance report (public preview) that shows when your pages are cited across Microsoft Copilot, Bing’s AI answers, and partner surfaces. It is the first direct, free window into AI citation behavior, and you should be reading it weekly.

    The four metrics that matter:

    • Total citations — how many times your site was cited as a source in AI answers over the period.
    • Average cited pages — the daily average count of unique URLs from your site that got referenced. This tells you whether citations are concentrated on one page or spread across the site.
    • Grounding queries — sample query phrases the AI used to retrieve and cite you. This is the single most actionable field in the report. It is a literal list of the sub-questions you are winning, which tells you exactly which operational/comparison angles to expand next.
    • Page-level citation activity — citations by URL, so you can see which pages are doing the work.

    Two limitations to keep in mind so you read the data honestly: the report does not show click data (you see citations, not visits from them), and it aggregates Copilot with Bing summaries, so you cannot isolate one surface from the other. For Google’s AI Overviews there is still no equivalent citation dashboard — the closest proxy is watching impressions and referral patterns in GA4 and Search Console, plus spot-checking your target queries by hand.

    The workflow that works: pull the grounding-queries list, find the patterns, and feed them straight back into your content plan. If you are getting cited for “claude mcp setup” variants, that is a signal to deepen pages like the Claude MCP setup guide and adjacent operational walkthroughs, not to chase a new head term.

    A repeatable checklist for citation-optimized pages

    Desk with laptop, checklist notebook, and billing card ready before creating an Anthropic API key
    Checklist for citation-optimized pages.

    Everything above reduces to a build pattern. For any page you want AI engines to cite:

    • Lead with the answer. Put a short, factual, quotable answer in the first 1-2 sentences under each heading. Assume the model reads only that passage.
    • Use question-shaped headings. H2s and H3s that mirror real queries (“How does IndexNow work?”, “How do I measure AI citations?”) match the grounding query and give the extractor a clean anchor.
    • Be specific and operational. Real commands, real config, real numbers, real error codes and fixes. Concrete text is extractable; vague advice is not.
    • Add a visible FAQ near the end. Plain question/answer pairs are the single most citation-friendly format, because each pair is a self-contained answer to a discrete sub-question. You do not need JSON-LD schema for this to work — visible Q&A text is what the model reads.
    • Date it and keep it current. A “Last verified” line plus genuine updates on fast-moving topics buys you the recency edge in grounding.
    • Push it with IndexNow so it is indexable the same day, then watch the AI Performance report to see which sub-questions it wins.

    If you want the larger system this fits into — the full toolchain for operating as an AI-first publisher, from MCP servers to publishing pipelines — start with the AI operator’s stack.

    FAQ

    Do AI engines cite the page that ranks #1 on Google?

    Not reliably. AI engines run their own grounding retrieval and cite the page that most directly answers the specific decomposed sub-question, which is often a niche, operational page rather than the head-term winner. Ranking helps your page be discoverable, but the citation goes to whichever passage best answers the exact grounding query.

    What is grounding in AI search?

    Grounding is the retrieval step where an AI assistant rewrites your question into search queries, fetches live web pages, reads them, and builds an answer with inline citations to those pages. It is why current, specific pages can get cited even by a model whose training data predates them.

    Does IndexNow guarantee my page will be cited by AI?

    No. IndexNow guarantees fast indexing, which is a precondition for being cited. The page still has to be the best, most specific answer to the sub-question the model is grounding on. Think of IndexNow as removing the crawl-latency excuse, not as buying a citation.

    How do I measure how often AI cites my site?

    Use the AI Performance report in Bing Webmaster Tools (public preview since February 2026). It shows total citations, average cited pages per day, sample grounding queries, and citation counts by URL across Microsoft Copilot and Bing AI answers. It does not yet show click-through from those citations, and there is no equivalent dashboard for Google AI Overviews.

    Do I need JSON-LD or schema markup to get cited?

    No. Citation extraction works on visible, well-structured text — question-shaped headings, short factual answers, and a plain visible FAQ. Schema can help search features generally, but it is not required for AI grounding to read and quote your page.

    What kind of pages get cited most?

    Three shapes dominate: operational pages with real commands, configs, and error fixes; comparison pages that resolve a “X vs Y” decision; and fresh, dated pages on fast-moving topics like pricing and model versions. Broad head-term content tends to get skipped because it rarely matches the literal grounding query and offers nothing concrete to quote.

    Related on Tygart Media: citation economy · AI search funnel · citation monitoring.

  • AEO Intent Classification: The Four-Query Framework

    AEO Intent Classification: The Four-Query Framework

    Related on Tygart Media: GEO tactics · AI search funnel · SEO vs GEO vs AEO.

  • RAG Optimization: Creating Source-Worthy Content for AI

    RAG Optimization: Creating Source-Worthy Content for AI

    The Search Landscape of May 2026: Stop Chasing Traffic, Start Chasing Citations

    The transition is complete. As of this month, Google’s AI Overviews (formerly SGE) appear for over 52% of all search queries. If you are looking at your Search Console and seeing a 30% drop in informational traffic compared to last year, you aren’t alone. You’re simply seeing the result of the “Zero-Click” era reaching its final form. For digital agency owners and systems architects, the old SEO playbook is a liability. If you are still optimizing for clicks on “What is…” or “How to…” keywords, you are effectively donating your intellectual property to train a model that will replace your visit.

    The currency of search has shifted. We have moved from the era of link equity to the era of Source-Worthy Content. In this new reality, the goal isn’t to get the user to click through to read a basic definition; it is to ensure that your data, your unique perspective, or your proprietary methodology is the primary source cited by the Retrieval-Augmented Generation (RAG) systems powering Google, Perplexity, and OpenAI.

    The Numbers Don’t Lie: The Death of the Click

    By mid-2026, the data across our portfolio is clear. Informational query traffic—the top-of-funnel “educational” content that used to drive massive awareness—has cratered by 20-40% across most B2B and technical sectors. Users are getting their answers directly in the search interface. They don’t need to visit your site to learn “how to configure a headless CMS” if Gemini can pull the five essential steps from your documentation and present them in a neat bulleted list.

    However, while traffic is down, the value of a single citation within an AI Overview has skyrocketed. We’ve found that being the primary citation in a RAG-driven answer drives higher-intent leads than the old-school organic #1 spot ever did. The users who do click through from an AI Overview have already been pre-qualified by the AI. They aren’t looking for a definition; they are looking for the operator who provided the insight. Optimizing for AI overviews is no longer a side project; it is the core of technical SEO.

    Understanding RAG: How Google Picks Its Sources

    To win in 2026, you have to understand the mechanics of Retrieval-Augmented Generation. Google’s AI isn’t just “hallucinating” answers based on its training data; it is actively searching the live web, retrieving specific “chunks” of information, and then synthesizing those chunks into a response. This is RAG optimization.

    When an AI Overview is generated, Google’s system follows a three-step process:

    1. Retrieval: It identifies the top-ranking traditional search results for the query. (This is why maintaining traditional page-one rankings is still a prerequisite for being a source).
    2. Selection: It selects specific paragraphs, data tables, or unique insights from those top results that best satisfy the user’s intent.
    3. Generation: It rewrites those insights into a cohesive answer, adding citations to the sources it used.

    If your content is generic—if it says exactly what every other site says—the AI will synthesize the answer without citing you specifically, or it will cite a larger authority (like Wikipedia or a massive news outlet) that says the same thing. To be cited, your content must be source-worthy. It must provide something the AI cannot find elsewhere or synthesize from common knowledge.

    Why Generic Content is Erased by AI

    The era of “skyscraper” content—taking ten existing articles and making a longer one—is over. AI is better at that than you are. In fact, most of that generic content is now being flagged by LLMs as “low information gain.”

    When we audit a site using the Gemini CLI, we look for “Information Gain” scores. If a paragraph doesn’t offer a new data point, a specific case study result, or a unique operator’s perspective, it’s invisible to the RAG process. Generic advice like “SEO requires good keywords” is discarded. Specific advice like “We saw a 12% lift in RAG citations by moving from 1,000-word articles to 400-word modular content blocks” is source-worthy.

    The LLM wants to cite the originator. If you are just a curator, you are a middleman that the AI has successfully bypassed.

    The ‘Source-Worthy’ SEO Framework

    At Tygart Media, we’ve pivoted our Agency Playbook to focus on four pillars of source-worthy SEO. This is how we ensure our clients remain the “source of truth” in an AI-dominated search engine.

    1. Proprietary Data and “Proof of Work”

    The AI cannot hallucinate your internal data (yet). Original surveys, technical benchmarks, and project post-mortems are the most cited pieces of content in 2026. If you run a test on a new deployment pipeline and publish the raw numbers, Google’s AI Overview will cite your specific numbers. We’ve moved away from “opinion pieces” and toward “experiment logs.” Every article should contain at least one table or chart of data that didn’t exist on the internet before you published it.

    2. The Operator’s Perspective (E-E-A-T)

    Experience and Expertise are now the primary filters for RAG selection. Google is prioritizing content that shows “Proof of Effort.” Use first-person accounts. Instead of writing “How to use Claude Code,” write “What we learned after 500 hours using Claude Code to refactor a legacy Python monolith.” The specific failures and technical hurdles you describe are unique identifiers that the AI recognizes as authoritative.

    3. Modular Content Architecture

    Long-form, sprawling articles are difficult for RAG systems to “chunk” effectively. We are now building content in modular blocks. Each section of an article is designed to stand alone as a complete answer to a sub-query. We use <section> tags and specific ID attributes to make it easy for the crawler to identify and retrieve the exact block it needs. This is optimizing for AI overviews by making your content “consumable” for machines, not just humans.

    4. Structured Data for RAG

    Schema.org hasn’t gone away; it has become the metadata for AI. We use Dataset, HowTo, and Review schema more aggressively than ever. But more importantly, we are using Gemini CLI to auto-generate JSON-LD that specifically maps out the “Claims” made in our articles. By explicitly stating “Our claim: Informational traffic is down 30%,” we make it easier for the AI to attribute that fact to us.

    Technical Execution: Modular E-E-A-T and Gemini CLI

    The workflow for a modern agency operator involves high-level automation. We don’t manually audit 500 pages for “source-worthiness.” We use tools like Claude Code and Gemini CLI to process our content libraries.

    Our current stack for RAG optimization looks like this:

    • Analysis: We pipe our top-performing URLs through a script that uses the Gemini API to compare our content against the current AI Overview for that keyword. The script identifies “content gaps”—information the AI is providing that isn’t on our page, or information we have that the AI is ignoring.
    • Refactoring: If a page is losing traffic but has high “Source Worthiness,” we use Claude Code to refactor the HTML into a more modular structure, adding Dataset schema to any tables.
    • Validation: we use Antigravity to simulate how a RAG system would “chunk” the page. If the chunks are incoherent, we rewrite the headers to be more explicit.

    One failure we saw early in 2026 was attempting to “game” the AI by over-optimizing for specific keywords. The AI sees through keyword density. It is looking for semantic weight. When we tried to force-feed keywords, our RAG citation rate dropped. When we focused on “operator-restrained” technical clarity, the citations returned.

    Case Study: The 40% Traffic Drop and the 15% Lead Increase

    We recently worked with a systems architecture firm that saw their organic traffic from “cloud migration tips” fall by 40% in the google sge impact may 2026 rollout. Initially, there was panic. However, upon closer inspection, their “Request a Consultation” conversions were actually up by 15%.

    What happened? Their generic “tips” were being swallowed by the AI Overview. But the AI Overview was citing their specific “Cloud Migration Cost Calculator” and their “2025 Migration Failure Report.” The traffic they lost was the “looky-loos” who just wanted a quick tip. The traffic they gained (via the AI citations) was from CTOs who saw their specific data cited as the authority and clicked through to hire them. This is the shift from “volume” to “value.”

    Action Plan: What You’d Do Tomorrow

    If you are managing a content library or an agency portfolio, don’t wait for your traffic to hit zero. Start the pivot to source-worthy SEO immediately. Here is the operator’s checklist for tomorrow morning:

    1. Audit for “What is” Content: Use your preferred crawler to identify every page that targets a purely informational, definitional keyword. These are your “donor” pages. Decide whether to delete them, consolidate them, or upgrade them with proprietary data.
    2. Inject Original Data: Find three pieces of internal data—even if they are small—and add them to your top 10 most important pages. Use tables. Add a “Methodology” section.
    3. Modularize Your Headers: Ensure every H3 in your articles can stand alone as a question and every following paragraph as a direct, concise answer. Remove the “fluff” and the “introductory transitions.” The AI doesn’t need a “In this section, we will explore…” lead-in. It needs the facts.
    4. Verify Citations: Perform a manual search for your primary keywords. Look at the AI Overview. If you are ranking #1-3 in organic but aren’t cited in the AI response, your content isn’t “Source-Worthy.” It’s too generic. Rewrite the top-ranking paragraph to offer a unique, data-backed perspective that the AI is currently missing.
    5. Update Your Schema: Move beyond basic Article schema. Implement Speakable, Dataset, and ClaimReview schema where applicable. Use a tool like Gemini CLI to automate the generation of these blocks based on your existing text.

    SEO isn’t dead; the middleman is dead. The search engine of 2026 doesn’t want to send users to a website; it wants to provide an answer. Your job is to be the only source that the answer cannot exist without. Build for the machine, provide for the human, and protect your intellectual property by making it too specific to be ignored.

    Related on Tygart Media: chunk-first GEO · GEO tactics · how AI engines cite.

  • ChatGPT Search Citations: The 2026 Optimization Guide

    ChatGPT Search Citations: The 2026 Optimization Guide

    ChatGPT Search cites 15% of the pages it retrieves. The other 85% get pulled into the model’s context window, evaluated, and silently discarded — no visibility, no referral, no trace. If you are doing GEO work and your pages keep getting retrieved but never quoted, you are losing at the second filter, not the first.

    This is the 2026 implementation guide for surviving both filters: getting retrieved by ChatGPT Search, then getting cited once you are there.

    How ChatGPT Search Actually Builds an Answer

    Topic platform fit visual for first-party AI citation measurement
    How ChatGPT Search builds an answer.

    ChatGPT Search runs a three-stage pipeline. Each stage kills most candidates.

    1. Retrieval — ChatGPT Search is powered by Bing’s index for real-time web retrieval. Seer Interactive’s analysis found 87% of SearchGPT citations match Bing’s top results, with the bulk in positions one through ten and a long tail in positions eleven through twenty. AirOps research separately put ChatGPT-to-Bing overlap at 73%. If you are not in Bing’s top 20 for a query, you almost certainly are not in ChatGPT’s candidate set.
    2. Crawlability check — OpenAI’s OAI-SearchBot is the user agent that builds the index used for ChatGPT’s search features. It is separate from GPTBot (training) and ChatGPT-User (browsing). Block OAI-SearchBot in robots.txt and you remove yourself from ChatGPT Search entirely, even if Bing has you ranked.
    3. Citation selection — Of the pages retrieved, AirOps found ChatGPT cites only 15%. The model picks what to quote based on structure, freshness, authority signals, and whether the page directly answers the query.

    Step 1: Verify You Are Indexed by Bing

    Most sites optimized for Google have never logged into Bing Webmaster Tools. Fix that first. Three checks before anything else:

    • site:yourdomain.com in Bing — confirms basic indexing.
    • Bing Webmaster Tools → URL Inspection — confirms the specific pages you want cited are indexed and have no crawl errors.
    • Bing rankings for your target queries — if you are not in the top 20 in Bing, ChatGPT will not see you.

    If pages are missing, submit a sitemap via Bing Webmaster Tools and request URL inspection on any priority page. Bing typically reflects changes within 24–72 hours, faster than Google.

    Step 2: Allow OAI-SearchBot in robots.txt

    Four ranked rows of AI crawler fleets reading publisher content
    Allow OAI-SearchBot in robots.txt.

    The single most-skipped step in GEO work. Add this block to your robots.txt:

    # Allow ChatGPT Search to retrieve and cite this site
User-agent: OAI-SearchBot
Allow: /

# Optional: allow on-demand browsing for ChatGPT users
User-agent: ChatGPT-User
Allow: /

# Optional: block training crawler if you want retrieval without training
User-agent: GPTBot
Disallow: /

    OpenAI publishes these three user agents and treats each independently. You can allow OAI-SearchBot for ChatGPT Search visibility and still disallow GPTBot from using your content for model training. The settings do not conflict. OpenAI’s systems typically recognize robots.txt changes within 24 hours.

    Step 3: Structure Pages for the Citation Filter

    Comparison of Claude how-to fit versus local service page fit for assistants
    Structure pages for the citation filter.

    Retrieval is necessary but not sufficient. Once your page is in the candidate set, the model decides whether to quote it. Pages that get quoted share a structural pattern.

    Direct answers in the first 100 words

    ChatGPT cites sources that answer the question fully. Partial answers lose to complete ones. Lead each page with a clean direct-answer paragraph: question implied or stated, answer in the next sentence, supporting detail after. This is the same pattern that wins featured snippets, which is not a coincidence — answer engines and snippet engines reward the same structure.

    JSON-LD schema

    An AirOps study of 548,534 pages found pages with JSON-LD markup posted a 38.5% citation rate versus 32.0% without it. Article, FAQPage, and HowTo schema are the highest-leverage types. Add them.

    Word count: 500–2,000

    Pages between 500 and 2,000 words performed best in the same AirOps study. Pages longer than 5,000 words were cited less often than pages under 500. The mechanism is mechanical: long pages overflow the retrieval context window, and the model defaults to shorter, denser sources it can quote in full.

    Freshness

    Content updated within 30 days received 3.2x more citations than older material. The fix is not faked freshness — it is genuine updates: a new stat, a new case, a corrected claim. Update the date when you update the content, not before.

    Step 4: Build the Authority Layer

    Structure gets you cited once. Authority gets you cited repeatedly. AirOps found sites with over 32,000 referring domains are 3.5x more likely to be cited by ChatGPT than sites with fewer than 200. You do not need 32,000 — you need to be in the upper band of your topical neighborhood.

    ChatGPT’s citation pattern leans heavily on Wikipedia (roughly 48% of top citations in multiple studies) and large news/media properties. The practitioner read on that: ChatGPT favors sources with multi-source third-party validation. Build the kind of citations on the open web that Wikipedia editors accept — peer-reviewed studies, primary sources, named author attribution, transparent methodology.

    Step 5: Track Your Citation Footprint

    You cannot manage what you do not measure. The minimum tracking stack for 2026:

    • Server log monitoring for OAI-SearchBot user agent — confirms OpenAI is actually crawling. If you allowed the bot in robots.txt three weeks ago and there are zero OAI-SearchBot hits in your logs, something is wrong (CDN block, IP firewall, misconfigured allow rule).
    • Manual citation audits — pick 10 priority queries, run them in ChatGPT with the Search toggle on, log which domains get cited. Repeat weekly. A spreadsheet beats no tracking.
    • Bing position tracking — because ChatGPT pulls from the Bing index, Bing rankings are a leading indicator. If your Bing position drops, ChatGPT visibility drops behind it.

    The Practitioner Summary

    Ranking in ChatGPT in 2026 is not mysterious. It is a four-gate funnel: Bing index → OAI-SearchBot crawl access → retrieval into the candidate set → citation selection. Most sites fail at gate one (not indexed in Bing) or gate two (OAI-SearchBot blocked or not addressed). Sites that clear those two gates and write pages that answer the question fully, with schema and a 500–2,000-word range, will land in the 15% that get quoted.

    Treat ChatGPT Search like a separate search engine that happens to share an index with Bing. Optimize for the index. Allow the crawler. Write the page. The rest follows.

    Related on Tygart Media: AI citation monitoring · AI search funnel · how AI engines cite.