We Asked 44 AI Models to Look at Five Web Pages. Seven Claudes Said the Video Wasn’t There.

On October 16, 2026, we handed 44 AI models the same plain job: visit five live web pages and report, for each one, the exact title, the exact top heading, whether the page contains an embedded YouTube video (and its ID), and how many section headings the page has. Twenty hard fields. We pulled the correct answers from the pages’ raw source before any model ran, and we scored every report against that key.

Thirty-six models scored a perfect 20 out of 20. Eight did not — and seven of the eight were Claude models, all failing the same way: on the only two pages that contained an embedded video, they reported that no video was there. Their heading counts on those same two pages came up short. Everywhere else, they were right.

Which models, exactly

Claude Fable 5.1 (16/20), Claude Opus 5 (16/20), Claude Sonnet 4.5 (16/20), and Claude Sonnet 5.5 (16/20) each missed both embeds and both heading counts. Claude Opus 4.5 and Claude Sonnet 4.6 scored 15/20. Claude Haiku 4.5 scored 14/20. Same shape of error, seven different models.

And on the same day, on the same task: Claude Fable 5, Opus 4.6, Opus 4.8, Opus 5.5, and Sonnet 5 all scored a perfect 20/20. This is not a story about a model family that cannot see. It is a story about what happened on particular runs — and the run records show what it was.

It was the fetch, not the model

Because the agent environment records every tool call a run makes, we could read how each model got its answers. Accuracy tracked one variable, field by field: how the run fetched the page. Runs that pulled the raw HTML and parsed it were exact. Runs that read the page through a built-in extraction tool received a cleaned text version — and in that version, embedded iframes are stripped out entirely. The video did not fail to load. In the copy those models were given, it did not exist.

The sharpest case: Opus 5 and Fable 5.1 both ran an explicit step whose job was to verify the embeds — and still reported none. A verification step only counts if it re-derives the answer from source. When the verifier looks through the same parser that made the error, it certifies the error. We call it verification theater.

What this means if you use agents

Three lessons we now run by:

• An agent’s report describes its fetch, not the world. If the fetch is lossy, the report inherits the losses silently — nothing in the final answer disclosed that the page had been read through a layer that removes embeds.
• Errors cluster by method, not by model. The identical misses across seven models lived one level below the models, in a tooling path they shared.
• Grade the trajectory, not the summary. The final answers alone suggest “these models can’t see videos.” The execution record shows the true cause. Any serious evaluation of an agent should score what it did, step by step, against ground truth fixed before the run.

The full record

The complete white paper — method, the seven models’ exact misses, the full 44-model results table with run times and metered costs, and the trajectory findings — is free to download:

Download the white paper (PDF): The Video That Wasn’t There — a 44-model page census

The websites tested are not named, at our choice. Every model is named, because the models are the subject. Run artifacts are retained and available on request.