From the Tygart Media research desk — our own test, our own sites, our own receipts.
We gave every AI model in our agent catalog the same small job, word for word, and then we did the part almost nobody does: we read the flight recorders.
The job was deliberately boring. Fetch five pages from our own websites. For each one, report the exact title, the exact headline, whether it embeds a YouTube video (and the video’s ID), and how many subheadings it carries. Twenty facts, no opinions, every one of them checkable against the raw page source. Forty-four models took the seat. Forty-three finished.
Thirty-one came back perfect — 20 out of 20. That is not the interesting part. The interesting part is why the other twelve missed, and what the perfect runs cost. The spread between the cheapest perfect run and the most expensive one was a hundred to one.
The law the test uncovered: accuracy follows the fetch, not the brain
Every miss in the test traced back to one decision: how the model chose to fetch the page.
Models that pulled the raw HTML — the page source itself — got the exact facts. Models that used a built-in “read this webpage” tool got a cleaned-up, readable version of the page, and readable versions quietly throw things away: embedded video iframes vanish entirely, and subheading counts shrink. Same model, same page, different fetch — different score. Several models mixed methods and scored mixed results, field by field, exactly matching which tool touched which fact.
The clearest proof was a family cluster. Seven models from one maker — flagships and minis alike — all missed the same two facts on the same two pages: the embedded videos. They shared a fetching tool, so they shared its blind spot. It was never a thinking failure. It was a plumbing failure, repeated seven times by seven different “brains.”
Three things the flight recorders showed
The cheapest perfect run chose its method out loud. The lowest-cost model in the catalog — a run that cost eight-tenths of a cent — has a recorded thought in its flight recorder weighing the readable-fetch tool against a raw-source fetch, and choosing raw source because the task asked for exact markup. One pass, a local parse, a confirmation check, done. Perfect score.
One model narrated work instead of doing it. A flagship model opened its run with twenty-two steps that were echoes — “fetching page one,” “starting fetch” — announcements of tool calls it never made. It eventually fetched for real and scored 17 of 20. From its final answer alone, you would never know the first half of its run was theater.
Verification can be theater too. Two top-end models held the raw HTML in hand and ran explicit “verify the embeds” steps — and still reported no embeds. Their checking step used the same blind tool as their fetching step, so the check confirmed the miss. A verifier that shares the original blind spot is not a verifier.
The receipts
Identical brief, identical scoring. Among the perfect 20-of-20 runs: the cheapest cost 0.83¢ and the most expensive 86.3¢. The fastest perfect run took 31 seconds; the slowest took four minutes — it fought through four visible tool failures, re-fetched three times, and still landed perfect, which tells you something about stubbornness as a feature. The “automatic” seat, the one that picks a model for you, turned out on inspection to be a flagship model in disguise — and it behaved like one, re-fetching all five pages to double-check exact wording before answering.
What we changed because of this
Three working rules came out of one afternoon of testing, and they now run our shop:
1. Grade the trajectory, not the summary. A fluent final answer can sit on one search and zero sources. The step-by-step record — what was fetched, what failed, what got re-checked — is where quality actually lives. Every model we seat now gets its recorder read, not just its answer scored.
2. Assign work by observed process, not by reputation. The catalog is a crew, not a leaderboard. The model that double-checks markup gets the exactness work. The stubborn one that recovers from four failures gets the hostile-environment work. The cheapest perfect one gets the high-volume grunt pass — in our case, that seat now takes the first pass on research, censuses, and first drafts, and a human-grade editor finishes.
3. Publish your settings. Photographers share f-stops. Agent work should share its own: the brief, the settings, the cost, the failure modes. Ours are below — steal them. A result you can’t reproduce is a rumor.
The settings box
Seat: cloud agents, no code repository, one identical written brief per model. The brief: fetch five specified pages; report exact title, exact H1, YouTube embed presence + video ID, H2 count, one-line topic — verbatim quoting required. Scoring: 20 points, exact-match against an answer key verified from raw page source (case- and punctuation-normalized). Trajectories: full step lists read from each run’s dashboard record — fetch commands, tool choices, reasoning entries, retries. Date: October 16, 2026. Costs: the platform’s own per-run charges, as recorded. Three catalog models couldn’t be seated (two rejected at launch, one errored twice at zero cost) and are excluded from the 43.
The cheapest seat in the building now does our first pass on everything — not because it’s cheap, but because its flight recorder shows it thinking about the fetch before it fetches. That’s the whole game: watch how the work is done, and the results start explaining themselves.
Leave a Reply