Grade the Flight Recorder, Not the Answer

From the Tygart Media research desk — our own instrument, our own data, our own receipts. Part of the GLM Desk.

Here is a failure you cannot see by reading answers. In one evaluation we scored, a major AI surface produced the longest answer of the five we compared — over 6,600 characters, company names, confident detail. Its execution record showed the whole answer was built on one sub-query and zero sources actually consulted. A second surface’s entire evidence base, beneath an equally confident answer, was a single maps link. Graded on the answer text, both look like research. Graded on the record of what they did, both are writing from memory with good posture.

That gap — between the closing pitch and the work underneath it — is why this shop grades trajectories. This page is the method, published the way we publish our settings: steal it.

The instrument

Our citation-measurement rig records more than final answers. For every chat it stores the execution trajectory: the sub-queries the surface issued while building its answer (in the order it issued them), the sources it actually consulted, and the call record — how many calls, what status. We built a scorer over that record. It grades the execution, not the summary, on four checks:

X1 — Retrieval happened. Sub-queries issued or sources consulted. If neither exists, the verdict is FROM-MEMORY, whatever the answer claims.

X2 — Intent match. What share of the sub-queries actually carry the question’s subject and place terms — did the surface go looking for what was asked, or for something adjacent and easier? This check is scored only where the telemetry exists. Some providers expose no sub-query record at all; there, the honest mark is NOT-SCORED, never a fail.

X3 — Evidence breadth. Distinct source hosts actually consulted: five or more is BROAD, two to four ADEQUATE, one THIN, zero NONE.

X4 — Execution health. The run completed and made at least one call. A surface that returned no answer at all is EXCLUDED from the denominators — per our standing convention, a non-answer is not a miss, it’s an absence — and is never failed.

The verdicts — SOUND, THIN, FROM-MEMORY, EXCLUDED — are then compared against the old outcome read (was the company cited in the answer?). Where the two reads disagree in kind, the scorer raises a DELTA flag. Those flagged runs are the ones worth a human’s eyes: the uncited-but-sound answers and, more dangerous, the cited-but-hollow ones.

What it caught that answer-grading couldn’t

The two cases above opened this piece. The census work caught the same failure family from the other side of the glass. In our 44-model census, one flagship model opened its run with twenty-two steps that were echoes — “fetching page one,” “starting fetch” — announcements of tool calls it never made, before it eventually fetched for real. From the final answer alone you would never know the first half of the run was theater. Two top-end models ran explicit “verify the embeds” steps while holding the raw page source — and still reported no embeds, because their checking step used the same blind fetch tool as their fetching step. A verifier that shares the original blind spot is not a verifier, and only the step record shows it.

On the GLM seat itself, trajectory reads are how this desk’s other findings were earned: the low-dial failure mechanism (one lens per item, deliberation stopping before the writing did) is a recorder observation, not a score inference — and so is the good news, like a code run that wrote and ran its own test fixtures before delivering, unprompted.

The honest limits

Telemetry is partial, and the method says so on its face. In the evaluation store this scorer runs against, sub-query records exist for about a fifth of the chats — 22 of 110 at adoption. Where they don’t exist, X2 is not scored and the verdict rests on the checks that remain; a run is never failed for its provider’s silence. The dashboard record on the agent side has the same texture: reasoning text is fully visible on tool-using runs, but some pure-text runs show thought duration only, and one run’s reasoning would not expand at all. The recorder is excellent. It is not total. Any read from it should state its basis — ours do.

The rule it produced

One line, now standing doctrine in our shop since October 16, 2026: grade the execution steps, not the summary. Every model we seat gets its recorder read, not just its answer scored. Every evaluation we publish says which read it stands on — and on the evaluation where the two reads first disagreed, we switched the standing read to the trajectory score and said so in the record. If you run agents, or buy answers from surfaces that do, the closing pitch is the last place the truth lives. The flight recorder is the first.

The method at work: the GLM limit map and the reasoning-dial test are both trajectory-graded records; the desk’s hub keeps the full shelf.