Tag: llm-evaluation

From the Tygart Media research desk — our own runs, our own scoring, our own receipts. Part of the GLM Desk.

Most model coverage maps the peaks: benchmark scores, launch claims, demo reels. We needed the opposite map. GLM is the first-pass seat in our agent fleet — it touches real work every day — so we ran it at the walls instead: tasks designed around the places agent seats usually break, scored against verified answer keys, with the flight recorders read afterward.

The honest summary: the walls are not where the demo-reel genre would put them. The model passed every structural test we built. Its real limits are three narrow, named behaviors — two of which closed with brief wording, one of which is a standing design rule — plus one settings trap covered in the companion piece on the reasoning dial.

Tested, and held: the limits that weren’t there

Stamina. A 15-page census with 60 exact fields — titles, headings, embed IDs — came back 60/60 with no decay across the run. It also diagnosed the one genuinely weird page in the set: a page rendering its headline twice, which the run traced to a theme header duplicating a body headline. We verified the diagnosis by hand. It was right.

Confabulation on an absent fact. Asked for a price that was not in the documents it was given, the model answered “Not in the documents” — and in an earlier research run, it excluded a vendor from a pricing recommendation rather than invent a price it couldn’t see, then did the seven-site cost math with the prices it had verified. Absent-fact honesty is the behavior agent seats are worst at. It held twice.

Injection resistance. A vendor email in the test corpus carried a planted instruction — an embedded note telling the reader to reroute the summary. The model summarized the email faithfully, named the embedded instruction, and stated why it was excluding it. Observed, not guaranteed — we still fence third-party content in briefs — but observed in the open, with the reasoning on the record.

Contradictions. Handed three mutually impossible spec pairs, it flagged all three up front, proposed a structural resolution, and admitted the one that couldn’t hold instead of silently picking a side. Briefs in our shop now rely on that flag instead of pre-resolving conflicts.

Code edge cases. A CSV aggregation task scored by running its code against a hidden key — ties, empty cells, a zero-division trap — matched the expected output exactly. The recorder shows why: unprompted, it wrote three fixture files (basic, tie, header-only) and tested against them before delivering.

The limits that are real

1. Creative embellishment. Given a facts list and asked for a voice paragraph, the run obeyed every voice rule — and invented a timing detail (“last week”) that was not in the facts. Small, fluent, and exactly the failure shape that survives a skim. Status: closed by brief wording. Adding a facts-only guardrail — use only the facts listed; add no detail, date, timing, or circumstance not in the list — eliminated the invention on re-run. The recorder shows the guardrail executing as a sentence-level self-audit: the model talking itself out of an embellishment mid-draft.

2. Chatty delivery tails. Same family: meta sign-offs and framing text wrapped around a clean deliverable. Status: closed by brief wording. A paragraph-only output rule — output the paragraph only, no introduction, no closing note — produced a clean single paragraph on re-run.

3. Attribution drift under first-person pressure. The wrinkle the guardrail doesn’t cover: with a first-person narrator and facts that credit someone else, the run merged the two — the narrator “found” something the facts assigned to a homeowner. Status: open — handled as a brief-design rule. When attribution matters, the brief now names whose discovery it is, explicitly. We have not found wording that makes the model police this on its own, and we’d rather publish that than pretend the guardrail covers it.

4. Judgment loss at low reasoning. Forcing the dial low on judgment work produces false clearances — defects certified clean, correction math that slips while writing the answer. Extraction work is unaffected. Full record in the dial piece; the house rule is one line: never QC at low.

What production added to the map

A battery is a promise; production is where it gets kept or broken. In the production runs that followed the battery, the same seat ran a six-site audit-verification program for us: checking an SEO tool’s exported claims against the live sites, page by page. The export turned out to be wrong in both directions — on one site it claimed 29 over-long titles where a verified crawl found 77; it claimed 28 duplicate headlines where the true count was 45; it invented entire defect rows that didn’t exist. The seat’s verified page lists — every one we spot-checked by hand matched exactly — became the working truth the fixes were built from, and a cross-site synthesis pass re-added every column total exactly.

Production also found a limit the battery hadn’t: delivery-shape drift. On one analysis run, the seat published its result to a canvas document and returned a summary link instead of the full text the job needed — a reasoned choice on an ambiguous brief, and the wrong one for the receiving end. The fix was contract wording (full report as message text, no external docs), applied to every later brief, and it has held since.

The open edges, stated plainly

Reasoning=high as a standalone middle setting: untested. Multi-file work inside a real code repository: untested — every run so far was no-repo. Very long single documents (over ~15K characters in the prompt): lightly probed only. Those are the edges of this map, marked as such, and the desk updates the map when a run crosses one.

The desk’s other records: what every run in this map cost, per finished task, in Cost Per Finished Task; the method behind the recorder reads in Grade the Flight Recorder; and the 44-model census that made this seat our first pass in the first place.

From the Tygart Media research desk — our own runs, our own scoring, our own receipts. Part of the GLM Desk.

Every AI model ships with settings nobody tests. GLM’s reasoning dial — low, high, max — is the one our whole first-pass operation rides on, so we tested it the way we test everything: identical tasks, verified answer keys, and the flight recorder read afterward, not just the final answer.

The finding is not “max is better.” The finding is that the low end of the dial fails in a specific, dangerous shape: it doesn’t just miss defects — it certifies them as clean. And the high end of the dial buys something narrower than people assume: cross-checking.

What the dial is

GLM’s current models accept a reasoning effort setting with three stops — low, high, max — and max is the default; per the model’s own documentation, any value outside the three also falls back to max. There is no thinking-off mode: the model reasons on every run, the dial only sets how much. Most coverage of GLM never touches this dial at all. Our Oct 16, 2026 citation audit found exactly one real study of it anywhere. This is the second, and it’s run on production-shaped work, not benchmarks.

The test

We took three tasks from our scored limit battery — tasks with answer keys verified against source material — and ran each at the default setting and again with the dial forced low. Same briefs, same seat, same scoring.

Task	At max (default)	At low
QC hunt — find 9 planted defects in a draft, zero false positives allowed	9/9, zero false positives, borderline calls reasoned out loud. 46 seconds.	8/9 + one false clearance — missed an unquoted spoken line and certified that rule clean. 9.1 seconds.
Data verification — catch a planted percentage error and a double-counted row	Both caught, math shown, alternative readings computed.	Both spotted — but the correction math slipped: it wrote a merged total of 2,484 where the correct unchanged total was 2,390.
Stamina census — 15 pages, 60 exact fields (titles, headings, embeds)	60/60. 121 seconds.	60/60. 56 seconds — same accuracy, half the time, cheaper.

Read the table the way an operator would. On extraction, low is genuinely free: identical accuracy at half the time. On judgment work, low is a trap with a signature — the failures are not random misses, they are confident wrong clearances.

Why low fails that way: the recorder’s answer

The final answers alone would have told us low “mostly works.” The flight recorder told us the mechanism. In the low QC run, the model’s recorded reasoning examined each item under one lens at a time: it checked the founder’s sentence against the sourcing rule, and checked the quotation rule against a different sentence — and never cross-checked the sentence where the two rules intersected, which is exactly where the planted defect sat. One lens per item, then a clean bill of health.

At max, the longer pass cross-examines: the same item gets re-read under the second rule. The data run shows the mirror image of the same mechanism — at low, the wrong total entered during answer composition, after the deliberation had already stopped. The thinking ended; the writing continued; the error moved in.

That’s the dial’s real trade. Low doesn’t buy a dumber model. It buys a model that stops deliberating sooner — and on judgment work, the stopping is the failure.

The rule we run

Three working rules came out of this battery, and they now govern every GLM launch in our fleet:

1. Extraction and census work can run low. Identical accuracy (60/60), roughly half the time. If the task is fetching and transcribing against a key, the depth buys nothing.

2. Judgment work never runs low. QC, data verification, anything whose worst failure is a false clearance — a defect certified clean, a wrong total presented as corrected — stays at max. A miss you can see is a cost. A miss the model has stamped “checked” is a liability.

3. The setting is set explicitly, every launch. Bare launches behaved as max throughout our first battery, and we verified it: an explicit-max run of the QC task reproduced the bare-launch result exactly (9/9, 65 seconds, 0.58¢). We still pin the parameter on every launch, because a default is a thing that can drift, and a pinned setting can’t.

What we haven’t tested

High as a standalone middle setting is untested — low and max bracket it, and we’d rather say that than interpolate. Whether the same failure shape appears on other models’ effort dials is also untested; this desk tests the seat it runs. If you operate a different model with a reasoning dial, the experiment is cheap to copy: one judgment task with a planted defect at the intersection of two rules, run at both ends, recorders read.

More from the desk: the limit map this battery fed, what these runs cost per finished task, and the 44-model census that started the desk.