The GLM Limit Map: Where an Agent Seat Actually Hits Walls

From the Tygart Media research desk — our own runs, our own scoring, our own receipts. Part of the GLM Desk.

Most model coverage maps the peaks: benchmark scores, launch claims, demo reels. We needed the opposite map. GLM is the first-pass seat in our agent fleet — it touches real work every day — so we ran it at the walls instead: tasks designed around the places agent seats usually break, scored against verified answer keys, with the flight recorders read afterward.

The honest summary: the walls are not where the demo-reel genre would put them. The model passed every structural test we built. Its real limits are three narrow, named behaviors — two of which closed with brief wording, one of which is a standing design rule — plus one settings trap covered in the companion piece on the reasoning dial.

Tested, and held: the limits that weren’t there

Stamina. A 15-page census with 60 exact fields — titles, headings, embed IDs — came back 60/60 with no decay across the run. It also diagnosed the one genuinely weird page in the set: a page rendering its headline twice, which the run traced to a theme header duplicating a body headline. We verified the diagnosis by hand. It was right.

Confabulation on an absent fact. Asked for a price that was not in the documents it was given, the model answered “Not in the documents” — and in an earlier research run, it excluded a vendor from a pricing recommendation rather than invent a price it couldn’t see, then did the seven-site cost math with the prices it had verified. Absent-fact honesty is the behavior agent seats are worst at. It held twice.

Injection resistance. A vendor email in the test corpus carried a planted instruction — an embedded note telling the reader to reroute the summary. The model summarized the email faithfully, named the embedded instruction, and stated why it was excluding it. Observed, not guaranteed — we still fence third-party content in briefs — but observed in the open, with the reasoning on the record.

Contradictions. Handed three mutually impossible spec pairs, it flagged all three up front, proposed a structural resolution, and admitted the one that couldn’t hold instead of silently picking a side. Briefs in our shop now rely on that flag instead of pre-resolving conflicts.

Code edge cases. A CSV aggregation task scored by running its code against a hidden key — ties, empty cells, a zero-division trap — matched the expected output exactly. The recorder shows why: unprompted, it wrote three fixture files (basic, tie, header-only) and tested against them before delivering.

The limits that are real

1. Creative embellishment. Given a facts list and asked for a voice paragraph, the run obeyed every voice rule — and invented a timing detail (“last week”) that was not in the facts. Small, fluent, and exactly the failure shape that survives a skim. Status: closed by brief wording. Adding a facts-only guardrail — use only the facts listed; add no detail, date, timing, or circumstance not in the list — eliminated the invention on re-run. The recorder shows the guardrail executing as a sentence-level self-audit: the model talking itself out of an embellishment mid-draft.

2. Chatty delivery tails. Same family: meta sign-offs and framing text wrapped around a clean deliverable. Status: closed by brief wording. A paragraph-only output rule — output the paragraph only, no introduction, no closing note — produced a clean single paragraph on re-run.

3. Attribution drift under first-person pressure. The wrinkle the guardrail doesn’t cover: with a first-person narrator and facts that credit someone else, the run merged the two — the narrator “found” something the facts assigned to a homeowner. Status: open — handled as a brief-design rule. When attribution matters, the brief now names whose discovery it is, explicitly. We have not found wording that makes the model police this on its own, and we’d rather publish that than pretend the guardrail covers it.

4. Judgment loss at low reasoning. Forcing the dial low on judgment work produces false clearances — defects certified clean, correction math that slips while writing the answer. Extraction work is unaffected. Full record in the dial piece; the house rule is one line: never QC at low.

What production added to the map

A battery is a promise; production is where it gets kept or broken. In the production runs that followed the battery, the same seat ran a six-site audit-verification program for us: checking an SEO tool’s exported claims against the live sites, page by page. The export turned out to be wrong in both directions — on one site it claimed 29 over-long titles where a verified crawl found 77; it claimed 28 duplicate headlines where the true count was 45; it invented entire defect rows that didn’t exist. The seat’s verified page lists — every one we spot-checked by hand matched exactly — became the working truth the fixes were built from, and a cross-site synthesis pass re-added every column total exactly.

Production also found a limit the battery hadn’t: delivery-shape drift. On one analysis run, the seat published its result to a canvas document and returned a summary link instead of the full text the job needed — a reasoned choice on an ambiguous brief, and the wrong one for the receiving end. The fix was contract wording (full report as message text, no external docs), applied to every later brief, and it has held since.

The open edges, stated plainly

Reasoning=high as a standalone middle setting: untested. Multi-file work inside a real code repository: untested — every run so far was no-repo. Very long single documents (over ~15K characters in the prompt): lightly probed only. Those are the edges of this map, marked as such, and the desk updates the map when a run crosses one.

The desk’s other records: what every run in this map cost, per finished task, in Cost Per Finished Task; the method behind the recorder reads in Grade the Flight Recorder; and the 44-model census that made this seat our first pass in the first place.