The Operator Is the Harness

Somebody I follow posted a testing result this week that most people scrolled past. His benchmark harness plants defects in a codebase, lets an agent fix them, and then checks something subtler than whether the defect went away: whether the agent broke something else on the way through. A changed defect signature, he pointed out, is not proof of a correct fix. The bug can change shape and still be a bug. What catches that isn’t a better model. It’s the harness — the checks around the work.

He’s right, and he’s also only halfway to the point. The harness matters. But who built the harness? Who decided what gets checked, what gets automated, and what never gets automated at all? That’s not the model. That’s the operator.

I run a small fleet of websites and a stable of AI agents, and I test models the way you’d test trucks: same load, same road, see what happens. This month I ran one identical production brief through 44 different models — flagships, mid-tiers, cheap ones, open ones. Five rounds of comparisons. The results rearranged everybody’s assumptions, including mine. A mid-priced model I’d never heard anyone brag about did the best first-pass work on the board, at a fraction of the flagship cost. Models with famous names flopped on judgment calls. The leaderboard the internet argues about had almost nothing to do with the leaderboard in my shop.

But here’s the part that actually matters: the brief was the same for all 44. Same instructions, same constraints, same definition of done. The spread in results wasn’t really a spread in models. It was a spread in how much each model needed the operator to have thought clearly. The ones that did well did well because the work was set up so that doing it well was the path of least resistance — the mission was written down completely, the boundaries were structural instead of hopeful, and the verification at the end didn’t care how confident the answer sounded.

I grade my agents the way that benchmark grades code: not by the summary they hand me, but by the steps they actually took. I built a scorer that reads the execution record — what the agent went and looked at, how many sources it actually consulted, whether the answer was built on evidence or generated from memory with good posture. It has caught polished, lengthy, completely confident answers that were built on zero consulted sources. Zero. The answer read beautifully. It was worthless. The model didn’t fail there; the operation almost did, because a summary is the easiest thing in the world to fake and I was nearly set up to accept summaries.

That’s the work nobody posts about. Deciding what the machine is allowed to do alone and what it is never allowed to do alone. In my shop, agents draft, research, extract, and stage. They do not touch money, they do not send anything with a commitment in it, and nothing publishes on an agent’s say-so — there’s a verification pass that reads the actual result at the receiving end, because “the API said 200” has lied to me more than any model ever has. Those rules aren’t prompts. Prompts are suggestions. The rules live in the structure: separate accounts, staged drafts, checkpoints that halt the batch when a check fails instead of improvising past it.

My dad had a saying: a locked door keeps an honest man honest. You don’t make work reliable by telling the worker “don’t mess up.” You make it reliable by building the room so the mess-up is hard to reach.

So when people argue about which model is best, I hear a question about drivers asked by people watching cars. If you’re a good driver, you can drive anything. Give me a mid-tier model, a cheap model, a local model running on a beat-up laptop — with a clear brief, structural boundaries, and honest verification, I’ll out-produce a careless operator running the flagship. Not because the model doesn’t matter at all. It matters the way a truck matters: payload, turning radius, fuel. But the truck has never once decided where the load needed to go.

The model is rented. It changes every few weeks and the prices change faster. The operation — the judgment about what to automate, the discipline of verifying at the receiving end, the memory of every way it has gone wrong before — that’s the asset, and it compounds. Everybody I know is shopping for a better engine. The race is being won by the people building better drivers.