Tag: API testing

  • We Ran Muse Spark’s Reasoning Dial Through 52 Calls. The Cheapest Setting Never Lost.

    We Ran Muse Spark’s Reasoning Dial Through 52 Calls. The Cheapest Setting Never Lost.

    Every API model ships with a dial most people never touch. On Meta’s Muse Spark 1.3, the dial is called reasoning_effort — it decides how long the model thinks before it answers. Turn it up and you pay more tokens and wait longer, on the theory that more thinking means better answers.

    We spent two days testing that theory on the contributor tier of Muse Spark 1.3 — the same engine our own publishing fleet runs on. Fifty-two API calls, seven different kinds of work, every rung of the dial, everything scored against known ground truth. The result surprised us, and it changes how we run the fleet:

    The cheapest setting never lost. Not once. On hard tasks, the expensive settings mostly bought latency — and twice, extra thinking was the failure itself.

    How we got the dial wrong first

    During a hackathon last month we tried to set the dial with a parameter called reasoning. The API rejected it, and we wrote down “the API won’t let us control reasoning.” That was wrong — the parameter is reasoning_effort, and it works fine. Lesson one of testing a model: a rejection usually means you asked wrong, not that the door is locked.

    We also carried a second error into the test: a belief that the API capped output at 8,000 tokens. It doesn’t. That cap was our own router setting, repeated so often it hardened into “fact.” Generation verified past 15,000 tokens in a single call. Both errors are corrected in our own records now, with the receipts attached — which is the point of running a test like this in the open. The test’s first findings were about us.

    The ladder has six rungs — and one of them doesn’t exist

    The full map, settled by the API’s own error messages when we fed it bad values: minimal, low, medium, high, xhigh, max. That is the complete list.

    Two things people assume about this dial are false. There is no “extra high” setting — xhigh is that rung, and the spelled-out version is rejected. And there is no off switch you can use: none appears in the generic schema, but this model rejects it with a message that lists exactly what it does support. On Muse Spark 1.3 contributor, the model always thinks at least a little. The only question is how much you pay for.

    The rounds

    Every task was run at multiple settings with identical prompts, and scored mechanically — no vibes, no “this answer feels smarter.”

    Contradiction checking (a spec paragraph, 24 statements, name the ones that contradict it): exact answers at every setting. Low used about a third of the reasoning tokens of the default.

    A buyer-brief judgment task — the hardest test we built. Four vendor blurbs against six hard rules (billing terms, export fees, Saturday support, a $7,200 year-one cost cap, HIPAA, cancellation windows), with planted traps: one vendor breaks two rules with numbers you have to multiply out, one hides a disqualifier in its cancellation clause, one is clean only if you notice the free-months clause. Every rung — minimal through max, plus unset — produced the identical score: correct winner, all five real traps caught, zero missed. Minimal used 776 reasoning tokens and 6.5 seconds. Xhigh used 6,847 tokens and 50 seconds. Same answer.

    Agentic planning (schedule an 8-step pipeline with dependencies and resource constraints into parallel waves): provably optimal schedules at every rung. Minimal solved it in 242 reasoning tokens.

    Code debugging (a function with three planted bugs): every rung found the inverted discount logic and the rounding bug; the cheapest setting found all three.

    Verdicts at scale (120 claim records, check each against a two-part rule): 120 out of 120 correct at low — and 120 out of 120 at xhigh. (Our first scorer reported 83/120 at both settings; the bug was in our regex, not the model. We re-ran it with the full text saved, corrected the record, and retracted the number. Test harnesses fail too, and the honest write-up says so.)

    Constrained writing (a briefing paragraph with five required facts, three forbidden claims, and a word band): every rung delivered all five facts, zero forbidden claims, in band. Reasoning ranged from 729 tokens to 6,687. The paragraph didn’t get nine times better. It got nine times slower.

    Where more thinking actually hurt

    Twice in 52 calls, the expensive end of the dial was the failure:

    • Left unset, the model burned 7,578 reasoning tokens on the vendor task — more than any explicit setting — for the same score minimal got for 776. The default thinks hard whether you asked it to or not. Unset is not neutral; it’s the expensive option wearing a disguise.
    • On the debugging task, an xhigh call consumed its entire 8,000-token budget on reasoning — 7,998 reasoning tokens, completion exactly 8,000 — and returned an empty answer. It thought so long it never spoke. Minimal answered the same task correctly with 500 reasoning tokens. Anyone who has watched an agent stall mid-job has seen this shape; here it is with a meter on it.

    And one anomaly worth naming: max is the top rung by name, but on the vendor task it burned less than xhigh (3,773 vs 6,847). The rungs are task-shaped budgets, not fixed ceilings — max buys you permission to think longer, not a guarantee of the deepest burn.

    What it cost

    Fifty-two calls, roughly 31,000 tokens in and 145,000 out across both days: $0 charged. The contributor tier’s billing never touched the $25 test envelope we run it under, and its per-token price remains unpublished as far as we can verify — we won’t invent one. At the standard tier’s published rates ($1.25 per million in, $4.25 per million out), the entire test programme would have cost about 65 cents. The dial still matters at fleet scale: reasoning tokens bill as output, and a fleet that runs xhigh by habit pays roughly nine times the reasoning cost of one that runs low — for, on this evidence, the same answers.

    The operating rule we run now

    • Mechanical work — extraction, checking, verdicts, formatting: minimal or low.
    • Judgment work: low, stepping up to medium when the task deserves a second think.
    • Xhigh and max are earned, not default: a task graduates to them only after it has demonstrably failed at low. In 52 calls across seven kinds of work, no task has earned it yet.
    • Never leave the dial unset. The default is the high band with none of the intent.

    The honest limits of this test

    One sample per cell. Seven task classes, all of them the kind of work a publishing fleet actually does — checking, judging against a brief, planning, debugging, classifying at scale, writing to constraints. We did not test open-ended research, long creative writing, or mathematics, and a task in those families might break the pattern; the battery is built to be re-run the day one appears. Reasoning-token counts also vary between sessions on identical tasks — sometimes by ~30% — so treat the exact numbers as readings, not constants.

    But the shape of the result is hard to argue with: on real operational work, this model’s floor is high, and the dial mostly prices your patience. If your fleet — agent, workflow, whatever you run — has a reasoning setting you’ve never touched, the test is cheap, the scoring is easy to make mechanical, and the answer might save you nine-tenths of your reasoning bill. Run the test. Publish what you find. We just did.

    Method and receipts: 52 calls to muse-spark-1.3-contributor via Meta’s Model API, November 6–7, 2026. Full findings, prompts, ground truth, and raw results: part of the Tygart-OS fact-gated fleet record on GitHub (github.com/TygartMedia/tygart-os-fact-gated-fleet). Errors found in our own earlier assumptions (the reasoning parameter name, the phantom 8,000-token cap, and one scorer bug) are documented in the same record, corrected where they stood.