Only the harness
changes.

The suite was built to find which harness solves more. For a long time it could not answer that, because nearly everything passed. Three rounds of harder tasks changed nothing. Adding more harnesses finally produced a difference. The more useful question turned out to be simpler: what each harness costs before the model does anything.

Measured

Fixed overhead varies several-fold. Before a single instruction is read, the lightest harness sends characters of tool schema and system prompt on every call. The heaviest sends .

Measured

The gap does not close on bigger work. Cheapest to dearest is on the smallest tasks and on the longest, so the premium is closer to a multiplier than a one-off toll.

Measured

Most of the tool surface goes unused. Averaged over the run, the heaviest harness sends tool definitions per attempt and makes calls.

Measured

Solve rate separates slightly. Most harnesses pass nearly everything. One of them fails more often. Treat that as a note about that harness.

Harness overhead is charged on every model call. A longer task means more calls, not cheaper ones. A lighter harness stays cheaper as work grows.

Each attempt runs one harness against one task in a disposable home directory, with a fresh copy of the task's workspace and the model pointed at a logging proxy. The proxy records every request and response, so token counts and tool calls are observed directly. A task checker then grades the resulting workspace.

• One model throughout. Every harness runs the same model, so a difference in the table is a difference in how the harness uses it.
• One price list, frozen. Costs are recomputed from recorded usage against a dated price file, so a provider's pricing change cannot silently rewrite an old result.
• Graded, not pass or fail. Each task returns a score from 0 to 1, because a boolean verdict throws away the difference between a near miss and no attempt.
• Tiers 1 to 3 are scored on effort as well as correctness. Those tasks are small. Across 182 attempts every score was 0 or 1, so the score alone showed almost nothing. What varied was the work to get there: 2 tool calls on one attempt, 9 on another for the same result. The score is correctness times an effort term. The reference is the fewest calls any harness used on a passing attempt. A wrong answer scores zero. Beating the reference caps at 1.000.
• Scope is checked separately. Every workspace is content-hashed before and after. A harness that edits files the task did not put in scope is recorded as out of scope whether or not it passed.

This is the figure the suite was built for. For most of its life it showed almost nothing. Every harness passed almost everything. It is still worth publishing. The flat result is itself a finding, and it has finally started to move.

Three rounds of harder tasks were written to force a difference. The puzzles looked hard: a wire format inferred from fixtures, an interlocking cache policy, a log far larger than the context window. Every harness solved them on the first or second turn. A capable model solves that kind of problem on its own. The surrounding harness barely matters there. Harder puzzles did not help.

Adding harnesses did. One of the six now fails a clear share of even the smallest tasks, and it does so across tiers. That says something about that harness on this model. The others stay bunched near the top, where this metric cannot tell them apart.

Every model call carries the harness's tool definitions and system instructions. That payload is fixed, it is sent again on each request, and it is the clearest difference between the rows.

The gap is almost entirely tool schema, not prose. A harness that ships twenty-five tools pays for twenty-five tool descriptions on every request, whether or not the task could use any of them. That cost is multiplied by the number of turns and reduced, but not removed, by prompt caching.

You might expect a fixed per-call cost to matter less on large tasks. The per-tier spend shows otherwise. A longer task means more calls, and the overhead is charged on each one. The extra cost grows with the work instead of fading into it.

Cost and the parts that make it up

Cost alone does not say much. The parts below are kept separate because each answers a different question. Tool calls and turns show how much work it took to reach a result the checker accepted. Tool errors and repeated calls show whether the harness recovered or kept repeating the failure. Cache hit rate and context growth show what it carries on every request after that. Fresh tokens drive most of the bill: about 94% of prompt tokens here are cache reads priced at a fiftieth of fresh input. What a harness adds to the prefix each turn matters more than the request total.

Every harness here was measured through the logging proxy, so context figures cover all of them. A harness measured natively would report its own token counts and skip the proxy. Its context growth would show as not measured. Repeated calls count a tool run again with identical arguments after it already answered. Compactions are the harness dropping context to make room. Only one harness here needed to.

Why the failures failed

A pass rate shows how often a harness finished. It does not show what went wrong in the rest: a timeout, a wrong solution, or a failing call repeated until the budget ran out. Each points to a different fix. The table below groups every failed attempt in this run, using a typed classifier over data each attempt already recorded: checker notes, exit code, stderr tail, duration, the most repeated tool, and which injected faults were met.

Every verdict went through the same confidence gate a live session would use. Answers below that bar are published as ungated instead of being forced into the nearest category. More than half landed there. A report file records what a harness did, not what it meant to do. For most failures that is not enough to name the cause. That gap is why --keep-transcripts exists: it saves the per-request shape of an attempt beside its report, so classification can improve without re-running anything. Ungated means the classifier did not commit. Unknown means it committed to "cannot tell from what was recorded".

Every figure is relative to the cheapest harness in that tier. Read across a row and most stay close to where they started: a harness costing twice the lightest on smoke tasks still costs about twice as much on the long one. The ratios shift a little. Nothing converges.

Two rows move more. Both get more expensive on the long task. That fits the task: a harness that takes more turns pays its per-call overhead more times there.

The proxy counts tool definitions sent and tool calls actually made. They differ by a lot, in every harness.

Every harness here offers far more than it uses, which is expected. A tool has to be offered before the model can decide it is not needed. What matters is the ratio, because the unused definitions are paid for on every call. This measurement is the reason the accounting was rewritten mid-run: an earlier version of the proxy counted offers and labelled them calls, which made the busiest harness look like the most productive one.

Everything above is this repository grading its own suite, and a suite cannot mark its own homework. So the same harnesses were run against Terminal-Bench 2.0 — somebody else's tasks, written without reference to anything here. Seven tasks, four attempts each, one model, through the same endpoint. Pi with no harness at all is included as the floor, because SpecPi, SpecPi + Jev and Oh My Pi are all Pi underneath, and without it there is nothing to measure what they add against.

Solve rate separates here in a way it never did on the tier suite: Codex passes 26 of 28 and the DeepSeek Harness passes 19. But the finding worth the section is the one that survives the small sample, and it is about overhead rather than skill.

* Cache hit and cost come from a separate clean-session run of the same harnesses, six attempts each; the callout below says why. Every other column is the full run, twenty-eight attempts per harness.

The harnesses agree almost exactly on how much work a task takes. Tool calls per attempt run from 15.6 to 20.0 across the six that can be counted — a spread of under a third, on tasks where prompt tokens differ by nearly three to one. Whatever separates these rows, it is not that one harness does more.

What differs is the size of each call. Dividing the requests out leaves the context a harness spends to ask one question, and that is where the rows come apart: the heaviest sends more than twice what the lightest does, on every call, for the same number of calls.

The tool mix says where the difference comes from. Codex routes nearly everything through a single command tool; Oh My Pi spreads the same work across five, and pays for all five on every call.

Per task, as solved over attempts and mean prompt tokens. One task, polyglot-c-py, is the only one that defeats the Pi family outright — Pi, SpecPi and SpecPi + Jev all go 0 for 4 where Oh My Pi goes 3 for 4. That is the clearest capability signal in the run, and it points the opposite way from the overhead columns.

One caveat belongs next to the numbers before anything else: three of these arms ran in an earlier sitting than the other four, under different machine load.

The SpecPi + Jev row spends 26% less context than SpecPi at an identical solve rate. That reads like the advisor working, and it is not: the layer's own ledger records 464 calls and zero applied across the arm. Nothing it decided ever changed what happened, so the token difference has a cause this run does not identify.

That is a weaker result than it sounds, because solve rate was never the thing the layer was built to move. Jev is eight systems, and each targets a situation rather than a score — a session long enough to accumulate dead tool results, a command that should not run unexamined, a loop that has stopped making progress, fetched text addressing the agent rather than the reader. A benchmark made of short, single-session, non-adversarial tasks does not contain those situations, so it cannot answer whether handling them helps.

Read down the calls column and the run describes itself. Five of the eight systems were never invoked at all. compaction is the clearest case: it guides the cut point when a session compacts, and no attempt in this run ever compacted — the largest single prompt anywhere in the arm was 72,117 tokens against a 200,000-token window. sources ranks files for a delegation snapshot, and nothing delegated; untrusted watches fetched content, and these tasks fetch almost nothing.

Of the three that did run, guard accounts for 431 of the 464 calls. That row measured the short-lived native command guard; gating has since returned to the pinned specpi-jev-guard package, which these homes do not install, so the numbers are kept as recorded rather than restated. It examined every shell and file call that local rules could not settle, and escalated none of them. On ordinary development work that is the outcome you want from a command guard — 431 chances to interrupt work that did not need interrupting, taken none of them — but it is a measurement of restraint, not of protection. This run contained nothing genuinely destructive for it to catch. One detail makes the claim narrower still: the guard's middle band asks a human, and headless there is no human, so it degrades to deferring. Only an outright block was reachable here, and the ask path went untested.

retention was asked 18 times and declined every time, all 18 at relevance-low-confidence — it would not drop a tool result it was not sure was dead, which is the safe direction to fail but means it shortened nothing. progress was asked 15 times and found the session healthy in 12.

The machine-readable record is terminal-bench.json. Benchmark task content is deliberately absent from this repository: Terminal-Bench ships canary strings that are not meant to enter a training corpus, so only aggregates and task names cross over.

The section above ends by saying that measuring an advisory layer over long sessions would need a suite built to contain those cases. Tier 6 is that suite, and the useful part of building it was finding out that the obvious approach does not work.

Three versions of it failed. The first was batchable — every task collapsed into a shell one‑liner. The second was greppable: one grep with four alternations returned the answer set exactly, on both tasks, with no false positives, while the prompt asserting that no keyword separated them was simply false. The third fixed both and still measured nothing, because a 254,000‑token corpus against a 200,000‑token window never reached the model at all. An agent with a shell does not load a large corpus. It chunks it, greps it, or writes a script, and peak context sat near 50,000 tokens however large the corpus got.

The control is clean. Codex and OpenCode reach their providers through their own configuration rather than this runner’s, so before window support was added for them they ran the same tasks at the provider default: peak 51,000–82,000 tokens and zero compactions, while every windowed harness compacted repeatedly. Declaring the window moved Codex from 0 compactions to 49 in a single attempt. Nothing else changed.

Two things about how to read this table. Turns are conversation turns. The advisor posts through the same proxy as the agent so that its spend lands in one place, and counting those posts as turns reported the SpecPi + Jev arm as taking two to three times the turns it actually took — a figure this page carried until it was corrected. And every failure in this tier so far is a session that ended without recording its answer, never a session that read the corpus and misjudged it.

Did the layer do anything?

For the first time, yes. Every previously published Jev run recorded zero applied verdicts — not because the layer was broken, but because no session had ever compacted, so the system that guides a compaction cut was never asked anything.

compaction is the row that changed. It was consulted on nearly every compaction that occurred, and its verdict was acted on the majority of those times. progress also returned a tool-error-loop verdict — the one situation a purpose-built task had entirely failed to produce, which appeared on its own once sessions got long enough.

guard still applies nothing. Headless, its middle band has no human to ask, so every “ask” verdict degrades to a defer and only an outright block is reachable. Four of the eight systems are still never invoked: gap, sources and capability need delegation and capability situations this tier does not create, and untrusted needs externally fetched content.

Does it help?

On this evidence, no — and the honest answer is that the sample cannot yet carry a verdict either way.

The direction is not consistent, which is itself the finding. On the marathon the layer compacts less than plain SpecPi (8.9 against 11.4) and costs less ($0.078 against $0.109) while passing less often (5/8 against 8/8). On the haystack it compacts more (5.4 against 3.0) and passes identically (7/8 each), and its apparent cost penalty is mostly one expensive attempt: the means are $0.047 against $0.023, the medians $0.025 against $0.022.

So the layer is not idle and it is not obviously paying either. Its spend buys applied compaction verdicts and several hundred guard consultations, and on the longer task it buys a cheaper session that finishes correctly less often. Which of those matters more is not a question sixteen attempts an arm can answer.

Sixteen attempts an arm, pooled across several runs on one model and one machine. Attempts that never reached a model call are excluded and counted rather than scored as losses — running many harnesses at once contends on a shared session store, and scoring a launch failure as a task failure blames a harness for this suite’s own plumbing. Oh My Pi and Codex report two valid attempts each and should be read as observations rather than rows: both thrash badly at this window, at 117 and 413 turns per attempt against 24 to 31 for the Pi family.

This is a starting point. It is a small suite, on one model, on one machine, and it should be read as such.

• Nothing about capability. Solve rate saturated. No row here supports a claim that one harness does better work than another.
• One model. Every result is conditional on it. A model that uses tools differently would move the tool-surface figures and could reorder the cost table.
• Two attempts per cell on tiers 1 to 3, one on tier 4. Enough to notice flakiness, not enough for a solve-rate interval. Where a harness failed, treat it as one observation. A single attempt supports even less than that.
• One tier 4 cell is a known artifact. Pi’s browser-triage attempt is recorded as a 600-second timeout, and it is not one: a stray process from an abandoned run was competing for the same provider while that tier ran. The same cell completes in 155 seconds with nothing else running. Contention distorts wall clock. Wall clock only becomes a score when it causes a timeout, so no other row is affected. Token, cost, tool-call and context figures are per-request accounting throughout. Pi’s tier 4 figure should be read as a lower bound.
• Tier 4 runs one attempt per cell, and that is not enough to rank on. Re-running Pi’s five tier 4 tasks twice more produced 26s, 56s and a 600-second timeout on the same cost-ladder cell, and 1.000 then 0.000 on the same research-dossier cell with no timeout involved either time. The spread within one harness on one task is larger than the differences between harnesses in the table. Read tier 4 as five observations per harness, not as a rate, and do not order the column.
• Tiers 4 and 5 ran under a shortened clock. Every harness ran every tier in one sitting, but tier 4 was capped at ten minutes against tasks budgeted for forty-five, and tier 5 at fifteen against thirty. Tiers 1 to 3 ran two attempts per cell, tiers 4 and 5 one. Pi's low tier 4 figure is that cap expiring on two of five tasks, not the tasks defeating it, and it would probably move at full budget. No other harness timed out anywhere.
• Tier 5 saturated; tier 4 mostly did not. Five of the six tier 5 tasks returned exactly 1.000 from every harness that produced an artifact, so that tier currently separates nothing on quality and should be read for cost and effort only. Tier 4 is the opposite: four of its five tasks show real spread, and only the research dossier is a ceiling. The one tier 5 task that moved was the authority boundary, and it moved because a harness failed to produce a valid artifact at all. No embedded instruction was followed. No harness produced a forbidden output on any attempt.
• Synthetic tasks. They are self-contained and deterministic by design, which is what makes them gradable and also what keeps them from resembling real work on a real repository.
• Defaults only. Each harness runs as installed. A harness with a large tool surface may well be configurable down, and that would change its fixed cost.

The suite will grow. Harder puzzles mostly measure the model. The next step is tasks where the harness has to help: long sessions where context handling decides the outcome, and cases where tools fail and the harness has to recover.