SpecPi
Research
What the evidence
shows.
Two questions decide how a harness is set up: is Pi worth running, and is it worth handing work to subagents. Neither has a universal answer. Here is what the published measurements support, and what they do not.
This page summarises public evidence, including the HarnessTax study published on 16 September 2026 by Pan, Yang, Arabzadeh, Chiang, Stoica and Zaharia. SpecPi itself has not been measured against an alternative. Nothing below is a benchmark of this base.
Summary
01The short version: Pi is a credible efficiency baseline, and delegation is worth it selectively. Both claims are conditional on the workload, and the strength of the evidence differs a lot between them.
- Supported
- Pi often costs less. Lower spending shows up across several independent comparisons, sometimes by a wide margin.
- Not supported
- A fixed savings multiplier. Reported ratios run from about 1.2x to 7x. None of them transfers to another workload.
- Not supported
- A success-rate winner. Pi leads on some benchmarks and trails on others. No study establishes parity either.
- Supported
- The harness moves cost more than correctness. Across 21 model–harness pairs, success rates stayed within a few points while the same model's bill varied by up to 5x.
- Supported
- Delegation is task-shaped. Extra agents help bounded research and fresh-context review, and hurt coupled code changes.
The practical consequence: harness and delegation choices are budget decisions, not correctness decisions. A cheaper agent that solves two fewer tasks is the better buy only if those tasks are worth less than the money saved. What they are worth is a number you have to supply.
Reading a cost number
02Most disagreement about harness cost is really disagreement about arithmetic. Four different quantities all get published as "cost", and they answer different questions.
| Quantity | How it is computed | What it tells you |
|---|---|---|
| Mean cost per attempt | total ÷ attempts | Spending across successes and failures alike. |
| Cost per success | total ÷ successes | What each result actually costs, failures included. |
| Success-conditioned cost | cost of wins ÷ successes | Flattering: hides everything spent on failures. |
| Median attempt cost | middle attempt | A typical run, but says nothing about the total. |
Why this matters here. In the FrontierHarness data, failed tasks consume 85.3% of Pi's total spending and 48.7% of Codex's. A success-conditioned metric would throw out most of the Pi bill and make Pi look much better than it is.
Pi's headline advantage also coexists with a slightly higher median attempt cost: $0.2347 against Codex's $0.2305. Both numbers are real; the rest of the two distributions just differ.
When some tasks were never priced, the honest figure is a lower bound, not a measurement. Missing costs do not become zeroes just because they are missing.
Harness choice
03HarnessTax: seven models across three harnesses
The largest controlled comparison so far, published 16 September 2026, evaluates 21 model–harness pairs, seven models across Claude Code, Codex CLI and Pi, on SWE-bench Lite and Terminal-Bench 2.0. It is the first study at this scale to measure Pi directly.
With thanks to the HarnessTax authors. This section summarises HarnessTax: How Much Does the Harness Matter for Coding Agents? by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia (UC Berkeley Sky Lab; Wei-Lin Chiang at Arena). The work is theirs. The summary below leaves things out, so read the study itself.
Read the full study, with its interactive figures and the complete 21-pair grid, at harnesstax.github.io. Please cite the authors rather than this page.
Their main result: the harness changes what you pay far more than what you get. Averaged across shared models, the harness effect on success rate stays within about ±2% on SWE-bench Lite and ±5% on Terminal-Bench 2.0, while the same model's bill varies by as much as 5x. Pi reaches the Pareto frontier on both benchmarks with four tools: read, write, edit and bash.
Both figures are drawn by SpecPi from the data published with the study; the design is ours and any error in it is ours. Method: 30 randomly sampled tasks per benchmark, three attempts each, each harness in its native configuration at its high effort setting, capped at 100 agent turns, priced from a fixed direct-API list dated 1 September 2026. Confidence intervals in the study come from 10,000 bootstrap resamples and are wide: most pairwise differences here are too small to call.
Two details matter more than the headline. First, the spending gap is not explained by the agent working harder: on SWE-bench Lite, Fable 5 averages 15.4 turns in Pi and 15.3 in Claude Code, yet Claude Code costs about twice as much for a 1.1-point gain. Second, the extra cost starts on the very first model call: longer instructions and larger tool schemas make Claude Code's initial context more than ten times Pi's before any work begins.
Their third finding cuts against provider loyalty: across the six Anthropic and OpenAI models and both benchmarks, an alternative harness achieved the highest observed success rate in nine of twelve comparisons. A model does not necessarily do its best work in its own vendor's harness.
What it does not settle. Thirty sampled tasks per benchmark, on two open-source benchmarks the authors note the models may have seen during training. Turn counts are not comparable across harnesses because each defines a turn differently. The authors state plainly that results may differ on other benchmarks and workloads, and they make no claim about any configured Pi distribution, including this one.
FrontierHarness: task-level results on a single model
The most auditable single-model comparison is FrontierHarness: 30 tasks, every harness on the same Kimi K3 model through the same provider, with task-level results released rather than promised. Pi ran at version 0.84.2.
| Harness | Solved | Cost / attempt | Cost / success | Cost coverage |
|---|---|---|---|---|
| Codex | 20 / 30 | $2.31 | $3.47 | 30 / 30 |
| Claude Code | 19 / 30 | ≥ $11.61 | ≥ $18.34 | 29 / 30 |
| Pi | 18 / 30 | $1.46 | $2.43 | 30 / 30 |
| Oh My Pi | 17 / 30 | $2.69 | $4.75 | 30 / 30 |
| Exo | 16 / 30 | $0.56 | $1.05 | 30 / 30 |
Selected rows from twelve configurations, at the benchmark's frozen historical prices. Values marked ≥ are lower bounds because some tasks were never priced. Pi is tied for fourth on accuracy and second-cheapest per success.
That $12.79 carries the whole argument. Below it, Pi's savings outweigh the two missed tasks; above it, they do not. Real tasks are not worth the same amount, and human review time often dwarfs API cost, so use it to frame the decision, not as a threshold to apply.
The task mixture drives the ranking
Pi solves 16 of 21 terminal tasks and 2 of 9 repository-repair tasks; Codex solves 15 and 5. Pi leads on terminal work and trails badly on repository repair. The aggregate ordering partly reflects the benchmark's 70:30 weighting, and neither subgroup is large enough to settle a domain ranking.
Other comparisons point in different directions
| Evaluation | Accuracy | Cost |
|---|---|---|
| Databricks · internal codebase, 6 matched model settings | −7 to +3 pp for Pi | 16.7–51.9% cheaper, every time |
| Augment · Auggie v2 (a Pi fork), 731 SWE-bench Pro | 61.0% vs 61.4% | 52.9% below Claude Code |
| Warden · 86 known vulnerabilities, Sonnet 4.6 | 25 found vs 24 | $19.84 vs $103.59 |
| XBOW · 104 security challenges, GPT-5 | 50.0% vs Codex 67.3% | $0.231 vs $0.396 per success |
| Composio · 25 SaaS workflows, Kimi K3 | 72% vs Oh My Pi 88% | $0.57 vs $0.52 per success |
The counterexamples are the point. On security challenges Pi trails Codex by 17 points, and loses to OpenCode on accuracy and cost per success. On SaaS workflows Oh My Pi (a fork of Pi whose first call is twelve times Pi's, measured in section 05) beats plain Pi by 16 points. A harness that wins on one workload can lose on the next.
Subagents
04Whether a second agent helps depends almost entirely on whether the work decomposes. Kim et al. ran matched token ceilings across six benchmarks and picked the best multi-agent architecture separately for each. The direction of the result flips.
Finance Agent centralized · 50 instances
WorkBench decentralized · 100 instances
BrowseComp-Plus decentralized · 100 instances
PlanCraft hybrid · 100 instances
SWE-bench Verified hybrid · 20 instances
Terminal-Bench independent · 20 instances
Higher is better on every row; bars share a common 0–0.7 scale. Scores keep their source metric, so read each row on its own and do not average across them. The two software rows have only 20 instances each. Matched token ceilings are not matched spending.
Coupled code changes get worse
CooperBench matters most if you write software. Across 652 deliberately overlapping feature tasks, GPT-5 solved 315 working alone and 183 cooperating. Letting the agents talk to each other reduced merge conflicts without improving success for any model tested.
Solo tasks solved out of 652
Cooperative two agents, 100 actions each
The action ceiling is per agent, so the cooperative arm was not cost-matched. This is evidence about agents writing interdependent code, not about read-only research workers.
Where delegation does earn its place
The pattern worth keeping is narrow: independent, breadth-first evidence gathering, and review from a context that never saw the implementation. One agent still owns every edit. Anthropic reports a 90.2% relative improvement on an internal research evaluation with that setup, though it also reports multi-agent systems using roughly 15x the tokens of chat.
Pi's own author lands in a similar spot, from practice rather than measurement. Mario Zechner names code review as the useful subagent case and objects to opaque context transfer and parallel feature-writing teams. His normal habit is a separate review session that he then reads himself.
| Task shape | Start with | Only add a worker if |
|---|---|---|
| Small edit or short lookup | Parent only | Never. Batch the tool calls instead. |
| Coupled change, unsettled interfaces | One implementation owner | Shared decisions are already resolved. |
| Two substantial evidence questions | One or two scouts | The questions and sources genuinely do not overlap. |
| Frozen change with review risk | One fresh reviewer | It gets requirements and the diff, not the author's verdict. |
| The same prompt already failed | Diagnose first | Something changed: new evidence or repaired state. |
Parallel is slower than it sounds. Two independent six-minute investigations take twelve minutes in sequence. Run them concurrently with a minute of preparation and two of synthesis and you get nine. That is a 25% saving, not a doubling. The dollar cost usually goes up.
The full bill is parent + workers + tools + retries + integration. Delegation
moves work into that last term rather than removing it.
For this base
05SpecPi 0.29.0 pins seven packages and keeps its own extensions to scope control and the improvement loop. Read against the evidence above, that shape holds up. It also leaves gaps. This section names both.
What the base already does right
-
Delegation is a pinned package, not a core feature.
specpi-delegation@0.2.0is one of the seven pins, so it can be configured, held at a version, or removed without touching the harness. Given how task-dependent the delegation evidence is, that separation is worth keeping. - Scope control targets the failure the research keeps coming back to. Context omission, duplicated investigation and stale review are the recurring multi-agent failure modes. A task with an explicit scope is the cheapest defence against all three.
- The improvement loop needs a human selection. The literature is blunt that agreement between agents is not verification, so a change needs a person to authorize it.
What the first call costs
The finding that cuts against this base. HarnessTax traces a large part of Claude Code's cost premium to its first model call, where longer instructions and larger tool schemas make its initial context more than ten times Pi's. Every package SpecPi pins adds to that same call, so we measured it instead of assuming Pi stays cheap under configuration.
node scripts/measure-context.mjs --chart --omp=<path to Oh My Pi's cli.js> --oc=<path to OpenCode's binary> --dsh=<path to the DeepSeek Harness bin>
(npm access and a Bun runtime required).
Measurement record and exact pins.
How the profiles were measured
Both SpecPi profiles include all seven installed packages, the installed working agreement,
and the usual skill discovery. The default leaves browser QA, delegation, and web access off.
The enabled profile runs /browser on, /delegate on and
/webaccess on first, and measures what that actually sends rather than adding up
registered schemas. Neither profile has an active goal, scope, improvement selection, or user
configuration.
The script runs in a disposable home and an empty workspace, with no personal credentials, histories, or paid inference. It first checks that all seven pins and all nine production extensions load.
Inside the enabled profile
The gap between the two SpecPi bars comes from several switches, so the enabled profile's tool schema is also published group by group: the same measured schemas, split into groups, with every tool in exactly one group.
Web access is the single largest group at 11,298 characters, and
web_search alone accounts for 4,858 of them. Browser QA's fourteen tools cost less
together than web access's four, which an estimate would get backwards. Leaving all three
opt-in groups hidden keeps 23,797 characters of tool schema out of every request.
Counts are compact tool-definition JSON plus system/developer text, in JavaScript UTF-16 code units. The user prompt and transport envelope are excluded; discovered skill metadata counts, and skill bodies count only if loaded into the request. Temporary path text is counted as-is, so totals move with host paths, dates, and versions. These figures are exact for these requests, not universal session sizes or token counts.
The other harnesses, measured the same way
Oh My Pi is not another package set but a Bun-based fork of Pi, measured as installed with its own tools, skills, and prompt. Nothing is added; only local rule and extension discovery is turned off. Its default session sends 65,816 characters across 11 tools: about 11.9x stock Pi and over four times a default SpecPi, putting a configured Pi fork between Codex CLI and Claude Code.
That belongs next to the two other places Oh My Pi appears on this page. It costs 84% more per attempt than Pi on FrontierHarness and solves one task fewer; it beats Pi by 16 points on Composio's SaaS workflows. A first call twelve times the size explains neither the win nor the loss. The overhead is real and measurable, and measuring it says nothing about what it buys.
OpenCode is measured the same way, as installed. It is a compiled binary with
its own tools, prompt and skill list, configured only to reach the same synthetic provider. Its
default build-agent session sends 31,043 characters across 10 tools: about
5.6x stock Pi and about twice a default SpecPi, landing it between a
default SpecPi and the enabled profile. One method detail:
opencode run also fires a small parallel call to write a session title, and
pinning --title suppresses it, so the recorded request is the conversation call,
the first and only model call the turn sends.
DeepSeek Harness is the same kind of measurement on a different stack: a Node application that composes its tools, skills and prompt from plugin bundles. It is measured as installed, with only the provider route declared through its own patch layer and its harness home kept out of the way. Its default session sends 31,743 characters across 25 tools: about 5.7x stock Pi and about twice a default SpecPi, within 700 characters of OpenCode. Its headless run also fires a small auxiliary request to write the session title; that request carries no tool schema, so the measured one is the conversation call.
Correction. The earlier 10,536-character SpecPi figure measured only first-party extensions, without the installed AGENTS guidance or the four third-party packages, so it was not a complete default-install measurement; the earlier “all on” row was not reproduced by enabling those tools either. Both are replaced here by observed requests. A draft of this revision also moved the Oh My Pi bar into the dimmed study group. That was wrong: HarnessTax covers Claude Code, Codex CLI and Pi only, and the Oh My Pi figure has always been ours. It is measured again above, and now reads 65,816 rather than the 65,843 published earlier. The 27-character drift sits in the instructions segment, the kind of variation the method notes warn about.
Less initial context does not establish lower total spending or better task outcomes. The external studies above remain separate evidence; none establishes that these SpecPi profiles improve task success. That part is still yours to measure.
Settings worth choosing deliberately
specpi-delegation handles most of what the evidence warns about in code,
and leaves two knobs to you. Both are set from the session with
/delegate, and are saved per agent directory.
| Control | Why the evidence bears on it |
|---|---|
/delegate budget <1–64> |
Retries and fan-out are where cost escapes. The session multiplier turns an unbounded bill into a bounded one; 8 is the default. |
/delegate timeout <1–60> |
A stalled child spends without converging. Ten minutes is the default ceiling per job, and lowering it costs less than a long failing run. |
Concurrency: fixed at 2 |
Parallelism pays only when concurrent work exceeds coordination overhead. The package does not expose a higher ceiling, so there is none to raise by accident. |
Recursion: not available |
Recursive delegation multiplies every coordination failure. A child holds three read-only tools and cannot delegate further. |
Child context: always fresh |
A reviewer that inherits the implementer's context is not a fresh reviewer. Children see a frozen source snapshot and the request, never the parent's conversation. |
The evidence points to something close to the package's defaults: bounded
spending, shallow delegation, fresh context, and children used for review and bounded research
while the parent session keeps writing the code. Delegation ships off. Run
/delegate on for this session, or /delegate startup on to save that preference.
Not established
06None of these follow from the evidence above. The page would mislead without them, so here they are.
- SpecPi task effectiveness has not been established against an alternative. The installed-profile comparison measures initial context only. External task results describe stock Pi or another product; version proximity does not establish transfer.
- No savings multiplier transfers. Published ratios span roughly 1.2x to 7x and each belongs to its own model, route, task set and pricing basis.
- A large p-value is not equivalence. The FrontierHarness comparison is underpowered at 30 tasks. It fails to show a difference; it does not show parity.
- Auggie v2 is not evidence about stock Pi. It changes retrieval, compaction and more at once, so nothing in that result attributes to any single mechanism.
- Benchmark scores do not pool. Repository repair, SaaS workflows, security challenges and biology workflows have different success events. An average across them would mean nothing.
- Token counts are not dollars. Cache categories, pricing and counting conventions differ between harnesses; one study found Pi using about 2.4x Codex's runtime tokens.
To settle any of this for your own work, run a matched comparison on your own repositories: the same model and provider, a task set chosen before you see results, held-out verification, and complete cost accounting that includes failed attempts and human review time.
Sources
07Primary evidence behind this page. Study prices are historical and say nothing about current pricing. Live pages were checked in September 2026.
-
Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei
Zaharia. HarnessTax: How Much Does the Harness Matter for Coding Agents? UC
Berkeley Sky Lab and Arena, 16 September 2026. 21 model–harness pairs across Claude Code,
Codex CLI and Pi, on SWE-bench Lite and Terminal-Bench 2.0. The primary source for
section 03, with our thanks to the authors.
harnesstax.github.io
@misc{pan2026harnesstax, title={{HarnessTax: How Much Does Harness Matter for Coding Agents?}}, author={Pan, Melissa Z. and Yang, Shuo and Arabzadeh, Negar and Chiang, Wei-Lin and Stoica, Ion and Zaharia, Matei}, year={2026}, url={https://harnesstax.github.io/}} - FrontierHarness Eval v1. Frozen 30-task benchmark and task-level results, generated 22 August 2026. github.com/frontier-harness-eval/eval
- Databricks. Benchmarking coding agents on a multi-million line codebase, 8 July 2026. Six matched model and effort settings.
- Augment Code. Rebuilding the Auggie CLI harness, August 2026. 731 SWE-bench Pro instances on Opus 4.7.
- Dhakal, Neupane & Chaudhary. Baselines Before Architecture, arXiv:2607.13085. 104 XBOW penetration-testing challenges.
- Composio. Best AI agent harnesses, 4 August 2026. 25 SaaS workflows on shared Kimi K3.
- Sentry / Warden. Known-vulnerability recall across 86 vulnerabilities.
- Kim et al.. Capable language models can outgrow the benefits of collaboration. Nature Machine Intelligence, 24 July 2026. nature.com/articles/s42256-026-01268-y
- CooperBench. Why coding agents cannot be your teammates yet, arXiv:2601.13295. 652 paired-feature tasks across 12 repositories.
- SwarmBench. Can large language models act as agent swarm orchestrators? arXiv:2608.30661. Cost-matched single-agent comparison.
- Anthropic. How we built our multi-agent research system, 13 June 2025. anthropic.com/engineering/multi-agent-research-system
- Walden Yan / Cognition. Multi-agents: what's actually working, 22 April 2026. cognition.com/blog/multi-agents-working
- Mario Zechner. What I learned building an opinionated and minimal coding agent, 30 November 2025. mariozechner.at/posts/2025-11-30-pi-coding-agent
-
Oh My Pi is a Bun-based fork of Pi, by can1357 and contributors. Not a study: the npm
package
@oh-my-pi/pi-coding-agent18.2.2, installed and measured here for the first-call figure in section 05. HarnessTax does not publish a figure for it. omp.sh -
OpenCode is an agent harness by SST. Not a study:
opencode-ai1.18.31, installed and measured here for the first-call figure in section 05. HarnessTax does not publish a figure for it. opencode.ai -
DeepSeek Harness is an agent harness from DeepSeek. Not a study:
@deepseek-ai/dsh0.1.5-rc.2, installed and measured here for the first-call figure in section 05. HarnessTax does not publish a figure for it. npmjs.com/package/@deepseek-ai/dsh - Zhang et al.. Stop comparing LLM agents without disclosing the harness, arXiv:2605.23950. Three-model, three-configuration factorial design.