SpecPi

Evaluations

Six harnesses.
Two benchmarks.

SWE-bench Verified and Terminal-Bench 2.0, one model, one endpoint. Only the harness changes.

attempts · Terminal-Bench sittings. SpecPi v0.33.0, Jev off. Data

Head to head

01

SWE-bench Verified

Pi (base)SpecPiSpecPi vs PiSolved12 tasks · p = 1.00level34/36 → 34/36Pi: 34/36SpecPi: 34/36Prompt tokensper attempt+2%303k → 309kPi: 303kSpecPi: 309kCostper attempt−1%$0.0069 → $0.0068Pi: $0.0069SpecPi: $0.0068
Real repository bugs, fixed and checked by the project's own tests. Level: this benchmark does not separate the two.

Terminal-Bench 2.0

Pi (base)SpecPiSpecPi vs PiSolvedwidened slice · p = 0.21+15 pts25/39 → 30/38Pi: 25/39SpecPi: 30/38sanitize-git-repo10 attempts · p = 0.003+70 pts3/10 → 10/10Pi: 3/10SpecPi: 10/10fix-git10 attempts · p = 1.00level10/10 → 10/10Pi: 10/10SpecPi: 10/10Prompt tokensper attempt−16%454k → 382kPi: 454kSpecPi: 382kCostper attempt−22%$0.0146 → $0.0114Pi: $0.0146SpecPi: $0.0114
Same sitting, same tasks. Green is better for SpecPi.

All harnesses

02
Solved, per attempt0%25%50%75%100%Terminal-Bench · Calibration · 7 tasks × 5Pi (base)30/35Oh My Pi31/35Claude Code31/35Terminal-Bench · Widened · 13 tasks × 3 per sittingPi (base)134/195SpecPi62/77Oh My Pi84/116Claude Code50/77OpenCode29/39DeepSeek Harness21/38Terminal-Bench · Git pair · 2 tasks × 7Pi (base)9/14SpecPi14/14Oh My Pi11/14Claude Code7/14OpenCode12/14DeepSeek Harness13/14SWE-bench Verified · 12 tasks × 3Pi (base)34/36SpecPi34/36Oh My Pi35/36Claude Code35/36OpenCode35/36DeepSeek Harness33/36share of attempts reaching reward 1
Reward 1 or a passing test suite counts as solved. The benchmarks sit side by side and are never added together.

SWE-bench Verified

HarnessSolvedRatePrompt tokCache hitCost/attempt
Pi (base)34/360.944302,91396.2%$0.0069
SpecPi34/360.944308,63496.4%$0.0068
Oh My Pi35/360.972378,22094.2%$0.0087
Claude Code35/360.972744,68996.0%$0.0140
OpenCode35/360.972387,63195.5%$0.0049
DeepSeek Harness33/360.917695,20095.7%$0.0121

Terminal-Bench 2.0

HarnessSolvedRatePer-sitting rangePrompt tokCache hitCost/attempt
Pi (base)164/2300.71356–77% (5)436,21492.9%$0.0155
SpecPi62/770.80579–82% (2)407,41692.4%$0.0144
Oh My Pi115/1510.76266–76% (2)1,108,07696.5%$0.0237
Claude Code81/1120.72364–66% (2)670,82895.4%$0.0273
OpenCode29/390.744one sitting494,17396.5%$0.0124
DeepSeek Harness21/380.553one sitting1,126,58695.4%$0.0226

Spend

03
Prompt tokens per attempt0k375k750k1125k1500kTerminal-Bench · Calibration · 7 tasks × 5Pi (base)315,895Oh My Pi618,939Claude Code463,868Terminal-Bench · Widened · 13 tasks × 3 per sittingPi (base)457,810SpecPi407,416Oh My Pi1,255,661Claude Code764,901OpenCode494,173DeepSeek Harness1,126,586Terminal-Bench · Git pair · 2 tasks × 7Pi (base)596,953SpecPi258,503Oh My Pi804,713Claude Code796,484OpenCode367,313DeepSeek Harness728,400SWE-bench Verified · 12 tasks × 3Pi (base)302,913SpecPi308,634Oh My Pi378,220Claude Code744,689OpenCode387,631DeepSeek Harness695,200tokens sent, summed over the attempt's model calls
Cost per attempt$0.000$0.007$0.015$0.022$0.030Terminal-Bench · Calibration · 7 tasks × 5Pi (base)$0.0108Oh My Pi$0.0122Terminal-Bench · Widened · 13 tasks × 3 per sittingPi (base)$0.0163SpecPi$0.0144Oh My Pi$0.0272Claude Code$0.0273OpenCode$0.0124DeepSeek Harness$0.0226Terminal-Bench · Git pair · 2 tasks × 7Pi (base)$0.0137SpecPi$0.0090Oh My Pi$0.0181Claude Code$0.0139OpenCode$0.0095DeepSeek Harness$0.0164SWE-bench Verified · 12 tasks × 3Pi (base)$0.0069SpecPi$0.0068Oh My Pi$0.0087Claude Code$0.0140OpenCode$0.0049DeepSeek Harness$0.0121USD at the frozen price list, for the rows whose cached share was recorded
Output tokens are of every Terminal-Bench bill, so cost does not track prompt tokens one for one.
Prompt cache hit rate0%25%50%75%100%Terminal-BenchPi (base)92.9%SpecPi92.4%Oh My Pi96.5%Claude Code95.4%OpenCode96.5%DeepSeek Harness95.4%SWE-bench Verified · 12 tasks × 3Pi (base)96.2%SpecPi96.4%Oh My Pi94.2%Claude Code96.0%OpenCode95.5%DeepSeek Harness95.7%cached share of prompt tokens, each benchmark pooled on its own

Noise

04
Solve rate, one bar per sitting0%25%50%75%100%Pi (base) · 56-77%21 Sep30/3922 Sep · b22/3922 Sep · d30/3922 Sep · f27/3924 Sep · a25/39SpecPi · 79-82%24 Sep · a30/3824 Sep · b32/39Oh My Pi · 66-76%21 Sep59/7824 Sep · b25/38Claude Code · 64-66%21 Sep25/3822 Sep · a25/39widened slice only; each bar is one launch of 39 attempts
Terminal-Bench, same software, same tasks, different hour. Bare Pi spans across sittings, wider than most gaps between harnesses. SWE-bench has one sitting so far.

Task by task

05

SWE-bench Verified

TaskPi (base)SpecPiOh My PiClaude CodeOpenCodeDeepSeek Harness
django__django-110663/33/33/33/33/33/3
django__django-121933/33/33/33/33/33/3
django__django-130893/33/33/33/33/33/3
django__django-135121/31/32/32/32/30/3
django__django-144043/33/33/33/33/33/3
django__django-168013/33/33/33/33/33/3
matplotlib__matplotlib-234763/33/33/33/33/33/3
psf__requests-11423/33/33/33/33/33/3
pytest-dev__pytest-100813/33/33/33/33/33/3
sphinx-doc__sphinx-104353/33/33/33/33/33/3
sphinx-doc__sphinx-93673/33/33/33/33/33/3
sympy__sympy-245393/33/33/33/33/33/3

Terminal-Bench 2.0

TaskPi (base)SpecPiOh My PiClaude CodeOpenCodeDeepSeek Harness
cobol-modernization10/156/67/93/62/33/3
configure-git-webserver10/153/63/92/53/31/3
extract-elf10/155/67/93/63/32/3
fix-git15/156/69/95/63/33/3
hf-model-inference14/156/66/96/63/31/3
multi-source-data-merger15/156/69/96/63/33/3
prove-plus-comm13/156/69/96/63/33/3
pytorch-model-cli3/150/61/93/60/31/3
raman-fitting3/152/54/90/62/31/3
sanitize-git-repo1/156/67/92/63/30/3
sqlite-with-gcov14/156/67/96/60/30/2
tune-mjcf11/154/67/92/61/30/3
vulnerable-secret15/156/68/86/63/33/3

Solved / attempts, shaded by rate. configure-git-webserver only passes when a test file is left behind: its checker's own push never succeeds.

Method

06
  • Model deepseek-v4.1-flash through one endpoint, run by Harbor on Docker.
  • SWE-bench 12 Verified tasks × 3 attempts for every harness. Pi and SpecPi ran overnight, the other four later the same day. Six attempts that failed to set up on a download were run again.
  • Terminal-Bench 7 calibration × 5 attempts, 13 widened × 3 per sitting, and a git pair of 7 attempts on each git task for every harness, kept out of the pooled totals.
  • SpecPi the published 0.33.0 with Jev off. Earlier SpecPi + Jev rows are not shown: the layer changed nothing measurable.
  • Cost recomputed from logged tokens at a frozen price list.
  • Left out one broken task; failed setups.