In the open • every run
Results
One of our research questions: how much real work can a small local model do on a long, loosely described task?
We measure purr on public benchmarks, with the settings the other harnesses were run with. The numbers come straight from the purr repo, with sample size and margin.
| Date | Benchmark | Tasks × tries | Result | ± | Bar | Note and source |
|---|---|---|---|---|---|---|
| Terminal-Bench 2.1purr 0.5.0+fae8051 | 89 × 1 | 82.0% | ± 4.1 | The best published run. No timeouts, no errors, cost $3.84. Source | ||
| Terminal-Bench 2.1purr 0.5.0+0736578 | 89 × 1 | 80.9% | ± 4.2 | One timeout and one error, cost $4.52. Source | ||
| Terminal-Bench 2.0, quick runpurr 0.3.1+3dde078 | 20 × 1 | 75.0% | ± 9.9 | The same 20 tasks from version to version. Wide margin: compare only with other quick runs, not full runs. Source | ||
| DeepSWE 1.1, stoppedpurr 0.4.0+bd79ea4 | 48 / 113 graded | 56.2% | — | Stopped by hand: a bug capped purr at 150 steps instead of 500. Fixed; it hasn't been rerun yet. Not comparable. Source | ||
| purr bench, own taskspurr before 0.3 | 9 × 1 | 8 / 9 | — | Not official. OpenCode on the same model: 7 / 9. The tasks were written alongside purr, so a task either way is noise. Source |
Every run above uses an API model (DeepSeek V4.1 Flash), because that's what the reference scores were published with. They show whether purr holds up on a public benchmark. Results for the small local models purr is really built for haven't been published yet.
Next to other harnesses
Same benchmark, same model, scores as their makers published them. It isn't a level race: theirs are full runs with no margin given, ours is our best one-try run, and in our runs the task container had network access.
How to read it: the short cells in purr's row are the uncertainty. Six of the nine published scores fall inside it or on its edge; the top three (88.0 to 90.6%) are clearly above it. Two tasks can't be passed by anyone right now (their grader's Debian mirror is gone); of the other 87, purr passed 73 (83.9%).
| Harness | Tries per task | pass@1 | Bar |
|---|---|---|---|
| DSH Minimal | not given | 90.6% | |
| mini-SWE-agent | not given | 90.3% | |
| Claude Code | not given | 88.0% | |
| Pi | not given | 86.1% | |
| DSH Standard | not given | 85.8% | |
| DSH PTC | not given | 85.8% | |
| OpenCode | not given | 85.0% | |
| Codex | not given | 84.1% | |
| Ante | 5 | 83.9% | |
| purr ours, best one-try run | 1 | 82.0% ± 4.1 |
How runs are done • hardware
Method
How the runs are done, so a result can stand next to the others and be checked.
- Runner
- Harbor, the runner the public leaderboards use. Each task in its own Docker container, purr exactly as shipped, and the commit recorded with every try.
- Model
- DeepSeek V4.1 Flash from DeepSeek's own host, with no fallback hosts, since some of those serve it at lower precision.
- Settings
- Temperature 1.0, top_p 0.95, up to 64k tokens per reply, 1M context, 500 model calls and each task's own time limit. As close as we can get to DeepSeek's own comparison.
- Scoring
- pass@1 is the average share of tries that passed. Timeouts and errors count as fails, as on the leaderboards. ± is the standard error over tasks.
- Differences
- On Terminal-Bench the task container's network is open in our runs and was closed in DeepSeek's, so tasks that download something are easier for us. One try per task is less precise than a full run with several tries.
Hardware
The benchmark runs use an API model, so our own machine mostly affects the times. The purr bench runs and local models:
- CPU
- AMD Ryzen 9 5900X
- Memory
- 32 GB
- GPU
- NVIDIA RTX 4080 (16 GB)
- OS
- Arch Linux
- Local models
- Ollama
In English, in the purr repo
Notes
| Name | Last modified | Description |
|---|---|---|
| ../ | The repository root | |
| README.md | What purr is, installing it, the modes, and why small models do better in it | |
| docs/guide.md | Every command, mode and setting | |
| docs/benchmarks.md | How purr is measured: purr bench, the chat bench and the real benchmarks | |
| benchmarks/terminal-bench/README.md | Terminal-Bench: how it's run, and the results version by version | |
| benchmarks/deepswe/README.md | DeepSWE: how it's run, and the stopped run | |
| benchmarks/purr-bench/README.md | purr bench: our own small tasks, not official | |
| CHANGELOG.md | Changes by version |
