Each run starts from a new git copy of a fixture in fixtures/, so runs cannot learn from each other.
Benchmarks
Small, public, and rerunnable.
Every number on this page comes from benchmarks/smoke in the public phonton-cli repository: fixed goals on pinned fixtures, scored by hidden tests the run never sees. The suite is small on purpose. It catches regressions and shows what a model can do through Phonton; it does not rank tools.
How a run is scored
Hidden tests decide, not Phonton
After Phonton finishes, tests it never saw are added and the whole suite runs. Accepted means every test passed. Each hidden test fails on the untouched fixture and passes with a reference fix.
Tokens and cost come from the provider-reported usage on the run's receipt. We report medians or totals over every run, never the best one.
Cloud model
Three fixtures · DeepSeek flash · CLI 0.22.2
36 of 36 runs accepted, 193,129 tokens and $0.10 in total. Three runs per task, Windows 11, release build, 2026-10-08. todo-api is four small files, ledger one 339-line module, shop one 28 KB module.
| Task | Accepted | Median s | Median tokens | Median cost | Notes |
|---|---|---|---|---|---|
| todo-remove | 3/3 | 8 | 2,028 | $0.0007 | |
| todo-count | 3/3 | 6 | 1,931 | $0.0007 | |
| todo-sort | 3/3 | 6 | 2,506 | $0.0011 | |
| todo-title-limit | 3/3 | 6 | 2,235 | $0.0008 | |
| ledger-thousands | 3/3 | 9 | 4,985 | $0.0019 | |
| ledger-parens | 3/3 | 8 | 5,104 | $0.0020 | |
| ledger-balance-on | 3/3 | 13 | 7,861 | $0.0051 | |
| ledger-csv-quote | 3/3 | 18 | 8,971 | $0.0067 | |
| shop-coupon-expiry | 3/3 | 14 | 7,924 | $0.0035 | |
| shop-cancel-restock | 3/3 | 14 | 6,504 | $0.0036 | |
| shop-top-ties | 3/3 | 9 | 6,475 | $0.0024 | |
| shop-review-duplicate | 3/3 | 13 | 8,119 | $0.0038 |
Local model
todo-api · qwen2.5-coder:3b on a 6 GB laptop GPU · CLI 0.22.0
Managed Ollama runtime, no key, no network calls for inference. Same tasks, same hidden tests.
| Task | Accepted | Median s | Median tokens | Median cost | Notes |
|---|---|---|---|---|---|
| todo-remove | 3/3 | 16 | 897 | $0 | |
| todo-count | 3/3 | 16 | 873 | $0 | |
| todo-sort | 3/3 | 12 | 449 | $0 | |
| todo-title-limit | 0/3 | 10 | 313 | $0 | Edited the wrong file and still passed the existing tests. Checks are evidence, not proof. |
Local model · CPU only · CI
qwen2.5-coder:3b on a GitHub Actions runner, all three fixtures
The workflow local-smoke.yml runs the local arm on a 4-vCPU Linux runner with no GPU, so anyone can rerun it. A false pass is a run Phonton marked review-ready that the hidden tests then failed: the number a coding tool should drive to zero. The other runs ended with Phonton refusing a candidate that failed checks or made no change.
| Fixture | 0.22.1 | 0.22.2 | 0.23.0, managed runtime | 0.23.0, 7B, 1 run per task |
|---|---|---|---|---|
| todo-api | 9/12 · 3 false | 12/12 · 0 false | 12/12 · 0 false | 4/4 · 0 false |
| ledger | 1/12 · 5 false | 2/12 · 2 false | 1/12 · 2 false | 1/4 · 1 false |
| shop | 1/12 · 0 false | 1/12 · 0 false | 1/12 · 2 false | 2/4 · 1 false |
Accepted runs, then false passes. A 3B model handles small, focused edits; the 339-line and 28 KB modules need a larger one. 0.23.0 made a 3B model's whole-class rewrites parse instead of duplicating the class; acceptance did not move, and two of those rewrites added an unused method that passed the visible tests. CPU decoding is not fully repeatable, so differences of one or two runs per fixture are noise.
Measured, then fixed
Recounting diff ranges halved tokens on mid-file edits
Models often miscount @@ line ranges while quoting every line correctly, and each rejected hunk cost a retry. 0.22.1 recounts each hunk and moves it to where its lines match. Same four ledger tasks (edits inside a 339-line module), DeepSeek flash, three runs each:
| Build | Accepted | Total tokens | Total cost | Total wall time |
|---|---|---|---|---|
| 0.22.0 | 12/12 | 218,139 | $0.172 | 533 s |
| 0.22.1 | 12/12 | 108,591 | $0.063 | 238 s |
Small n, one model, one machine. Reproduce by running the harness with --bin pointed at each build.
- Files over 24 KB used to reach the worker as signatures only. Sending excerpts around the names a task mentions took the four
shoptasks from 11/12 to 12/12 accepted and from 422,615 to 203,489 tokens. - A goal containing “release” (as in “release the stock”) was planned as broad, multi-agent work and split into four or five subtasks, costing 32k-214k tokens per run. Fixed in 0.22.1: one subtask, 4.9k median.
- Every new function got a generated test-writing subtask, asked for or not. Planning tests only when a goal asks for them kept the suite at 36/36 and cut its cost from $0.186 to $0.096 (0.22.2).
Withdrawn
Earlier numbers are no longer published
This page used to show results and cross-tool comparisons from Phonton 0.16 to 0.21. Those builds contained hardcoded answers for some benchmark fixtures (removed in 0.22.0), and the raw artifacts were never public, so those numbers cannot back a claim. Comparisons with other tools will return when they can be rerun from public fixtures.
Reproduce
Run the suite yourself
git clone https://github.com/phonton-dev/phonton-cli
cd phonton-cli
cargo build --release -p phonton-cli
# uses your provider key; PHONTON_HOME keeps your own memory untouched
PHONTON_HOME=/tmp/phonton-bench node benchmarks/smoke/run.mjs --arm cloud --runs 3
node benchmarks/smoke/run.mjs --arm local --runs 3 # with a local model selectedProduct shape, not scores
ADE vs IDE agent vs terminal agent
| Dimension | Typical IDE agent | Terminal agent | Phonton |
|---|---|---|---|
| Primary surface | IDE (file/selection) | Terminal session | Terminal ADE + TUI |
| Unit of work | Edit selection / chat task | Prompt / task | Goal with a visible plan |
| Before your tree changes | Product-dependent | Varies | Your checks pass on a patched copy |
| Handoff | Diff in editor | Diff + chat summary | Receipt: files, checks, model route, cost |
| Undo | Editor history | git | review reject restores the task’s files |
| Local models | Rare | Varies | Calibrated per model; no cloud fallback |
Claim boundary
What these numbers support
Supported
- Acceptance, time, tokens and cost on these fixtures, models and versions.
- Before/after comparisons of Phonton builds run the same way.
- A 3B local model completing small, focused edits through Phonton.
Not supported
- “Phonton beats Claude Code / Codex / Cursor.”
- Percentage token savings against other tools.
- Results on large or unfamiliar codebases.