METRICS

The deliverable. One row per change that could plausibly move the numbers.

Numbers come from go run ./evals -n 3. The runner prints its own METRICS.md row at the end; paste it in and name the change.

Stopping condition: 7/10 tasks passing on every repeat, or 2026-10-28, whichever comes first.

Change	Pass rate	Edit success	Mean turns	Mean cost
Baseline, claude-opus-5, effort default, 2026-10-02	10/10	93%	6.8	$0.0891

The baseline is 30 runs out of 30, which says more about the suite than about Codey: the stopping condition was met on the first measurement, so these ten tasks can no longer tell a good change from a bad one. Every row after this one needs harder tasks before its number means anything — real repository snapshots first, as below. What the baseline does show is where the misses are: write_file failed 3 of 9 calls and edit_file 3 of 43, all recovered from, and those are the transcripts worth reading.

Reading the numbers

• Pass rate counts a task solved only when every repeat passed. A task that passes two runs in three is a task Codey cannot do reliably, and averaging that away hides the thing worth fixing.
• Edit success is edit_file calls that returned a diff, over all edit_file calls. write_file is deliberately a separate tool so a whole-file rewrite cannot flatter this number.
• Mean cost assumes Claude Opus 5 list pricing ($5/$25 per MTok, cache reads at 0.1x, writes at 1.25x). Change the constants at the top of evals/report.go for another model.

Running it

export ANTHROPIC_API_KEY=...          # or -base-url for a gateway
./evals/selftest.sh                   # no API calls: checks the tasks measure something
go run ./evals -n 3                   # the real thing
go run ./evals -n 1 -tasks rename-symbol -keep   # one task, work dir left behind

Transcripts land in evals/runs/<timestamp>/<task>/<n>.json, with the full message history and every tool call. Read them. Most regressions are obvious within ten seconds of reading a trajectory.

About the tasks

The ten tasks follow the spread in PLAN.md: two failing-test fixes (one with the cause two files from the failure), a feature with tests, a cross-file rename, error propagation, a compile error from a half-finished refactor, a test for untested code, a bug given only as a description, an API migration across call sites, and one task that is impossible — its verifier passes only if the agent reports the blocker instead of deleting the test, skipping it, or inventing the missing fixtures.

They are synthetic Go projects rather than snapshots of real repositories. That is the suite's main weakness: real code is messier, and these will overestimate. Swapping in real repo snapshots is the highest-value improvement available, and the task format makes it a drop-in — setup.sh can clone and patch just as easily as it can write files.

./evals/selftest.sh checks every task's verify.sh fails on the untouched repo and passes after solution.sh. A verifier that passes before the agent does anything measures nothing; run it after touching any task.