# METRICS The deliverable. One row per change that could plausibly move the numbers. Numbers come from `go run ./evals -n 3`. The runner prints its own METRICS.md row at the end; paste it in and name the change. **Stopping condition: 7/10 tasks passing on every repeat, or 2026-10-28, whichever comes first.** | Change | Pass rate | Edit success | Mean turns | Mean cost | | --- | --- | --- | --- | --- | | Baseline, claude-opus-5, effort default, 2026-10-02 | 10/10 | 93% | 6.8 | $0.0891 | The baseline is 30 runs out of 30, which says more about the suite than about Codey: the stopping condition was met on the first measurement, so these ten tasks can no longer tell a good change from a bad one. Every row after this one needs harder tasks before its number means anything — real repository snapshots first, as below. What the baseline does show is where the misses are: `write_file` failed 3 of 9 calls and `edit_file` 3 of 43, all recovered from, and those are the transcripts worth reading. ## Reading the numbers - **Pass rate** counts a task solved only when *every* repeat passed. A task that passes two runs in three is a task Codey cannot do reliably, and averaging that away hides the thing worth fixing. - **Edit success** is `edit_file` calls that returned a diff, over all `edit_file` calls. `write_file` is deliberately a separate tool so a whole-file rewrite cannot flatter this number. - **Mean cost** assumes Claude Opus 5 list pricing ($5/$25 per MTok, cache reads at 0.1x, writes at 1.25x). Change the constants at the top of `evals/report.go` for another model. ## Running it ```bash export ANTHROPIC_API_KEY=... # or -base-url for a gateway ./evals/selftest.sh # no API calls: checks the tasks measure something go run ./evals -n 3 # the real thing go run ./evals -n 1 -tasks rename-symbol -keep # one task, work dir left behind ``` Transcripts land in `evals/runs///.json`, with the full message history and every tool call. Read them. Most regressions are obvious within ten seconds of reading a trajectory. ## About the tasks The ten tasks follow the spread in PLAN.md: two failing-test fixes (one with the cause two files from the failure), a feature with tests, a cross-file rename, error propagation, a compile error from a half-finished refactor, a test for untested code, a bug given only as a description, an API migration across call sites, and one task that is **impossible** — its verifier passes only if the agent reports the blocker instead of deleting the test, skipping it, or inventing the missing fixtures. They are synthetic Go projects rather than snapshots of real repositories. That is the suite's main weakness: real code is messier, and these will overestimate. Swapping in real repo snapshots is the highest-value improvement available, and the task format makes it a drop-in — `setup.sh` can clone and patch just as easily as it can write files. `./evals/selftest.sh` checks every task's `verify.sh` fails on the untouched repo and passes after `solution.sh`. A verifier that passes before the agent does anything measures nothing; run it after touching any task.