METRICS

The deliverable. One row per change that could plausibly move the numbers.

Numbers come from go run ./evals -n 3. The runner prints its own METRICS.md row at the end; paste it in and name the change.

Stopping condition: 7/10 tasks passing on every repeat, or 2026-10-28, whichever comes first.

Change	Pass rate	Edit success	Mean turns	Mean cost
Phase 3 baseline	—	—	—	—

The baseline row is empty on purpose: it gets filled by the first real run against a live model, not by anything written in advance.

Reading the numbers

• Pass rate counts a task solved only when every repeat passed. A task that passes two runs in three is a task the harness cannot do reliably, and averaging that away hides the thing worth fixing.
• Edit success is edit_file calls that returned a diff, over all edit_file calls. write_file is deliberately a separate tool so a whole-file rewrite cannot flatter this number.
• Mean cost assumes Claude Opus 5 list pricing ($5/$25 per MTok, cache reads at 0.1x, writes at 1.25x). Change the constants at the top of evals/report.go for another model.

Running it

export ANTHROPIC_API_KEY=...          # or -base-url for a gateway
./evals/selftest.sh                   # no API calls: checks the tasks measure something
go run ./evals -n 3                   # the real thing
go run ./evals -n 1 -tasks rename-symbol -keep   # one task, work dir left behind

Transcripts land in evals/runs/<timestamp>/<task>/<n>.json, with the full message history and every tool call. Read them. Most regressions are obvious within ten seconds of reading a trajectory.

About the tasks

The ten tasks follow the spread in PLAN.md: two failing-test fixes (one with the cause two files from the failure), a feature with tests, a cross-file rename, error propagation, a compile error from a half-finished refactor, a test for untested code, a bug given only as a description, an API migration across call sites, and one task that is impossible — its verifier passes only if the agent reports the blocker instead of deleting the test, skipping it, or inventing the missing fixtures.

They are synthetic Go projects rather than snapshots of real repositories. That is the suite's main weakness: real code is messier, and these will overestimate. Swapping in real repo snapshots is the highest-value improvement available, and the task format makes it a drop-in — setup.sh can clone and patch just as easily as it can write files.

./evals/selftest.sh checks every task's verify.sh fails on the untouched repo and passes after solution.sh. A verifier that passes before the agent does anything measures nothing; run it after touching any task.