# METRICS

The deliverable. One row per change that could plausibly move the numbers.

Numbers come from `go run ./evals -n 3`. The runner prints its own METRICS.md
row at the end; paste it in and name the change.

**Stopping condition: 7/10 tasks passing on every repeat, or 2026-10-28,
whichever comes first.**

| Change | Pass rate | Edit success | Mean turns | Mean cost |
| --- | --- | --- | --- | --- |
| Phase 3 baseline | — | — | — | — |

The baseline row is empty on purpose: it gets filled by the first real run
against a live model, not by anything written in advance.

## Reading the numbers

- **Pass rate** counts a task solved only when *every* repeat passed. A task
  that passes two runs in three is a task Codey cannot do reliably, and
  averaging that away hides the thing worth fixing.
- **Edit success** is `edit_file` calls that returned a diff, over all
  `edit_file` calls. `write_file` is deliberately a separate tool so a
  whole-file rewrite cannot flatter this number.
- **Mean cost** assumes Claude Opus 5 list pricing ($5/$25 per MTok, cache
  reads at 0.1x, writes at 1.25x). Change the constants at the top of
  `evals/report.go` for another model.

## Running it

```bash
export ANTHROPIC_API_KEY=...          # or -base-url for a gateway
./evals/selftest.sh                   # no API calls: checks the tasks measure something
go run ./evals -n 3                   # the real thing
go run ./evals -n 1 -tasks rename-symbol -keep   # one task, work dir left behind
```

Transcripts land in `evals/runs/<timestamp>/<task>/<n>.json`, with the full
message history and every tool call. Read them. Most regressions are obvious
within ten seconds of reading a trajectory.

## About the tasks

The ten tasks follow the spread in PLAN.md: two failing-test fixes (one with
the cause two files from the failure), a feature with tests, a cross-file
rename, error propagation, a compile error from a half-finished refactor, a
test for untested code, a bug given only as a description, an API migration
across call sites, and one task that is **impossible** — its verifier passes
only if the agent reports the blocker instead of deleting the test, skipping
it, or inventing the missing fixtures.

They are synthetic Go projects rather than snapshots of real repositories.
That is the suite's main weakness: real code is messier, and these will
overestimate. Swapping in real repo snapshots is the highest-value improvement
available, and the task format makes it a drop-in — `setup.sh` can clone and
patch just as easily as it can write files.

`./evals/selftest.sh` checks every task's `verify.sh` fails on the untouched
repo and passes after `solution.sh`. A verifier that passes before the agent
does anything measures nothing; run it after touching any task.