Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Showing 1 changed file with 8 additions and 3 deletions.
| @@ -10,10 +10,15 @@ whichever comes first.** | ||
| 10 | 10 | |
| 11 | 11 | | Change | Pass rate | Edit success | Mean turns | Mean cost | |
| 12 | 12 | | --- | --- | --- | --- | --- | |
| 13 | - | Phase 3 baseline | — | — | — | — | | |
| 13 | + | Baseline, claude-opus-5, effort default, 2026-10-02 | 10/10 | 93% | 6.8 | $0.0891 | | |
| 14 | 14 | |
| 15 | - The baseline row is empty on purpose: it gets filled by the first real run | |
| 16 | - against a live model, not by anything written in advance. | |
| 15 | + The baseline is 30 runs out of 30, which says more about the suite than about | |
| 16 | + Codey: the stopping condition was met on the first measurement, so these ten | |
| 17 | + tasks can no longer tell a good change from a bad one. Every row after this | |
| 18 | + one needs harder tasks before its number means anything — real repository | |
| 19 | + snapshots first, as below. What the baseline does show is where the misses | |
| 20 | + are: `write_file` failed 3 of 9 calls and `edit_file` 3 of 43, all recovered | |
| 21 | + from, and those are the transcripts worth reading. | |
| 17 | 22 | |
| 18 | 23 | ## Reading the numbers |
| 19 | 24 | |