# Building Codey, a Coding Agent Harness in Go — Build Plan

Target: a harness that can be pointed at a real Go or TypeScript repo, told
"make the failing test pass," and actually do it — with numbers proving how
often it works.

## Assumptions

- Single provider: Anthropic Messages API. Multi-provider is a later refactor,
  not a starting constraint.
- Go 1.22+. Generics are used for the tool registry.
- Terminal interface. No TUI framework at first — plain stdout, add Bubble Tea
  only if you want it later.
- Solo project, roughly 5–6 weeks part-time.

**Stopping condition: 7 out of 10 eval tasks passing, or six weeks, whichever
comes first.** (See METRICS.md.)

## Repo layout

```
codey/
  cmd/codey/main.go           # flag parsing, wiring, REPL entry
  internal/
    llm/
      client.go               # HTTP + SSE against the Messages API
      types.go                # Message, ContentBlock, ToolUse, ToolResult
      stream.go               # partial tool-call JSON accumulation
    agent/
      loop.go                 # the core turn loop
      state.go                # conversation history, file-read tracking
      compact.go              # summarization when the window fills
    tools/
      registry.go             # Tool interface + typed wrapper
      read.go glob.go grep.go
      edit.go write.go
      bash.go
      truncate.go             # shared output-capping helper
    approval/
      policy.go               # allow/deny/ask, with session memory
    metrics/
      log.go                  # JSONL event log — every tool call, every token count
    ui/
      render.go diff.go
  evals/
    tasks/                    # one dir per task: setup.sh, prompt.txt, verify.sh
    runner.go
  METRICS.md                  # you fill this in as you go. This is the deliverable.
```

Two rules that save pain later:

1. **`agent` never imports `ui`.** The loop must be runnable headless, because
   the eval runner drives it.
2. **Every tool returns `(string, error)` where the error is for the model to
   read**, not for you to log and swallow. A failed edit is a normal event in
   the transcript, not an exception.

## Phase 0 — Skeleton and the tool registry

Time: half a day. **Done when:** `go test ./...` passes with a fake tool
registered and dispatched.

The one piece of Go design worth getting right on day one is typed tool
arguments, because otherwise you'll be doing `map[string]any` assertions in
every tool for the rest of the project.

Derive `schemaFor[T]()` from struct tags with `github.com/invopop/jsonschema`
rather than hand-writing JSON Schema. Hand-written schemas drift from the
structs and you won't notice until the model sends a field you don't parse.

Also in this phase: `metrics.Log(event)` writing one JSON object per line to
`./.codey/events.jsonl`. Wire it before you have anything to measure.
Retrofitting instrumentation is how projects end up with no numbers.

## Phase 1 — The minimal loop

Time: 1–2 days. **Done when:** you ask "what does main.go do?" and the agent
reads the file and answers.

Non-streaming first. Streaming is a distraction until the loop is correct.

```
build messages → POST /v1/messages → response
  if stop_reason == "tool_use":
      for each tool_use block: dispatch, collect result
      append assistant message + user message containing tool_result blocks
      repeat
  else: print text, return
```

Things that bite here:

- Tool results go back as a **user** message containing `tool_result` blocks,
  one per `tool_use` id, in a **single** message. Multiple parallel tool calls
  means multiple result blocks in that one message — not one message each.
- Every `tool_use` id must get a result, even if execution failed. Send the
  error text with `is_error: true`. A missing result is an API error.
- Set a max-turn cap (start at 30) and return a clear message when hit.
  Unbounded loops burn money fast.
- Thread `context.Context` from main through the loop into every tool from the
  very start. Retrofitting cancellation later means touching every signature.

Only one tool this phase: `read_file(path, offset?, limit?)`.

## Phase 2 — The real tool set

Time: 1 week. **Done when:** the agent can find code it hasn't been shown,
change it, and run the tests.

### `edit_file(path, old_str, new_str)` — the centerpiece

- Reject if the file hasn't been read this session. Track
  `map[string]time.Time` of reads; if the file's mtime changed since, force a
  re-read. This single rule kills a large class of hallucinated edits.
- Count exact occurrences of `old_str`.
  - **1 match** → apply. Return a short unified diff of the changed region, not
    "OK". The model uses the diff to verify it did what it meant.
  - **0 matches** → do not just say "not found." Retry the match with
    normalized whitespace; if that hits, tell the model the indentation differs
    and show the actual line. If it still misses, return the 20 lines around
    the closest-matching line. A failure that carries context gets fixed in one
    turn; a bare "not found" costs three.
  - **>1 match** → return the line numbers of every match and ask for a longer
    unique span. Never silently pick the first.
- Preserve the file's existing line endings. Detect on read, restore on write.
- Log every attempt to the metrics file with the outcome. Edit success rate is
  your headline metric.

### `bash(command, timeout_ms)`

- Run via `exec.CommandContext` so cancellation works.
- Cap output. Use head+tail (e.g. first 100 and last 100 lines with a
  `… N lines elided …` marker), never head alone — test failures live at the end.
- Default timeout 60s; kill the process **group**, not just the process, or
  orphaned children hang the loop.
- Return exit code, stdout and stderr together, clearly labelled.

### `glob(pattern)` and `grep(pattern, path?, glob?)`

Shell out to `rg` if present, fall back to `filepath.WalkDir` + `regexp`. Cap
results at ~100 with a count of how many were suppressed. This is your entire
code-search story; you do not need embeddings.

### `write_file(path, content)`

New files only, or full rewrites the model explicitly chose. Keep it separate
from edit so your metrics don't blend them.

**Hard rule from this phase onward:** run Codey only against a throwaway
git clone, or inside a container. There is no approval layer until Phase 6.
Assume it will run `rm -rf` on something eventually.

## Phase 3 — Evals. Do not skip this, do not defer it

Time: 3–4 days. **Done when:** `go run ./evals -n 3` prints a pass rate and a
metrics table.

This is the pivot point of the whole project. Everything after this gets
measured; everything before it was guesswork.

Task format — one directory each:

```
evals/tasks/fix-nil-deref/
  setup.sh      # clone/copy a repo into a temp dir, apply a breaking patch
  prompt.txt    # what the user would type
  verify.sh     # exit 0 == task solved
```

Ten tasks, drawn from real repos you know. A good spread:

1. Fix a failing unit test (obvious cause)
2. Fix a failing unit test (cause is two files away from the failure)
3. Add a small feature with a test
4. Rename a symbol across 5+ files
5. Add error handling to an existing function
6. Fix a compile error from a partial refactor
7. Write a test for existing untested code
8. Find and fix a bug from a description only, no stack trace
9. Update a dependency call site after an API change
10. A task that *should* fail — verify it says so instead of claiming success

Runner requirements:

- Fresh temp dir per run, parallel across tasks, `-n 3` to repeat each (agents
  are nondeterministic; a single run tells you nothing).
- Persist the full transcript per run to `evals/runs/<ts>/<task>/<n>.json`. You
  will read these constantly. Most regressions are obvious within ten seconds
  of reading a trajectory.
- Print: pass rate, mean turns, mean input/output tokens, mean wall time, edit
  success rate, tool call counts by name.

Record the baseline in METRICS.md that day. Every later phase adds a row.

## Phase 4 — Context management, measured

Time: 4–5 days. **Done when:** a task requiring 40+ turns completes, and your
eval numbers didn't drop.

- Per-tool output caps (from Phase 2) — tune the numbers against evals rather
  than intuition.
- **Compaction:** at ~70% of the window, summarize the older half of the
  conversation into a single message. Preserve: the original task, files
  touched and why, decisions made, things discovered the hard way ("the config
  is loaded from X, not Y"), and the last few turns verbatim. Do it at 70%, not
  95% — compaction itself needs headroom.
- **The 97% floor:** the 70% rule keeps the last six messages, so it declines a
  history too short to cut six off — a few enormous tool results that have
  nevertheless filled the window. That restraint is right until the window is
  nearly full, at which point a verbatim tail buys nothing if the next request
  cannot be sent. Past `-compact-floor` (0.97), compact anyway with a
  one-message tail — the smallest that is still a valid conversation; if even
  that finds nothing to fold, say so rather than let the next request fail as an
  opaque API error. This is a backstop, not a second opinion about when to
  compact: 70% is still where it normally happens.
- **`/compact`:** the same fold on demand, for starting a long task with a
  clean slate. It waits for the current turn, since it rewrites the history
  that turn is building on, and past the floor it takes the smaller tail too —
  with auto-compaction off it is the only move left.
- **Turning it off:** `auto_compact: "off"` (or `-no-auto-compact`) for anyone
  who would rather lose the turn than the transcript. The floor then stops the
  turn instead of folding, refusing the prompt before it joins the history so
  that compacting and retrying does not send it twice. `/compact` is unaffected:
  the setting is about the automatic fold, not the manual one.
- **Stale reads:** if a file was read 20 turns ago and edited since, its old
  contents in history are actively harmful. Simplest fix that works: replace
  superseded `read_file` results with a stub pointing at the current version.
- **Prompt caching:** keep the prefix byte-stable — system prompt, then tool
  definitions, then project context, then the growing conversation. No
  timestamps, no session ids, no "files changed" block near the front. Add
  cache breakpoints and check `cache_read_input_tokens` in responses to confirm
  hits. This is your single biggest cost and latency lever, and compaction
  breaks it, which is another argument for compacting rarely.

Run evals after each change. Expect at least one of these to make things worse.

## Phase 5 — Streaming, interruption, and the interface

Time: 3–4 days. **Done when:** you can hit Ctrl-C mid-`go test` and give a
correction without losing the session.

- Switch `llm.Client` to SSE. The fiddly part is accumulating
  `input_json_delta` fragments per tool-call index into complete JSON before
  dispatch — buffer per block index, parse on `content_block_stop`.
- Ctrl-C cancels the current turn's context. The partially-run tool returns an
  error result, and control comes back to you for a follow-up message. Second
  Ctrl-C within a second or two exits.
- Show diffs before applying, with color.
- Show running token and cost totals per session. Cheap to add, and it changes
  how you design.
- Persist sessions to `.codey/sessions/<id>.json` and add `--resume`.

Keep `agent` free of terminal dependencies. The eval runner depends on that.

## Phase 6 — Safety

Time: 2–3 days. **Done when:** you'd run it on your actual working repo without
flinching.

- **Approval policy:** read-only tools auto-approve. `bash`, `write_file`,
  `edit_file` prompt, with "yes / no / always for this exact command / always
  for this tool" and session memory.
- **Do not allowlist shell commands with string matching.** `git status; rm -rf ~`
  defeats it, and so do backticks, `$()`, `&&`, `xargs`, and any interpreter
  with `-c`. Either parse with `mvdan.cc/sh/syntax` and check each command
  node, or don't pretend and rely on the sandbox plus prompts.
- **Path jail:** resolve every file path with `filepath.EvalSymlinks` and
  verify it's inside the working root. Cheap and catches a lot.
- **Sandbox:** the honest answer for a Go project on Linux is a container with
  the repo bind-mounted and network off by default. Decide explicitly about
  egress — that's the line between "broke my repo" and "read my .env and made
  an HTTP request."

## Phase 7 — Optional, pick at most one

Only after you've hit your eval target.

- Sub-agents: spawn a fresh context for a search task, return only the summary.
  Big context win, real complexity cost.
- Multi-provider (OpenAI, local models). Note that this means maintaining
  per-model prompt variants, not one abstraction.
- A CLAUDE.md-style project context file loaded into the cached prefix.
- Bubble Tea TUI.

## Handing phases to Claude Code

Give it one phase at a time, with the interfaces fixed. A prompt shape that
works:

> Implement Phase 2 of the plan in PLAN.md: the `edit_file` tool in
> `internal/tools/edit.go`. It must satisfy the `tools.Tool` interface already
> defined in `registry.go` — don't change that interface. Requirements: reject
> edits to files not read this session (use the tracker in `agent/state.go`);
> exact-match `old_str`; on zero matches retry with normalized whitespace and
> return the 20 surrounding lines of the closest candidate; on multiple matches
> return every match's line number and refuse; on success return a unified diff
> of the changed region; preserve the file's original line endings. Write
> table-driven tests covering all four match outcomes plus CRLF preservation.
> Run `go test ./internal/tools/...` and iterate until green.

What makes this work: one file, an interface it may not change, explicit
failure-mode requirements, and a command it can run to check itself. Vague
phase-level prompts ("build the tool system") produce plausible code that fails
in ways you won't find for a week.

## What you're actually producing

The code is the byproduct. The deliverable is METRICS.md.