Building a Coding Agent Harness in Go — Build Plan

Target: a harness that can be pointed at a real Go or TypeScript repo, told "make the failing test pass," and actually do it — with numbers proving how often it works.

Assumptions

• Single provider: Anthropic Messages API. Multi-provider is a later refactor, not a starting constraint.
• Go 1.22+. Generics are used for the tool registry.
• Terminal interface. No TUI framework at first — plain stdout, add Bubble Tea only if you want it later.
• Solo project, roughly 5–6 weeks part-time.

Stopping condition: 7 out of 10 eval tasks passing, or six weeks, whichever comes first. (See METRICS.md.)

Repo layout

harness/
  cmd/harness/main.go         # flag parsing, wiring, REPL entry
  internal/
    llm/
      client.go               # HTTP + SSE against the Messages API
      types.go                # Message, ContentBlock, ToolUse, ToolResult
      stream.go               # partial tool-call JSON accumulation
    agent/
      loop.go                 # the core turn loop
      state.go                # conversation history, file-read tracking
      compact.go              # summarization when the window fills
    tools/
      registry.go             # Tool interface + typed wrapper
      read.go glob.go grep.go
      edit.go write.go
      bash.go
      truncate.go             # shared output-capping helper
    approval/
      policy.go               # allow/deny/ask, with session memory
    metrics/
      log.go                  # JSONL event log — every tool call, every token count
    ui/
      render.go diff.go
  evals/
    tasks/                    # one dir per task: setup.sh, prompt.txt, verify.sh
    runner.go
  METRICS.md                  # you fill this in as you go. This is the deliverable.

Two rules that save pain later:

1. agent never imports ui. The loop must be runnable headless, because the eval runner drives it.
2. Every tool returns (string, error) where the error is for the model to read, not for you to log and swallow. A failed edit is a normal event in the transcript, not an exception.

Phase 0 — Skeleton and the tool registry

Time: half a day. Done when: go test ./... passes with a fake tool registered and dispatched.

The one piece of Go design worth getting right on day one is typed tool arguments, because otherwise you'll be doing map[string]any assertions in every tool for the rest of the project.

Derive schemaFor[T]() from struct tags with github.com/invopop/jsonschema rather than hand-writing JSON Schema. Hand-written schemas drift from the structs and you won't notice until the model sends a field you don't parse.

Also in this phase: metrics.Log(event) writing one JSON object per line to ./.harness/events.jsonl. Wire it before you have anything to measure. Retrofitting instrumentation is how projects end up with no numbers.

Phase 1 — The minimal loop

Time: 1–2 days. Done when: you ask "what does main.go do?" and the agent reads the file and answers.

Non-streaming first. Streaming is a distraction until the loop is correct.

build messages → POST /v1/messages → response
  if stop_reason == "tool_use":
      for each tool_use block: dispatch, collect result
      append assistant message + user message containing tool_result blocks
      repeat
  else: print text, return

Things that bite here:

• Tool results go back as a user message containing tool_result blocks, one per tool_use id, in a single message. Multiple parallel tool calls means multiple result blocks in that one message — not one message each.
• Every tool_use id must get a result, even if execution failed. Send the error text with is_error: true. A missing result is an API error.
• Set a max-turn cap (start at 30) and return a clear message when hit. Unbounded loops burn money fast.
• Thread context.Context from main through the loop into every tool from the very start. Retrofitting cancellation later means touching every signature.

Only one tool this phase: read_file(path, offset?, limit?).

Phase 2 — The real tool set

Time: 1 week. Done when: the agent can find code it hasn't been shown, change it, and run the tests.

edit_file(path, old_str, new_str) — the centerpiece

• Reject if the file hasn't been read this session. Track map[string]time.Time of reads; if the file's mtime changed since, force a re-read. This single rule kills a large class of hallucinated edits.
• Count exact occurrences of old_str.
  • 1 match → apply. Return a short unified diff of the changed region, not "OK". The model uses the diff to verify it did what it meant.
  • 0 matches → do not just say "not found." Retry the match with normalized whitespace; if that hits, tell the model the indentation differs and show the actual line. If it still misses, return the 20 lines around the closest-matching line. A failure that carries context gets fixed in one turn; a bare "not found" costs three.
  • >1 match → return the line numbers of every match and ask for a longer unique span. Never silently pick the first.
• Preserve the file's existing line endings. Detect on read, restore on write.
• Log every attempt to the metrics file with the outcome. Edit success rate is your headline metric.

bash(command, timeout_ms)

• Run via exec.CommandContext so cancellation works.
• Cap output. Use head+tail (e.g. first 100 and last 100 lines with a … N lines elided … marker), never head alone — test failures live at the end.
• Default timeout 60s; kill the process group, not just the process, or orphaned children hang the loop.
• Return exit code, stdout and stderr together, clearly labelled.

glob(pattern) and grep(pattern, path?, glob?)

Shell out to rg if present, fall back to filepath.WalkDir + regexp. Cap results at ~100 with a count of how many were suppressed. This is your entire code-search story; you do not need embeddings.

write_file(path, content)

New files only, or full rewrites the model explicitly chose. Keep it separate from edit so your metrics don't blend them.

Hard rule from this phase onward: run the harness only against a throwaway git clone, or inside a container. There is no approval layer until Phase 6. Assume it will run rm -rf on something eventually.

Phase 3 — Evals. Do not skip this, do not defer it

Time: 3–4 days. Done when: go run ./evals -n 3 prints a pass rate and a metrics table.

This is the pivot point of the whole project. Everything after this gets measured; everything before it was guesswork.

Task format — one directory each:

evals/tasks/fix-nil-deref/
  setup.sh      # clone/copy a repo into a temp dir, apply a breaking patch
  prompt.txt    # what the user would type
  verify.sh     # exit 0 == task solved

Ten tasks, drawn from real repos you know. A good spread:

1. Fix a failing unit test (obvious cause)
2. Fix a failing unit test (cause is two files away from the failure)
3. Add a small feature with a test
4. Rename a symbol across 5+ files
5. Add error handling to an existing function
6. Fix a compile error from a partial refactor
7. Write a test for existing untested code
8. Find and fix a bug from a description only, no stack trace
9. Update a dependency call site after an API change
10. A task that should fail — verify it says so instead of claiming success

Runner requirements:

• Fresh temp dir per run, parallel across tasks, -n 3 to repeat each (agents are nondeterministic; a single run tells you nothing).
• Persist the full transcript per run to evals/runs/<ts>/<task>/<n>.json. You will read these constantly. Most regressions are obvious within ten seconds of reading a trajectory.
• Print: pass rate, mean turns, mean input/output tokens, mean wall time, edit success rate, tool call counts by name.

Record the baseline in METRICS.md that day. Every later phase adds a row.

Phase 4 — Context management, measured

Time: 4–5 days. Done when: a task requiring 40+ turns completes, and your eval numbers didn't drop.

• Per-tool output caps (from Phase 2) — tune the numbers against evals rather than intuition.
• Compaction: at ~70% of the window, summarize the older half of the conversation into a single message. Preserve: the original task, files touched and why, decisions made, things discovered the hard way ("the config is loaded from X, not Y"), and the last few turns verbatim. Do it at 70%, not 95% — compaction itself needs headroom.
• The 97% floor: the 70% rule keeps the last six messages, so it declines a history too short to cut six off — a few enormous tool results that have nevertheless filled the window. That restraint is right until the window is nearly full, at which point a verbatim tail buys nothing if the next request cannot be sent. Past -compact-floor (0.97), compact anyway with a one-message tail — the smallest that is still a valid conversation; if even that finds nothing to fold, say so rather than let the next request fail as an opaque API error. This is a backstop, not a second opinion about when to compact: 70% is still where it normally happens.
• /compact: the same fold on demand, for starting a long task with a clean slate. It waits for the current turn, since it rewrites the history that turn is building on, and past the floor it takes the smaller tail too — with auto-compaction off it is the only move left.
• Turning it off: auto_compact: "off" (or -no-auto-compact) for anyone who would rather lose the turn than the transcript. The floor then stops the turn instead of folding, refusing the prompt before it joins the history so that compacting and retrying does not send it twice. /compact is unaffected: the setting is about the automatic fold, not the manual one.
• Stale reads: if a file was read 20 turns ago and edited since, its old contents in history are actively harmful. Simplest fix that works: replace superseded read_file results with a stub pointing at the current version.
• Prompt caching: keep the prefix byte-stable — system prompt, then tool definitions, then project context, then the growing conversation. No timestamps, no session ids, no "files changed" block near the front. Add cache breakpoints and check cache_read_input_tokens in responses to confirm hits. This is your single biggest cost and latency lever, and compaction breaks it, which is another argument for compacting rarely.

Run evals after each change. Expect at least one of these to make things worse.

Phase 5 — Streaming, interruption, and the interface

Time: 3–4 days. Done when: you can hit Ctrl-C mid-go test and give a correction without losing the session.

• Switch llm.Client to SSE. The fiddly part is accumulating input_json_delta fragments per tool-call index into complete JSON before dispatch — buffer per block index, parse on content_block_stop.
• Ctrl-C cancels the current turn's context. The partially-run tool returns an error result, and control comes back to you for a follow-up message. Second Ctrl-C within a second or two exits.
• Show diffs before applying, with color.
• Show running token and cost totals per session. Cheap to add, and it changes how you design.
• Persist sessions to .harness/sessions/<id>.json and add --resume.

Keep agent free of terminal dependencies. The eval runner depends on that.

Phase 6 — Safety

Time: 2–3 days. Done when: you'd run it on your actual working repo without flinching.

• Approval policy: read-only tools auto-approve. bash, write_file, edit_file prompt, with "yes / no / always for this exact command / always for this tool" and session memory.
• Do not allowlist shell commands with string matching. git status; rm -rf ~ defeats it, and so do backticks, $(), &&, xargs, and any interpreter with -c. Either parse with mvdan.cc/sh/syntax and check each command node, or don't pretend and rely on the sandbox plus prompts.
• Path jail: resolve every file path with filepath.EvalSymlinks and verify it's inside the working root. Cheap and catches a lot.
• Sandbox: the honest answer for a Go project on Linux is a container with the repo bind-mounted and network off by default. Decide explicitly about egress — that's the line between "broke my repo" and "read my .env and made an HTTP request."

Phase 7 — Optional, pick at most one

Only after you've hit your eval target.

• Sub-agents: spawn a fresh context for a search task, return only the summary. Big context win, real complexity cost.
• Multi-provider (OpenAI, local models). Note that this means maintaining per-model prompt variants, not one abstraction.
• A CLAUDE.md-style project context file loaded into the cached prefix.
• Bubble Tea TUI.

Handing phases to Claude Code

Give it one phase at a time, with the interfaces fixed. A prompt shape that works:

Implement Phase 2 of the plan in PLAN.md: the edit_file tool in internal/tools/edit.go. It must satisfy the tools.Tool interface already defined in registry.go — don't change that interface. Requirements: reject edits to files not read this session (use the tracker in agent/state.go); exact-match old_str; on zero matches retry with normalized whitespace and return the 20 surrounding lines of the closest candidate; on multiple matches return every match's line number and refuse; on success return a unified diff of the changed region; preserve the file's original line endings. Write table-driven tests covering all four match outcomes plus CRLF preservation. Run go test ./internal/tools/... and iterate until green.

What makes this work: one file, an interface it may not change, explicit failure-mode requirements, and a command it can run to check itself. Vague phase-level prompts ("build the tool system") produce plausible code that fails in ways you won't find for a week.

What you're actually producing

The code is the byproduct. The deliverable is METRICS.md.