A coding agent harness in Go. Point it at a repository, tell it to make the failing test pass, and measure how often it actually does.
The code is the byproduct. The deliverable is METRICS.md. PLAN.md is the build plan this follows.
go build -o harness ./cmd/harness
./harness -C /path/to/a/throwaway/clone # full-screen TUI
./harness -C /path/to/repo -p "why does TestFoo fail?" # one prompt
If you are logged into Claude Code, that is all the setup there is: the harness
borrows the OAuth credential from ~/.claude/.credentials.json when nothing
else supplied one. Otherwise, or to use a different account:
export ANTHROPIC_API_KEY=sk-ant-...
A different endpoint — a gateway, a proxy, a local server speaking the Messages API:
./harness -base-url http://127.0.0.1:4000 -api-key ... -C .
# or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN
Flags beat the saved config, which beats the built-in defaults; the config is
what the setup wizard writes on a first run. -auth-token sends
Authorization: Bearer, which is what OAuth tokens and most gateways want;
-api-key sends x-api-key. A non-default endpoint announces itself on
stderr, because your prompt and every file the agent reads go wherever that URL
points.
The Claude Code login is the last resort, tried only when no flag and no
environment variable supplied a credential, and only against Anthropic's own
endpoint — a credential found on disk is not one you chose to send, so it is
never forwarded to a -base-url you pointed the harness at. -no-claude-code-auth
skips it entirely. Note that the credential is issued to Claude Code; using it
from another client is outside Anthropic's terms for it, which API keys and
gateway tokens do not carry.
On macOS Claude Code keeps this in the Keychain rather than a file, so there is nothing to borrow there. An expired access token is refreshed automatically; the credentials file is left alone unless the server rotates the refresh token, in which case the replacement is written back — leaving the dead one there is what would break your Claude Code login.
Useful flags: -model, -effort low|medium|high|xhigh|max, -max-turns,
-resume latest, -no-stream, -no-cache, -no-auto-compact, -v, -yolo,
-no-tui, -no-skills, -no-claude-code-auth.
-compact-at (default 0.7) is where the conversation is normally folded into a
summary, keeping the last six messages verbatim. Early and rare is deliberate:
compaction is itself a model call with the history in it, and it is the single
biggest thing that breaks prompt caching.
-compact-floor (default 0.97) is the fraction past which compaction stops
being optional. The normal threshold declines a history it cannot split safely —
a handful of enormous tool results is too few messages to cut six off the end of
— and below the floor that restraint is right. Past it, a verbatim tail is worth
nothing if the next request has no room to be sent, so the floor folds anyway
and settles for a one-message tail. /compact follows the same rule past the
floor, since with auto-compaction off it is the only move left. When even that
finds nothing to fold, the harness says so, rather than letting the next
request fail as a bare API error.
-context-window (default 1,000,000) is what both fractions are measured
against; lower it for a model with less room.
-no-auto-compact turns the automatic fold off for one run, the same as
"auto_compact": "off" in the config. The floor then stops the turn instead of
folding: the session stays intact, nothing is summarized behind your back, and
/compact is what makes room. In -p mode the stop is a non-zero exit.
On a terminal, the interactive mode is a full-screen Bubble Tea app. It opens on a welcome block naming the model, the endpoint and the root, and settles into a transcript above a prompt:
⏺ Let me see what the test expects before changing anything.
⏺ ReadFile(path=internal/agent/loop_test.go)
⎿ 1: package agent
…
⏺ EditFile(path=internal/agent/compact.go)
4 additions, 1 removal
@@ -50,7 +50,9 @@
+ if window == 0 {
✢ Noodling… (14s · turn 3 · ↓ 1.3k tokens · esc to interrupt) ● high · /effort
╭──────────────────────────────────────────────────────────────────────────────╮
│ › _ │
╰──────────────────────────────────────────────────────────────────────────────╯
joni ✳ Opus 5 ◐ 62.2k (6%) ↑ 4.2k ⟳ 58.0k ↓ 1.3k $0.0835
⏵ approvals on (shift+tab to cycle) · /help for commands
Every event hangs off a dot in the left gutter, and everything belonging to it is indented under that dot, so a screen of tool output still reads as a sequence of things that happened. Diffs are shown in full and in colour with a count of what changed; other output is elided after six lines.
What the model says is Markdown, and it is rendered as Markdown — headings, weight, lists, block quotes, rules, and tables laid out in columns:
⏺ Findings
The refresh path drops the rotated token. Three things are going on:
1. refresh() returns the new pair but never writes it back
2. the old refresh token is already dead by then
File │ Line │ Change
────────────────────────┼───────────────────────┼───────────────────────
refresh.go │ 84 │ write back
func refresh(tok string) (Creds, error) {
// TODO: persist the rotation
}
Fenced code is syntax-highlighted by chroma,
and every colour — prose, headings, keywords, strings — comes from the theme in
force, so /setup repaints Markdown along with everything else. Headings drop
their hashes: the levels are told apart by colour and weight instead. It
renders as it streams, so a table fills in row by row.
Only what the model says goes through this. Tool results and the output of your
own ! and $ commands are program output and are shown as they came, and
-no-tui prints raw text so -p stays pipeable.
The status bar reports the context gauge — the last turn's input size against the window, which drops when the conversation is compacted — and separates fresh input tokens from cache reads, because the gap between the two is what METRICS.md is about. The prompt box grows with what you type, up to six lines.
The conversation folds itself into a summary at 70% of the window, and
/compact does it on demand — before a long task, say, or when the gauge is
climbing faster than the work is. /compact waits for the current turn to
finish, because it rewrites the history that turn is building on. Afterwards
the gauge goes blank rather than keep reporting a number the compaction just
made untrue; the next turn measures the window again and it comes back. At 97%
compaction stops being optional, and it can be turned off altogether: see
the flags below.
| Key | Does |
|---|---|
enter | send |
/ | open the command menu — see below |
! | run a shell command yourself — see below |
$ | the same, through python3 |
ctrl+j | newline without sending |
shift+tab | cycle the approval mode |
ctrl+r | expand elided tool output |
pgup/pgdn | scroll the transcript |
esc | interrupt the turn |
ctrl+c | interrupt the turn; again when idle to exit |
Commands: /allowed, /clear, /compact, /effort, /exit, /help,
/model, /mouse, /settings, /setup, /skills, and one per discovered
skill.
| Does | How |
|---|---|
| scroll the transcript | the wheel, anywhere |
| select | drag; double-click a word, triple-click a line, shift+click to extend |
| copy | let go — the selection lands on the clipboard on release |
| choose | click a menu row, a model, the effort slider, an approval, the mode line |
The prompt box is keyboard-driven: a click does not move the caret in it, and what you have typed is the one thing the mouse cannot select. Dragging against the top or bottom edge of the transcript scrolls, so a selection can be longer than the screen; it keeps going while the pointer moves, because a terminal reporting button events says nothing about a hand held still.
The harness runs in the alternate screen, where the terminal has no scrollback of its own to roll — so asking for mouse reporting is what makes the wheel work at all. The cost is the terminal's own text selection, which stops being the terminal's the moment drags are reported to the program. That is why the selection above exists: it is the replacement, and copying goes out over both OSC 52 (which works over ssh) and the desktop clipboard, because terminals disagree about which of the two they allow.
/mouse hands the mouse back. Selection and copy become the terminal's again,
exactly as before this existed, and the wheel goes quiet. shift+drag usually
reaches the terminal's own selection without turning reporting off at all.
A slash at the start of an empty prompt opens the command menu, which filters as you type — a name that starts with what you typed before one that merely contains it:
╭──────────────────────────────────────────────────────────────────────────────╮
│ › /s │
╰──────────────────────────────────────────────────────────────────────────────╯
❯ /skills the instruction sets found at startup
/systematic-debugging find the cause before changing code (project)
/brainstorming explore an idea before building it (personal)
↑/↓ move, tab or enter completes the highlighted name into the prompt —
completing is not sending, so there is room to type arguments for a skill or a
level for /effort — and esc dismisses the menu without touching what you
typed. The menu offers exactly the commands the TUI answers, so nothing it
suggests can come back as "unknown command".
! on an empty prompt points the box at a shell instead of at the model; $
points it at python3. The marker replaces the ›, the frame changes colour,
and enter runs the line where the agent's own bash tool would have run it —
from the working root, with the same timeout and the same output cap.
╭──────────────────────────────────────────────────────────────────────────────╮
│ ! go test ./internal/agent/ │
╰──────────────────────────────────────────────────────────────────────────────╯
⏺ ! go test ./internal/agent/
⎿ exit code: 1 (2.3s)
--- stdout ---
--- FAIL: TestCompactKeepsTheLastExchange (0.00s)
compact_test.go:88: history has 2 messages, want 4
… +9 more lines (ctrl+r)
The agent is shown what you saw. The command and its output are appended to the conversation as a user message, so the next thing you type can be "fix that" and the model already has the failure. Nothing is sent at that moment — it waits in the history and goes up with your next real turn, so several commands can be run before saying anything about them.
What reaches the model is the capped report, not the raw firehose: a noisy
command cannot cost more context from the prompt box than it would through the
tool. The transcript collapses it further to six lines, and ctrl+r expands to
exactly what the model was given, so the two never drift.
esc leaves the mode, as does backspace over the marker. The mode is
per-command — after a run the prompt returns to normal, the way a shell prompt
comes back. One command runs at a time; while it does, esc interrupts it and
a message you type waits rather than jumping ahead of the output it is about.
A ! typed mid-sentence is an exclamation mark.
These bypass the approval policy by design: you typed the command, so there is nobody left to ask. They are also the one part of the TUI that runs something without the model's involvement — worth remembering when the root is not a throwaway clone.
/model asks the endpoint what it serves — GET /v1/models, paginated — and
opens a picker on the answer:
Select model
from anthropic · ↑↓ move · enter apply · esc cancel
1. Claude Fable 5.1 claude-fable-5-1 · 1.0M in · $10 / $50 per Mtok
2. Claude Opus 5 claude-opus-5 · 1.0M in · $5 / $25 per Mtok ✓
❯ 3. Claude Sonnet 5 claude-sonnet-5 · 1.0M in · $2 / $10 per Mtok
4. Claude Haiku 4.5 claude-haiku-4-5-20251001 · 200.0k in · $1 / $5 per Mtok
The cursor starts on the model in force, which is also the one marked. The id, the context window and the list price come from the endpoint and from the pricing table; a model neither knows about simply says less. The switch applies from the next turn — the conversation so far is replayed to the new model — and costs already incurred stay priced at the model that incurred them, because a session that switched models was not all billed at the model it ended on.
Not every endpoint serves that route. When the fetch fails the picker says so,
names the host and the reason, and closes; /model <id> and the -model flag
still switch without asking anyone's permission.
/effort opens a slider in place of the prompt, on the level in force — an
unset level is the API's own default, high, so the marker starts there rather
than at an end:
Faster Smarter
──────────────────────────────────────▲───────────────────────────────────
low medium high xhigh max
←→ move · 1-5 jump · enter apply · esc cancel
Moving the marker changes nothing until enter; esc leaves the level alone.
/effort xhigh still applies a level directly for anyone who can type it, and
either way it is the -effort flag reached from the keyboard. The level in
force shows above the prompt on the right. A terminal too narrow for five
labels gets ◀ high ▶ and the same keys rather than a track with the words
sliced in half.
/model and /effort change the session. /settings and /setup change what
the next session starts with — see Setup.
shift+tab cycles between asking before every mutating call and accepting them
all, which is -yolo reached from the keyboard. The mode line says which is in
force at all times, and the switch is recorded in the transcript, because a
mode that silently lets the agent write to disk is not one to keep in a corner.
Approvals appear inline as a bordered prompt, answered by number or by the key
in brackets — y, n, a for this exact command, t for the whole tool. The
agent goroutine blocks until you answer, which is the point: the tool has not
run yet. Interrupting while that prompt is open denies the call and cancels the
turn rather than leaving the agent waiting on a channel nobody will read.
The TUI is skipped automatically when stdin or stdout is not a terminal, and by
-p and -no-tui, which use the line-based renderer instead. Both front ends
consume the same agent.Event stream, so neither can show something the other
cannot.
The first run on a terminal opens a full-screen wizard instead of the chat: the
mark assembles, the name lights up, and it asks four questions one at a time.
"First run" means the config file does not say "setup": "done" — the marker
the wizard stamps when the last question is answered — rather than that no file
exists, so a config written by hand, or by a build with fewer questions in it,
still gets asked the ones it is missing.
harness setup step 1 of 4
◆━━━━━━━━━━━━━━━◇───────────────◇───────────────◇
Which model should the agent use?
▌ ◆ Claude Opus 5 $5 / $25 per Mtok
◇ Claude Sonnet 5 $2 / $10 per Mtok
◇ Claude Haiku 4.5 $1 / $5 per Mtok
◇ custom…
claude-opus-5 · 1.0M token context
↑↓ move enter next esc back ctrl+c cancel
| Question | Answers |
|---|---|
| Model | whatever GET /v1/models returns, plus custom… for any id |
| Effort | the API's own llm.EffortLevels: low medium high xhigh max |
| Approvals | ask before every mutating call, or accept them all |
| Theme | ember mint iris mono |
It is the same catalogue and the same pricing table /model uses, through the
same GET /v1/models call, so the two cannot disagree about what a model is
called or what it costs. Where they differ is failure: /model reports the
error and closes, because there is a session to go back to. The wizard cannot
do that on a first run, so it falls back to the models this build can price and
says so. custom… is there because a gateway may serve something no build has
heard of.
The theme step previews by applying the palette for real — reading a swatch tells you nothing about what a transcript looks like in it — and it recolours the whole TUI, pickers and slider included. Cancelling puts the old one back.
Answers land in ~/.config/harness/config.json (XDG_CONFIG_HOME is
honoured):
{
"version": 1,
"model": "claude-opus-5",
"effort": "high",
"approvals": "ask",
"theme": "ember",
"auto_compact": "on",
"setup": "done"
}
auto_compact is the one key neither the wizard nor /settings offers a row
for — set it by hand, or pass -no-auto-compact for a single run. Both screens
write the whole file but only touch what they asked about, so it survives them.
Set to "off", the conversation is never folded on its own: the window becomes
a wall rather than a trigger, and at -compact-floor the turn stops before
sending a request that would not fit,
naming /compact and /clear as the ways on. The prompt is refused before it
joins the history, so compacting and asking again sends it once rather than
twice. /compact keeps working throughout — the setting governs the automatic
fold, not the manual one. Anything other than "on" or "off" is dropped, and
a missing key means on, so a config written before this existed does not
suddenly start stopping the agent.
Nothing in that file is required, and none of it wins an argument with a flag:
precedence is flag, then config, then the built-in default, so
-model claude-sonnet-5 is still a one-run override. accept-all saved as a
default is honoured, but -yolo can only turn approvals off, never on — a file
should not be able to quietly grant the agent more than the command line did.
-no-auto-compact runs the same way round: the flag can only turn the automatic
fold off, and a config that never mentions it leaves it on.
A value this build does not recognise is dropped rather than forwarded to the
API, and effort is checked against llm.EffortLevels rather than a second copy
of the same five strings.
/setup reopens the wizard at any time. Finishing applies the answers through
the same setters /model and /effort use, so a setting chosen in the wizard
and the same setting chosen from the prompt cannot come out different — the
wizard just reports them in one line instead of three. It is refused mid-turn:
changing the model while a turn is in flight would finish it on a different
model than it started. A config file that cannot be parsed is reported and then
ignored, and is never overwritten until you finish the wizard.
/model and /effort change the running session and say "from the next turn
on"; they do not write to the file. The two screens that do are the wizard and
/settings.
/settings is the same four settings without the interview: one row each,
changed in place, applied as you move.
Settings ↑↓ row · ←→ change · esc done
❯ Model Opus 5 $5 / $25 per Mtok
Effort high default
Approvals Ask me first recommended
Theme ember
claude-opus-5 · enter to pick from anthropic
saved to ~/.config/harness/config.json
←→ cycles a row and applies it there and then — the theme repaints under the
panel, the way it previews in the wizard — and esc closes and writes the
file. There is no cancel, because there is nothing left to confirm: the session
already has every change. A panel that was only read leaves the file alone, and
so does a value cycled back to where it started.
The rows come from the same option lists the wizard's steps are built from, so
neither screen can invent a name for a theme or a sentence about an effort
level the other does not use. The model row is the exception: its choices come
from the endpoint rather than from this build, so enter hands over to the
/model picker and lands back here once you have chosen.
What the panel shows is the session, not the file — /effort, shift+tab
and -yolo all change a setting without the file hearing about it, and a panel
that disagreed with the status bar two lines below it would be worse than
useless. Only the rows actually touched are written back, which is what keeps a
-yolo run from quietly saving "accept everything" as the default for every
run after it. And the setup marker is not stamped here: changing a theme is
not answering the setup questions, so someone who has only ever used this panel
is still offered the wizard.
The whole thing is ↑↓ and enter, esc to go back. Animations collapse to
their final frame under NO_COLOR, a dumb terminal, or a window under 60×20,
and any key skips the one playing.
| Tool | Approval | Notes |
|---|---|---|
read_file | automatic | Numbered lines, offset/limit for paging |
glob | automatic | Newest first; *.go matches at any depth |
grep | automatic | ripgrep when present, pure Go otherwise, identical output either way |
edit_file | prompts | Requires a prior read; refuses ambiguous matches; returns a diff |
write_file | prompts | Whole files only; separate from edit_file so the metrics stay honest |
bash | prompts | Own process group, killed whole on timeout; output capped head and tail |
A skill is a directory of instructions the agent pulls in when it is relevant:
~/.agents/skills/commit-helper/SKILL.md every repo
./.agents/skills/commit-helper/SKILL.md this one, checked in
---
name: commit-helper
description: Use when writing commit messages for this repo
---
Read the last twenty commits for tone before writing anything.
Conventions are in references/conventions.md.
The directory name is the skill's name, and a name: that disagrees with it is
a load error. Only the description reaches the model until the skill is
invoked, so it is what decides whether the skill fires at all; the body can be
as long as you like.
Two ways in, one way to load. The agent calls the skill tool when a
description matches what it is doing; you can force one with /commit-helper fix the flaky test, which the model answers by making the same call. /skills
lists what was found, project shadowing personal on a name collision.
Skill directories are readable by read_file, glob and grep even though
they sit outside the working root, so a skill can ship reference files and
point at them. They are not writable by anything: edit_file, write_file and
bash keep the working root as their only root. A skill is input, not a work
product.
Relative paths still anchor to the working root, so a skill pointing at its own
files is read with the directory the skill tool prints joined on. If the model
forgets, the "no such file" it gets back names the skill directories, which is
usually enough to recover on the next turn. A shadowed skill's directory is not
readable — only the skills actually in play widen anything.
Discovery happens at startup. The name-and-description listing goes in the
cached prompt prefix, so re-reading the directory each turn would cost a cache
miss on every turn for a file that never changes. Restart to pick up a new
skill. -no-skills turns the whole thing off; the eval runner never discovers
skills at all, so METRICS.md is unaffected by what is in your home directory.
./evals/selftest.sh # no API calls: checks the tasks measure something
go run ./evals -n 3 # the real thing
Ten tasks, a fresh temp directory per run, full transcripts under
evals/runs/<timestamp>/. See METRICS.md.
Three layers, in increasing order of how much they actually protect you.
Path jail. Every path a tool is given is resolved with
filepath.EvalSymlinks and refused if it lands outside the working root,
including through a symlink that points out of the tree. Cheap, and it catches
a lot.
Approval prompts. Read-only tools run unattended. bash, edit_file and
write_file ask, with "yes / no / always this exact one / always this tool" and
session memory. /allowed lists what you have agreed to.
Shell commands are parsed with mvdan.cc/sh/syntax so the prompt can tell you
what a command will actually run — git status; rm -rf ~ is shown as running
both git and rm. This is not a security boundary, and the harness does
not pretend otherwise: a command containing $(...), backticks, or an
interpreter with -c cannot be analysed at all, so it is marked opaque and
"always allow this exact command" is refused for it. String-matching
allowlists for shell commands do not work; this exists to make the prompt
specific, not to make the decision for you.
A sandbox. This is the layer that actually contains a mistake, and the harness cannot provide it for you. The honest recipe on Linux:
podman run --rm -it \
-v "$PWD":/work:Z -w /work \
-e ANTHROPIC_API_KEY \
docker.io/library/golang:1.26 \
/work/harness -C /work -yolo
Decide about egress explicitly. The container above has network access,
because the harness needs to reach the API — which means the agent's bash
has it too. That is the line between "broke my repo" and "read my .env and
made an HTTP request". If that matters for the tree you are pointing it at:
bash in a
network-less namespace. That is not implemented here.-yolo skips every prompt. It is correct inside a container or against a
throwaway clone, and wrong everywhere else. The eval runner sets it, because it
is headless and operates on a temp directory it created.
cmd/harness flag parsing, wiring, REPL
internal/llm Messages API: types, HTTP, SSE, pricing
internal/agent turn loop, conversation state, compaction, sessions
internal/tools the tool registry and the tools
internal/skills discovery and parsing of SKILL.md
internal/config saved defaults: model, effort, approvals, theme, compaction
internal/approval allow / deny / ask, with shell analysis
internal/metrics JSONL event log
internal/ui line-based terminal rendering
internal/tui full-screen Bubble Tea front end, and the setup wizard
evals task suite and runner
Two rules hold the shape: agent never imports ui or tui, so the eval
runner can drive the loop headless; and every tool returns (string, error)
where the error is text the model reads, not an exception the harness swallows.
A failed edit is a normal event in the transcript.
The TUI obeys the first rule through two one-way hatches. Events reach it via
agent.Observe, forwarded into the Bubble Tea loop with Program.Send, which
is safe from the agent's goroutine. Approvals go out the same way and then
block the agent on a reply channel until a key arrives. That channel is
buffered and the wait honours the turn's context, so neither side can strand
the other.