# harness

A coding agent harness in Go. Point it at a repository, tell it to make the
failing test pass, and measure how often it actually does.

The code is the byproduct. The deliverable is [METRICS.md](METRICS.md).
[PLAN.md](PLAN.md) is the build plan this follows.

## Quick start

```bash
go build -o harness ./cmd/harness

./harness -C /path/to/a/throwaway/clone          # full-screen TUI
./harness -C /path/to/repo -p "why does TestFoo fail?"   # one prompt
```

If you are logged into Claude Code, that is all the setup there is: the harness
borrows the OAuth credential from `~/.claude/.credentials.json` when nothing
else supplied one. Otherwise, or to use a different account:

```bash
export ANTHROPIC_API_KEY=sk-ant-...
```

A different endpoint — a gateway, a proxy, a local server speaking the Messages
API:

```bash
./harness -base-url http://127.0.0.1:4000 -api-key ... -C .
# or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN
```

Flags override the environment. `-auth-token` sends `Authorization: Bearer`,
which is what OAuth tokens and most gateways want; `-api-key` sends
`x-api-key`. A non-default endpoint announces itself on stderr, because your
prompt and every file the agent reads go wherever that URL points.

The Claude Code login is the last resort, tried only when no flag and no
environment variable supplied a credential, and only against Anthropic's own
endpoint — a credential found on disk is not one you chose to send, so it is
never forwarded to a `-base-url` you pointed the harness at. `-no-claude-code-auth`
skips it entirely. Note that the credential is issued to Claude Code; using it
from another client is outside Anthropic's terms for it, which API keys and
gateway tokens do not carry.

On macOS Claude Code keeps this in the Keychain rather than a file, so there is
nothing to borrow there. An expired access token is not refreshed — run `claude`
and the harness picks up the new one.

Useful flags: `-model`, `-effort low|medium|high|xhigh|max`, `-max-turns`,
`-resume latest`, `-no-stream`, `-no-cache`, `-v`, `-yolo`, `-no-tui`, `-no-skills`,
`-no-claude-code-auth`.

## The TUI

On a terminal, the interactive mode is a full-screen [Bubble Tea](https://github.com/charmbracelet/bubbletea)
app. It opens on a welcome block naming the model, the endpoint and the root,
and settles into a transcript above a prompt:

```
⏺ Let me see what the test expects before changing anything.

⏺ ReadFile(path=internal/agent/loop_test.go)
  ⎿ 1: package agent
    …

⏺ EditFile(path=internal/agent/compact.go)
  4 additions, 1 removal
  @@ -50,7 +50,9 @@
  +    if window == 0 {

 ✢ Noodling… (14s · turn 3 · ↓ 1.3k tokens · esc to interrupt)   ● high · /effort
╭──────────────────────────────────────────────────────────────────────────────╮
│ › _                                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯
 joni   ✳ Opus 5   ◐ 62.2k (6%)   ↑ 4.2k ⟳ 58.0k ↓ 1.3k  $0.0835
 ⏵ approvals on (shift+tab to cycle) · /help for commands
```

Every event hangs off a dot in the left gutter, and everything belonging to it
is indented under that dot, so a screen of tool output still reads as a
sequence of things that happened. Diffs are shown in full and in colour with a
count of what changed; other output is elided after six lines.

The status bar reports the context gauge — the last turn's input size against
the window, which drops when the conversation is compacted — and separates
fresh input tokens from cache reads, because the gap between the two is what
[METRICS.md](METRICS.md) is about. The prompt box grows with what you type, up
to six lines.

| Key | Does |
| --- | --- |
| `enter` | send |
| `ctrl+j` | newline without sending |
| `shift+tab` | cycle the approval mode |
| `ctrl+r` | expand elided tool output |
| `pgup`/`pgdn` | scroll the transcript |
| `esc` | interrupt the turn |
| `ctrl+c` | interrupt the turn; again when idle to exit |

Commands: `/allowed`, `/clear`, `/help`, `/exit`, `/skills`.

`shift+tab` cycles between asking before every mutating call and accepting them
all, which is `-yolo` reached from the keyboard. The mode line says which is in
force at all times, and the switch is recorded in the transcript, because a
mode that silently lets the agent write to disk is not one to keep in a corner.

Approvals appear inline as a bordered prompt, answered by number or by the key
in brackets — `y`, `n`, `a` for this exact command, `t` for the whole tool. The
agent goroutine blocks until you answer, which is the point: the tool has not
run yet. Interrupting while that prompt is open denies the call and cancels the
turn rather than leaving the agent waiting on a channel nobody will read.

The TUI is skipped automatically when stdin or stdout is not a terminal, and by
`-p` and `-no-tui`, which use the line-based renderer instead. Both front ends
consume the same `agent.Event` stream, so neither can show something the other
cannot.

### Driving it without an API key

`internal/tui/testdata/mockapi` is a fake Messages API that streams a canned
reply and asks to run a shell command, which is enough to exercise streaming,
tool rendering, approve, deny and interrupt for nothing:

```bash
go run ./internal/tui/testdata/mockapi &
mkdir -p /tmp/demo && git -C /tmp/demo init -q
go build -o harness ./cmd/harness
./harness -C /tmp/demo -base-url http://127.0.0.1:8899 -api-key mock
```

Type anything, and the approval prompt appears. `-slow 150ms` stretches the
stream out if you want to watch the spinner or test `ctrl+c` mid-turn. It sits
under `testdata` so `./...` ignores it.

## Tools

| Tool | Approval | Notes |
| --- | --- | --- |
| `read_file` | automatic | Numbered lines, `offset`/`limit` for paging |
| `glob` | automatic | Newest first; `*.go` matches at any depth |
| `grep` | automatic | ripgrep when present, pure Go otherwise, identical output either way |
| `edit_file` | prompts | Requires a prior read; refuses ambiguous matches; returns a diff |
| `write_file` | prompts | Whole files only; separate from `edit_file` so the metrics stay honest |
| `bash` | prompts | Own process group, killed whole on timeout; output capped head and tail |

## Skills

A skill is a directory of instructions the agent pulls in when it is relevant:

```
~/.agents/skills/commit-helper/SKILL.md      every repo
./.agents/skills/commit-helper/SKILL.md      this one, checked in
```

```markdown
---
name: commit-helper
description: Use when writing commit messages for this repo
---

Read the last twenty commits for tone before writing anything.
Conventions are in references/conventions.md.
```

The directory name is the skill's name, and a `name:` that disagrees with it is
a load error. Only the description reaches the model until the skill is
invoked, so it is what decides whether the skill fires at all; the body can be
as long as you like.

Two ways in, one way to load. The agent calls the `skill` tool when a
description matches what it is doing; you can force one with `/commit-helper
fix the flaky test`, which the model answers by making the same call. `/skills`
lists what was found, project shadowing personal on a name collision.

Skill directories are readable by `read_file`, `glob` and `grep` even though
they sit outside the working root, so a skill can ship reference files and
point at them. They are not writable by anything: `edit_file`, `write_file` and
`bash` keep the working root as their only root. A skill is input, not a work
product.

Relative paths still anchor to the working root, so a skill pointing at its own
files is read with the directory the `skill` tool prints joined on. If the model
forgets, the "no such file" it gets back names the skill directories, which is
usually enough to recover on the next turn. A shadowed skill's directory is not
readable — only the skills actually in play widen anything.

Discovery happens at startup. The name-and-description listing goes in the
cached prompt prefix, so re-reading the directory each turn would cost a cache
miss on every turn for a file that never changes. Restart to pick up a new
skill. `-no-skills` turns the whole thing off; the eval runner never discovers
skills at all, so METRICS.md is unaffected by what is in your home directory.

## Evals

```bash
./evals/selftest.sh    # no API calls: checks the tasks measure something
go run ./evals -n 3    # the real thing
```

Ten tasks, a fresh temp directory per run, full transcripts under
`evals/runs/<timestamp>/`. See [METRICS.md](METRICS.md).

## Safety

Three layers, in increasing order of how much they actually protect you.

**Path jail.** Every path a tool is given is resolved with
`filepath.EvalSymlinks` and refused if it lands outside the working root,
including through a symlink that points out of the tree. Cheap, and it catches
a lot.

**Approval prompts.** Read-only tools run unattended. `bash`, `edit_file` and
`write_file` ask, with "yes / no / always this exact one / always this tool" and
session memory. `/allowed` lists what you have agreed to.

Shell commands are parsed with `mvdan.cc/sh/syntax` so the prompt can tell you
what a command will actually run — `git status; rm -rf ~` is shown as running
both `git` and `rm`. **This is not a security boundary**, and the harness does
not pretend otherwise: a command containing `$(...)`, backticks, or an
interpreter with `-c` cannot be analysed at all, so it is marked opaque and
"always allow this exact command" is refused for it. String-matching
allowlists for shell commands do not work; this exists to make the prompt
specific, not to make the decision for you.

**A sandbox.** This is the layer that actually contains a mistake, and the
harness cannot provide it for you. The honest recipe on Linux:

```bash
podman run --rm -it \
  -v "$PWD":/work:Z -w /work \
  -e ANTHROPIC_API_KEY \
  docker.io/library/golang:1.26 \
  /work/harness -C /work -yolo
```

**Decide about egress explicitly.** The container above has network access,
because the harness needs to reach the API — which means the agent's `bash`
has it too. That is the line between "broke my repo" and "read my `.env` and
made an HTTP request". If that matters for the tree you are pointing it at:

- Treat every secret reachable from inside the container as exposed.
- Use a token scoped to nothing else, and a clone rather than your working tree.
- For real isolation, run the harness on the host and put only `bash` in a
  network-less namespace. That is not implemented here.

`-yolo` skips every prompt. It is correct inside a container or against a
throwaway clone, and wrong everywhere else. The eval runner sets it, because it
is headless and operates on a temp directory it created.

## Layout

```
cmd/harness      flag parsing, wiring, REPL
internal/llm     Messages API: types, HTTP, SSE, pricing
internal/agent   turn loop, conversation state, compaction, sessions
internal/tools   the tool registry and the tools
internal/skills   discovery and parsing of SKILL.md
internal/approval  allow / deny / ask, with shell analysis
internal/metrics JSONL event log
internal/ui      line-based terminal rendering
internal/tui     full-screen Bubble Tea front end
evals            task suite and runner
```

Two rules hold the shape: `agent` never imports `ui` or `tui`, so the eval
runner can drive the loop headless; and every tool returns `(string, error)`
where the error is text the model reads, not an exception the harness swallows.
A failed edit is a normal event in the transcript.

The TUI obeys the first rule through two one-way hatches. Events reach it via
`agent.Observe`, forwarded into the Bubble Tea loop with `Program.Send`, which
is safe from the agent's goroutine. Approvals go out the same way and then
block the agent on a reply channel until a key arrives. That channel is
buffered and the wait honours the turn's context, so neither side can strand
the other.