# harness

A coding agent harness in Go. Point it at a repository, tell it to make the
failing test pass, and measure how often it actually does.

The code is the byproduct. The deliverable is [METRICS.md](METRICS.md).
[PLAN.md](PLAN.md) is the build plan this follows.

## Quick start

```bash
export ANTHROPIC_API_KEY=sk-ant-...
go build -o harness ./cmd/harness

./harness -C /path/to/a/throwaway/clone          # full-screen TUI
./harness -C /path/to/repo -p "why does TestFoo fail?"   # one prompt
```

A different endpoint — a gateway, a proxy, a local server speaking the Messages
API:

```bash
./harness -base-url http://127.0.0.1:4000 -api-key ... -C .
# or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN
```

Flags override the environment. `-auth-token` sends `Authorization: Bearer`,
which is what OAuth tokens and most gateways want; `-api-key` sends
`x-api-key`. A non-default endpoint announces itself on stderr, because your
prompt and every file the agent reads go wherever that URL points.

Useful flags: `-model`, `-effort low|medium|high|xhigh|max`, `-max-turns`,
`-resume latest`, `-no-stream`, `-no-cache`, `-v`, `-yolo`, `-no-tui`, `-no-skills`.

## The TUI

On a terminal, the interactive mode is a full-screen [Bubble Tea](https://github.com/charmbracelet/bubbletea)
app. It opens on a welcome block naming the model, the endpoint and the root,
and settles into a transcript above a prompt:

```
⏺ Let me see what the test expects before changing anything.

⏺ ReadFile(path=internal/agent/loop_test.go)
  ⎿ 1: package agent
    …

⏺ EditFile(path=internal/agent/compact.go)
  4 additions, 1 removal
  @@ -50,7 +50,9 @@
  +    if window == 0 {

 ✢ Noodling… (14s · turn 3 · ↓ 1.3k tokens · esc to interrupt)   ● high · /effort
╭──────────────────────────────────────────────────────────────────────────────╮
│ › _                                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯
 joni   ✳ Opus 5   ◐ 62.2k (6%)   ↑ 4.2k ⟳ 58.0k ↓ 1.3k  $0.0835
 ⏵ approvals on (shift+tab to cycle) · /help for commands
```

Every event hangs off a dot in the left gutter, and everything belonging to it
is indented under that dot, so a screen of tool output still reads as a
sequence of things that happened. Diffs are shown in full and in colour with a
count of what changed; other output is elided after six lines.

The status bar reports the context gauge — the last turn's input size against
the window, which drops when the conversation is compacted — and separates
fresh input tokens from cache reads, because the gap between the two is what
[METRICS.md](METRICS.md) is about. The prompt box grows with what you type, up
to six lines.

| Key | Does |
| --- | --- |
| `enter` | send |
| `ctrl+j` | newline without sending |
| `shift+tab` | cycle the approval mode |
| `ctrl+r` | expand elided tool output |
| `pgup`/`pgdn` | scroll the transcript |
| `esc` | interrupt the turn |
| `ctrl+c` | interrupt the turn; again when idle to exit |

Commands: `/allowed`, `/clear`, `/help`, `/exit`, `/skills`.

`shift+tab` cycles between asking before every mutating call and accepting them
all, which is `-yolo` reached from the keyboard. The mode line says which is in
force at all times, and the switch is recorded in the transcript, because a
mode that silently lets the agent write to disk is not one to keep in a corner.

Approvals appear inline as a bordered prompt, answered by number or by the key
in brackets — `y`, `n`, `a` for this exact command, `t` for the whole tool. The
agent goroutine blocks until you answer, which is the point: the tool has not
run yet. Interrupting while that prompt is open denies the call and cancels the
turn rather than leaving the agent waiting on a channel nobody will read.

The TUI is skipped automatically when stdin or stdout is not a terminal, and by
`-p` and `-no-tui`, which use the line-based renderer instead. Both front ends
consume the same `agent.Event` stream, so neither can show something the other
cannot.

### Driving it without an API key

`internal/tui/testdata/mockapi` is a fake Messages API that streams a canned
reply and asks to run a shell command, which is enough to exercise streaming,
tool rendering, approve, deny and interrupt for nothing:

```bash
go run ./internal/tui/testdata/mockapi &
mkdir -p /tmp/demo && git -C /tmp/demo init -q
go build -o harness ./cmd/harness
./harness -C /tmp/demo -base-url http://127.0.0.1:8899 -api-key mock
```

Type anything, and the approval prompt appears. `-slow 150ms` stretches the
stream out if you want to watch the spinner or test `ctrl+c` mid-turn. It sits
under `testdata` so `./...` ignores it.

## Tools

| Tool | Approval | Notes |
| --- | --- | --- |
| `read_file` | automatic | Numbered lines, `offset`/`limit` for paging |
| `glob` | automatic | Newest first; `*.go` matches at any depth |
| `grep` | automatic | ripgrep when present, pure Go otherwise, identical output either way |
| `edit_file` | prompts | Requires a prior read; refuses ambiguous matches; returns a diff |
| `write_file` | prompts | Whole files only; separate from `edit_file` so the metrics stay honest |
| `bash` | prompts | Own process group, killed whole on timeout; output capped head and tail |

## Skills

A skill is a directory of instructions the agent pulls in when it is relevant:

```
~/.agents/skills/commit-helper/SKILL.md      every repo
./.agents/skills/commit-helper/SKILL.md      this one, checked in
```

```markdown
---
name: commit-helper
description: Use when writing commit messages for this repo
---

Read the last twenty commits for tone before writing anything.
Conventions are in references/conventions.md.
```

The directory name is the skill's name, and a `name:` that disagrees with it is
a load error. Only the description reaches the model until the skill is
invoked, so it is what decides whether the skill fires at all; the body can be
as long as you like.

Two ways in, one way to load. The agent calls the `skill` tool when a
description matches what it is doing; you can force one with `/commit-helper
fix the flaky test`, which the model answers by making the same call. `/skills`
lists what was found, project shadowing personal on a name collision.

Skill directories are readable by `read_file`, `glob` and `grep` even though
they sit outside the working root, so a skill can ship reference files and
point at them. They are not writable by anything: `edit_file`, `write_file` and
`bash` keep the working root as their only root. A skill is input, not a work
product.

Relative paths still anchor to the working root, so a skill pointing at its own
files is read with the directory the `skill` tool prints joined on. If the model
forgets, the "no such file" it gets back names the skill directories, which is
usually enough to recover on the next turn. A shadowed skill's directory is not
readable — only the skills actually in play widen anything.

Discovery happens at startup. The name-and-description listing goes in the
cached prompt prefix, so re-reading the directory each turn would cost a cache
miss on every turn for a file that never changes. Restart to pick up a new
skill. `-no-skills` turns the whole thing off; the eval runner never discovers
skills at all, so METRICS.md is unaffected by what is in your home directory.

## Evals

```bash
./evals/selftest.sh    # no API calls: checks the tasks measure something
go run ./evals -n 3    # the real thing
```

Ten tasks, a fresh temp directory per run, full transcripts under
`evals/runs/<timestamp>/`. See [METRICS.md](METRICS.md).

## Safety

Three layers, in increasing order of how much they actually protect you.

**Path jail.** Every path a tool is given is resolved with
`filepath.EvalSymlinks` and refused if it lands outside the working root,
including through a symlink that points out of the tree. Cheap, and it catches
a lot.

**Approval prompts.** Read-only tools run unattended. `bash`, `edit_file` and
`write_file` ask, with "yes / no / always this exact one / always this tool" and
session memory. `/allowed` lists what you have agreed to.

Shell commands are parsed with `mvdan.cc/sh/syntax` so the prompt can tell you
what a command will actually run — `git status; rm -rf ~` is shown as running
both `git` and `rm`. **This is not a security boundary**, and the harness does
not pretend otherwise: a command containing `$(...)`, backticks, or an
interpreter with `-c` cannot be analysed at all, so it is marked opaque and
"always allow this exact command" is refused for it. String-matching
allowlists for shell commands do not work; this exists to make the prompt
specific, not to make the decision for you.

**A sandbox.** This is the layer that actually contains a mistake, and the
harness cannot provide it for you. The honest recipe on Linux:

```bash
podman run --rm -it \
  -v "$PWD":/work:Z -w /work \
  -e ANTHROPIC_API_KEY \
  docker.io/library/golang:1.26 \
  /work/harness -C /work -yolo
```

**Decide about egress explicitly.** The container above has network access,
because the harness needs to reach the API — which means the agent's `bash`
has it too. That is the line between "broke my repo" and "read my `.env` and
made an HTTP request". If that matters for the tree you are pointing it at:

- Treat every secret reachable from inside the container as exposed.
- Use a token scoped to nothing else, and a clone rather than your working tree.
- For real isolation, run the harness on the host and put only `bash` in a
  network-less namespace. That is not implemented here.

`-yolo` skips every prompt. It is correct inside a container or against a
throwaway clone, and wrong everywhere else. The eval runner sets it, because it
is headless and operates on a temp directory it created.

## Layout

```
cmd/harness      flag parsing, wiring, REPL
internal/llm     Messages API: types, HTTP, SSE, pricing
internal/agent   turn loop, conversation state, compaction, sessions
internal/tools   the tool registry and the tools
internal/skills   discovery and parsing of SKILL.md
internal/approval  allow / deny / ask, with shell analysis
internal/metrics JSONL event log
internal/ui      line-based terminal rendering
internal/tui     full-screen Bubble Tea front end
evals            task suite and runner
```

Two rules hold the shape: `agent` never imports `ui` or `tui`, so the eval
runner can drive the loop headless; and every tool returns `(string, error)`
where the error is text the model reads, not an exception the harness swallows.
A failed edit is a normal event in the transcript.

The TUI obeys the first rule through two one-way hatches. Events reach it via
`agent.Observe`, forwarded into the Bubble Tea loop with `Program.Send`, which
is safe from the agent's goroutine. Approvals go out the same way and then
block the agent on a reply channel until a key arrives. That channel is
buffered and the wait honours the turn's context, so neither side can strand
the other.