# harness

A coding agent harness in Go. Point it at a repository, tell it to make the
failing test pass, and measure how often it actually does.

The code is the byproduct. The deliverable is [METRICS.md](METRICS.md).
[PLAN.md](PLAN.md) is the build plan this follows.

## Quick start

```bash
export ANTHROPIC_API_KEY=sk-ant-...
go build -o harness ./cmd/harness

./harness -C /path/to/a/throwaway/clone          # REPL
./harness -C /path/to/repo -p "why does TestFoo fail?"   # one prompt
```

A different endpoint — a gateway, a proxy, a local server speaking the Messages
API:

```bash
./harness -base-url http://127.0.0.1:4000 -api-key ... -C .
# or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN
```

Flags override the environment. `-auth-token` sends `Authorization: Bearer`,
which is what OAuth tokens and most gateways want; `-api-key` sends
`x-api-key`. A non-default endpoint announces itself on stderr, because your
prompt and every file the agent reads go wherever that URL points.

Useful flags: `-model`, `-effort low|medium|high|xhigh|max`, `-max-turns`,
`-resume latest`, `-no-stream`, `-no-cache`, `-v`, `-yolo`.

## Tools

| Tool | Approval | Notes |
| --- | --- | --- |
| `read_file` | automatic | Numbered lines, `offset`/`limit` for paging |
| `glob` | automatic | Newest first; `*.go` matches at any depth |
| `grep` | automatic | ripgrep when present, pure Go otherwise, identical output either way |
| `edit_file` | prompts | Requires a prior read; refuses ambiguous matches; returns a diff |
| `write_file` | prompts | Whole files only; separate from `edit_file` so the metrics stay honest |
| `bash` | prompts | Own process group, killed whole on timeout; output capped head and tail |

## Evals

```bash
./evals/selftest.sh    # no API calls: checks the tasks measure something
go run ./evals -n 3    # the real thing
```

Ten tasks, a fresh temp directory per run, full transcripts under
`evals/runs/<timestamp>/`. See [METRICS.md](METRICS.md).

## Safety

Three layers, in increasing order of how much they actually protect you.

**Path jail.** Every path a tool is given is resolved with
`filepath.EvalSymlinks` and refused if it lands outside the working root,
including through a symlink that points out of the tree. Cheap, and it catches
a lot.

**Approval prompts.** Read-only tools run unattended. `bash`, `edit_file` and
`write_file` ask, with "yes / no / always this exact one / always this tool" and
session memory. `/allowed` lists what you have agreed to.

Shell commands are parsed with `mvdan.cc/sh/syntax` so the prompt can tell you
what a command will actually run — `git status; rm -rf ~` is shown as running
both `git` and `rm`. **This is not a security boundary**, and the harness does
not pretend otherwise: a command containing `$(...)`, backticks, or an
interpreter with `-c` cannot be analysed at all, so it is marked opaque and
"always allow this exact command" is refused for it. String-matching
allowlists for shell commands do not work; this exists to make the prompt
specific, not to make the decision for you.

**A sandbox.** This is the layer that actually contains a mistake, and the
harness cannot provide it for you. The honest recipe on Linux:

```bash
podman run --rm -it \
  -v "$PWD":/work:Z -w /work \
  -e ANTHROPIC_API_KEY \
  docker.io/library/golang:1.26 \
  /work/harness -C /work -yolo
```

**Decide about egress explicitly.** The container above has network access,
because the harness needs to reach the API — which means the agent's `bash`
has it too. That is the line between "broke my repo" and "read my `.env` and
made an HTTP request". If that matters for the tree you are pointing it at:

- Treat every secret reachable from inside the container as exposed.
- Use a token scoped to nothing else, and a clone rather than your working tree.
- For real isolation, run the harness on the host and put only `bash` in a
  network-less namespace. That is not implemented here.

`-yolo` skips every prompt. It is correct inside a container or against a
throwaway clone, and wrong everywhere else. The eval runner sets it, because it
is headless and operates on a temp directory it created.

## Layout

```
cmd/harness      flag parsing, wiring, REPL
internal/llm     Messages API: types, HTTP, SSE, pricing
internal/agent   turn loop, conversation state, compaction, sessions
internal/tools   the tool registry and the tools
internal/approval  allow / deny / ask, with shell analysis
internal/metrics JSONL event log
internal/ui      terminal rendering
evals            task suite and runner
```

Two rules hold the shape: `agent` never imports `ui`, so the eval runner can
drive the loop headless; and every tool returns `(string, error)` where the
error is text the model reads, not an exception the harness swallows. A failed
edit is a normal event in the transcript.