harness

A coding agent harness in Go. Point it at a repository, tell it to make the failing test pass, and measure how often it actually does.

The code is the byproduct. The deliverable is METRICS.md. PLAN.md is the build plan this follows.

Quick start

export ANTHROPIC_API_KEY=sk-ant-...
go build -o harness ./cmd/harness

./harness -C /path/to/a/throwaway/clone          # REPL
./harness -C /path/to/repo -p "why does TestFoo fail?"   # one prompt

A different endpoint — a gateway, a proxy, a local server speaking the Messages API:

./harness -base-url http://127.0.0.1:4000 -api-key ... -C .
# or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN

Flags override the environment. -auth-token sends Authorization: Bearer, which is what OAuth tokens and most gateways want; -api-key sends x-api-key. A non-default endpoint announces itself on stderr, because your prompt and every file the agent reads go wherever that URL points.

Useful flags: -model, -effort low|medium|high|xhigh|max, -max-turns, -resume latest, -no-stream, -no-cache, -v, -yolo.

Tools

Tool	Approval	Notes
read_file	automatic	Numbered lines, offset/limit for paging
glob	automatic	Newest first; *.go matches at any depth
grep	automatic	ripgrep when present, pure Go otherwise, identical output either way
edit_file	prompts	Requires a prior read; refuses ambiguous matches; returns a diff
write_file	prompts	Whole files only; separate from edit_file so the metrics stay honest
bash	prompts	Own process group, killed whole on timeout; output capped head and tail

Evals

./evals/selftest.sh    # no API calls: checks the tasks measure something
go run ./evals -n 3    # the real thing

Ten tasks, a fresh temp directory per run, full transcripts under evals/runs/<timestamp>/. See METRICS.md.

Safety

Three layers, in increasing order of how much they actually protect you.

Path jail. Every path a tool is given is resolved with filepath.EvalSymlinks and refused if it lands outside the working root, including through a symlink that points out of the tree. Cheap, and it catches a lot.

Approval prompts. Read-only tools run unattended. bash, edit_file and write_file ask, with "yes / no / always this exact one / always this tool" and session memory. /allowed lists what you have agreed to.

Shell commands are parsed with mvdan.cc/sh/syntax so the prompt can tell you what a command will actually run — git status; rm -rf ~ is shown as running both git and rm. This is not a security boundary, and the harness does not pretend otherwise: a command containing $(...), backticks, or an interpreter with -c cannot be analysed at all, so it is marked opaque and "always allow this exact command" is refused for it. String-matching allowlists for shell commands do not work; this exists to make the prompt specific, not to make the decision for you.

A sandbox. This is the layer that actually contains a mistake, and the harness cannot provide it for you. The honest recipe on Linux:

podman run --rm -it \
  -v "$PWD":/work:Z -w /work \
  -e ANTHROPIC_API_KEY \
  docker.io/library/golang:1.26 \
  /work/harness -C /work -yolo

Decide about egress explicitly. The container above has network access, because the harness needs to reach the API — which means the agent's bash has it too. That is the line between "broke my repo" and "read my .env and made an HTTP request". If that matters for the tree you are pointing it at:

• Treat every secret reachable from inside the container as exposed.
• Use a token scoped to nothing else, and a clone rather than your working tree.
• For real isolation, run the harness on the host and put only bash in a network-less namespace. That is not implemented here.

-yolo skips every prompt. It is correct inside a container or against a throwaway clone, and wrong everywhere else. The eval runner sets it, because it is headless and operates on a temp directory it created.

Layout

cmd/harness      flag parsing, wiring, REPL
internal/llm     Messages API: types, HTTP, SSE, pricing
internal/agent   turn loop, conversation state, compaction, sessions
internal/tools   the tool registry and the tools
internal/approval  allow / deny / ask, with shell analysis
internal/metrics JSONL event log
internal/ui      terminal rendering
evals            task suite and runner

Two rules hold the shape: agent never imports ui, so the eval runner can drive the loop headless; and every tool returns (string, error) where the error is text the model reads, not an exception the harness swallows. A failed edit is a normal event in the transcript.