johannSoCodeyPublic

Phase 0: skeleton, typed tool registry, metrics log

Tool interface with a generic typed[T] wrapper so tools never touch
map[string]any, and input schemas derived from struct tags via
invopop/jsonschema rather than hand-written JSON.

Registry.Execute is the single dispatch point, which makes it the single
place tool calls get logged. List() sorts by name to keep the tool
definitions byte-stable in the cached prompt prefix.

metrics is a *Logger rather than a package-level Log: the eval runner in
Phase 3 needs one file per parallel run. A nil *Logger is a no-op so call
sites need no guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Browse Files

Jjonicommitted last monthcommit0642294

Showing 10 changed files with 1045 additions and 0 deletions.

+3−0.gitignoreaddedView File

++ /.harness/
++ /evals/runs/
++ /harness

+13−0METRICS.mdaddedView File

++ # METRICS
++ The deliverable. One row per change that could plausibly move the numbers.
++ Numbers come from `go run ./evals -n 3` (Phase 3); anything hand-waved gets `—`.
++ Stopping condition: **7/10 eval tasks passing, or 2026-10-28, whichever comes first.**
++ | Change | Pass rate | Edit success | Mean turns | Mean cost |
++ | --- | --- | --- | --- | --- |
++ | Phase 0 skeleton | — | — | — | — |
++ Baseline gets recorded the day the eval runner first prints a table. Until then
++ this file is a promise, not data.

+291−0PLAN.mdaddedView File

++ # Building a Coding Agent Harness in Go — Build Plan
++ Target: a harness that can be pointed at a real Go or TypeScript repo, told
++ "make the failing test pass," and actually do it — with numbers proving how
++ often it works.
++ ## Assumptions
++ - Single provider: Anthropic Messages API. Multi-provider is a later refactor,
++   not a starting constraint.
++ - Go 1.22+. Generics are used for the tool registry.
++ - Terminal interface. No TUI framework at first — plain stdout, add Bubble Tea
++   only if you want it later.
++ - Solo project, roughly 5–6 weeks part-time.
++ **Stopping condition: 7 out of 10 eval tasks passing, or six weeks, whichever
++ comes first.** (See METRICS.md.)
++ ## Repo layout
++ ```
++ harness/
++   cmd/harness/main.go         # flag parsing, wiring, REPL entry
++   internal/
++     llm/
++       client.go               # HTTP + SSE against the Messages API
++       types.go                # Message, ContentBlock, ToolUse, ToolResult
++       stream.go               # partial tool-call JSON accumulation
++     agent/
++       loop.go                 # the core turn loop
++       state.go                # conversation history, file-read tracking
++       compact.go              # summarization when the window fills
++     tools/
++       registry.go             # Tool interface + typed wrapper
++       read.go glob.go grep.go
++       edit.go write.go
++       bash.go
++       truncate.go             # shared output-capping helper
++     approval/
++       policy.go               # allow/deny/ask, with session memory
++     metrics/
++       log.go                  # JSONL event log — every tool call, every token count
++     ui/
++       render.go diff.go
++   evals/
++     tasks/                    # one dir per task: setup.sh, prompt.txt, verify.sh
++     runner.go
++   METRICS.md                  # you fill this in as you go. This is the deliverable.
++ ```
++ Two rules that save pain later:
++ 1. **`agent` never imports `ui`.** The loop must be runnable headless, because
++    the eval runner drives it.
++ 2. **Every tool returns `(string, error)` where the error is for the model to
++    read**, not for you to log and swallow. A failed edit is a normal event in
++    the transcript, not an exception.
++ ## Phase 0 — Skeleton and the tool registry
++ Time: half a day. **Done when:** `go test ./...` passes with a fake tool
++ registered and dispatched.
++ The one piece of Go design worth getting right on day one is typed tool
++ arguments, because otherwise you'll be doing `map[string]any` assertions in
++ every tool for the rest of the project.
++ Derive `schemaFor[T]()` from struct tags with `github.com/invopop/jsonschema`
++ rather than hand-writing JSON Schema. Hand-written schemas drift from the
++ structs and you won't notice until the model sends a field you don't parse.
++ Also in this phase: `metrics.Log(event)` writing one JSON object per line to
++ `./.harness/events.jsonl`. Wire it before you have anything to measure.
++ Retrofitting instrumentation is how projects end up with no numbers.
++ ## Phase 1 — The minimal loop
++ Time: 1–2 days. **Done when:** you ask "what does main.go do?" and the agent
++ reads the file and answers.
++ Non-streaming first. Streaming is a distraction until the loop is correct.
++ ```
++ build messages → POST /v1/messages → response
++   if stop_reason == "tool_use":
++       for each tool_use block: dispatch, collect result
++       append assistant message + user message containing tool_result blocks
++       repeat
++   else: print text, return
++ ```
++ Things that bite here:
++ - Tool results go back as a **user** message containing `tool_result` blocks,
++   one per `tool_use` id, in a **single** message. Multiple parallel tool calls
++   means multiple result blocks in that one message — not one message each.
++ - Every `tool_use` id must get a result, even if execution failed. Send the
++   error text with `is_error: true`. A missing result is an API error.
++ - Set a max-turn cap (start at 30) and return a clear message when hit.
++   Unbounded loops burn money fast.
++ - Thread `context.Context` from main through the loop into every tool from the
++   very start. Retrofitting cancellation later means touching every signature.
++ Only one tool this phase: `read_file(path, offset?, limit?)`.
++ ## Phase 2 — The real tool set
++ Time: 1 week. **Done when:** the agent can find code it hasn't been shown,
++ change it, and run the tests.
++ ### `edit_file(path, old_str, new_str)` — the centerpiece
++ - Reject if the file hasn't been read this session. Track
++   `map[string]time.Time` of reads; if the file's mtime changed since, force a
++   re-read. This single rule kills a large class of hallucinated edits.
++ - Count exact occurrences of `old_str`.
++   - **1 match** → apply. Return a short unified diff of the changed region, not
++     "OK". The model uses the diff to verify it did what it meant.
++   - **0 matches** → do not just say "not found." Retry the match with
++     normalized whitespace; if that hits, tell the model the indentation differs
++     and show the actual line. If it still misses, return the 20 lines around
++     the closest-matching line. A failure that carries context gets fixed in one
++     turn; a bare "not found" costs three.
++   - **>1 match** → return the line numbers of every match and ask for a longer
++     unique span. Never silently pick the first.
++ - Preserve the file's existing line endings. Detect on read, restore on write.
++ - Log every attempt to the metrics file with the outcome. Edit success rate is
++   your headline metric.
++ ### `bash(command, timeout_ms)`
++ - Run via `exec.CommandContext` so cancellation works.
++ - Cap output. Use head+tail (e.g. first 100 and last 100 lines with a
++   `… N lines elided …` marker), never head alone — test failures live at the end.
++ - Default timeout 60s; kill the process **group**, not just the process, or
++   orphaned children hang the loop.
++ - Return exit code, stdout and stderr together, clearly labelled.
++ ### `glob(pattern)` and `grep(pattern, path?, glob?)`
++ Shell out to `rg` if present, fall back to `filepath.WalkDir` + `regexp`. Cap
++ results at ~100 with a count of how many were suppressed. This is your entire
++ code-search story; you do not need embeddings.
++ ### `write_file(path, content)`
++ New files only, or full rewrites the model explicitly chose. Keep it separate
++ from edit so your metrics don't blend them.
++ **Hard rule from this phase onward:** run the harness only against a throwaway
++ git clone, or inside a container. There is no approval layer until Phase 6.
++ Assume it will run `rm -rf` on something eventually.
++ ## Phase 3 — Evals. Do not skip this, do not defer it
++ Time: 3–4 days. **Done when:** `go run ./evals -n 3` prints a pass rate and a
++ metrics table.
++ This is the pivot point of the whole project. Everything after this gets
++ measured; everything before it was guesswork.
++ Task format — one directory each:
++ ```
++ evals/tasks/fix-nil-deref/
++   setup.sh      # clone/copy a repo into a temp dir, apply a breaking patch
++   prompt.txt    # what the user would type
++   verify.sh     # exit 0 == task solved
++ ```
++ Ten tasks, drawn from real repos you know. A good spread:
++ 1. Fix a failing unit test (obvious cause)
++ 2. Fix a failing unit test (cause is two files away from the failure)
++ 3. Add a small feature with a test
++ 4. Rename a symbol across 5+ files
++ 5. Add error handling to an existing function
++ 6. Fix a compile error from a partial refactor
++ 7. Write a test for existing untested code
++ 8. Find and fix a bug from a description only, no stack trace
++ 9. Update a dependency call site after an API change
++ 10. A task that *should* fail — verify it says so instead of claiming success
++ Runner requirements:
++ - Fresh temp dir per run, parallel across tasks, `-n 3` to repeat each (agents
++   are nondeterministic; a single run tells you nothing).
++ - Persist the full transcript per run to `evals/runs/<ts>/<task>/<n>.json`. You
++   will read these constantly. Most regressions are obvious within ten seconds
++   of reading a trajectory.
++ - Print: pass rate, mean turns, mean input/output tokens, mean wall time, edit
++   success rate, tool call counts by name.
++ Record the baseline in METRICS.md that day. Every later phase adds a row.
++ ## Phase 4 — Context management, measured
++ Time: 4–5 days. **Done when:** a task requiring 40+ turns completes, and your
++ eval numbers didn't drop.
++ - Per-tool output caps (from Phase 2) — tune the numbers against evals rather
++   than intuition.
++ - **Compaction:** at ~70% of the window, summarize the older half of the
++   conversation into a single message. Preserve: the original task, files
++   touched and why, decisions made, things discovered the hard way ("the config
++   is loaded from X, not Y"), and the last few turns verbatim. Do it at 70%, not
++   95% — compaction itself needs headroom.
++ - **Stale reads:** if a file was read 20 turns ago and edited since, its old
++   contents in history are actively harmful. Simplest fix that works: replace
++   superseded `read_file` results with a stub pointing at the current version.
++ - **Prompt caching:** keep the prefix byte-stable — system prompt, then tool
++   definitions, then project context, then the growing conversation. No
++   timestamps, no session ids, no "files changed" block near the front. Add
++   cache breakpoints and check `cache_read_input_tokens` in responses to confirm
++   hits. This is your single biggest cost and latency lever, and compaction
++   breaks it, which is another argument for compacting rarely.
++ Run evals after each change. Expect at least one of these to make things worse.
++ ## Phase 5 — Streaming, interruption, and the interface
++ Time: 3–4 days. **Done when:** you can hit Ctrl-C mid-`go test` and give a
++ correction without losing the session.
++ - Switch `llm.Client` to SSE. The fiddly part is accumulating
++   `input_json_delta` fragments per tool-call index into complete JSON before
++   dispatch — buffer per block index, parse on `content_block_stop`.
++ - Ctrl-C cancels the current turn's context. The partially-run tool returns an
++   error result, and control comes back to you for a follow-up message. Second
++   Ctrl-C within a second or two exits.
++ - Show diffs before applying, with color.
++ - Show running token and cost totals per session. Cheap to add, and it changes
++   how you design.
++ - Persist sessions to `.harness/sessions/<id>.json` and add `--resume`.
++ Keep `agent` free of terminal dependencies. The eval runner depends on that.
++ ## Phase 6 — Safety
++ Time: 2–3 days. **Done when:** you'd run it on your actual working repo without
++ flinching.
++ - **Approval policy:** read-only tools auto-approve. `bash`, `write_file`,
++   `edit_file` prompt, with "yes / no / always for this exact command / always
++   for this tool" and session memory.
++ - **Do not allowlist shell commands with string matching.** `git status; rm -rf ~`
++   defeats it, and so do backticks, `$()`, `&&`, `xargs`, and any interpreter
++   with `-c`. Either parse with `mvdan.cc/sh/syntax` and check each command
++   node, or don't pretend and rely on the sandbox plus prompts.
++ - **Path jail:** resolve every file path with `filepath.EvalSymlinks` and
++   verify it's inside the working root. Cheap and catches a lot.
++ - **Sandbox:** the honest answer for a Go project on Linux is a container with
++   the repo bind-mounted and network off by default. Decide explicitly about
++   egress — that's the line between "broke my repo" and "read my .env and made
++   an HTTP request."
++ ## Phase 7 — Optional, pick at most one
++ Only after you've hit your eval target.
++ - Sub-agents: spawn a fresh context for a search task, return only the summary.
++   Big context win, real complexity cost.
++ - Multi-provider (OpenAI, local models). Note that this means maintaining
++   per-model prompt variants, not one abstraction.
++ - A CLAUDE.md-style project context file loaded into the cached prefix.
++ - Bubble Tea TUI.
++ ## Handing phases to Claude Code
++ Give it one phase at a time, with the interfaces fixed. A prompt shape that
++ works:
++ > Implement Phase 2 of the plan in PLAN.md: the `edit_file` tool in
++ > `internal/tools/edit.go`. It must satisfy the `tools.Tool` interface already
++ > defined in `registry.go` — don't change that interface. Requirements: reject
++ > edits to files not read this session (use the tracker in `agent/state.go`);
++ > exact-match `old_str`; on zero matches retry with normalized whitespace and
++ > return the 20 surrounding lines of the closest candidate; on multiple matches
++ > return every match's line number and refuse; on success return a unified diff
++ > of the changed region; preserve the file's original line endings. Write
++ > table-driven tests covering all four match outcomes plus CRLF preservation.
++ > Run `go test ./internal/tools/...` and iterate until green.
++ What makes this work: one file, an interface it may not change, explicit
++ failure-mode requirements, and a command it can run to check itself. Vague
++ phase-level prompts ("build the tool system") produce plausible code that fails
++ in ways you won't find for a week.
++ ## What you're actually producing
++ The code is the byproduct. The deliverable is METRICS.md.

+53−0cmd/harness/main.goaddedView File

++ // Command harness is the CLI entry point: flag parsing, wiring, and (from
++ // Phase 1) the REPL.
++ package main
++ import (
++ 	"encoding/json"
++ 	"flag"
++ 	"fmt"
++ 	"os"
++ 	"path/filepath"
++ 	"time"
++ 	"harness/internal/metrics"
++ 	"harness/internal/tools"
++ )
++ func main() {
++ 	if err := run(); err != nil {
++ 		fmt.Fprintln(os.Stderr, "harness:", err)
++ 		os.Exit(1)
++ 	}
++ }
++ func run() error {
++ 	workdir := flag.String("C", ".", "working root the agent operates in")
++ 	logPath := flag.String("metrics", metrics.DefaultPath, "JSONL event log path")
++ 	flag.Parse()
++ 	root, err := filepath.Abs(*workdir)
++ 	if err != nil {
++ 		return err
++ 	}
++ 	session := time.Now().UTC().Format("20060102T150405Z")
++ 	log, err := metrics.Open(filepath.Join(root, *logPath), session)
++ 	if err != nil {
++ 		return err
++ 	}
++ 	defer log.Close()
++ 	log.Log(metrics.Event{Type: "session_start", Extra: map[string]any{"root": root}})
++ 	reg := tools.NewRegistry(log)
++ 	// Tools land here in Phase 1 (read_file) and Phase 2 (the rest).
++ 	fmt.Printf("harness session %s\nroot:    %s\nmetrics: %s\ntools:   %d registered\n",
++ 		session, root, filepath.Join(root, *logPath), len(reg.List()))
++ 	for _, t := range reg.List() {
++ 		schema, _ := json.Marshal(json.RawMessage(t.InputSchema()))
++ 		fmt.Printf("  %-12s %s\n               %s\n", t.Name(), t.Description(), schema)
++ 	}
++ 	fmt.Println("\nno agent loop yet — Phase 1.")
++ 	return nil
++ }

+12−0go.modaddedView File

++ module harness
++ go 1.26.8
++ require github.com/invopop/jsonschema v0.14.0
++ require (
++ 	github.com/bahlo/generic-list-go v0.2.0 // indirect
++ 	github.com/buger/jsonparser v1.1.2 // indirect
++ 	github.com/pb33f/ordered-map/v2 v2.3.1 // indirect
++ 	go.yaml.in/yaml/v4 v4.0.0-rc.2 // indirect
++ )

+18−0go.sumaddedView File

++ github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
++ github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
++ github.com/buger/jsonparser v1.1.2 h1:frqHqw7otoVbk5M8LlE/L7HTnIq2v9RX6EJ48i9AxJk=
++ github.com/buger/jsonparser v1.1.2/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
++ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
++ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
++ github.com/invopop/jsonschema v0.14.0 h1:MHQqLhvpNUZfw+hM3AZDYK7jxO8FZoQeQM77g8iyZjg=
++ github.com/invopop/jsonschema v0.14.0/go.mod h1:ygm6C2EaVNMBDPpaPlnOA2pFAxBnxGjFlMZABxm9n2I=
++ github.com/pb33f/ordered-map/v2 v2.3.1 h1:5319HDO0aw4DA4gzi+zv4FXU9UlSs3xGZ40wcP1nBjY=
++ github.com/pb33f/ordered-map/v2 v2.3.1/go.mod h1:qxFQgd0PkVUtOMCkTapqotNgzRhMPL7VvaHKbd1HnmQ=
++ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
++ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
++ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
++ github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
++ go.yaml.in/yaml/v4 v4.0.0-rc.2 h1:/FrI8D64VSr4HtGIlUtlFMGsm7H7pWTbj6vOLVZcA6s=
++ go.yaml.in/yaml/v4 v4.0.0-rc.2/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
++ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
++ gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

+154−0internal/metrics/log.goaddedView File

++ // Package metrics writes a JSONL event log: one JSON object per line, one line
++ // per thing worth counting later. It is wired in before there is anything to
++ // measure on purpose — retrofitting instrumentation is how projects end up with
++ // no numbers.
++ package metrics
++ import (
++ 	"encoding/json"
++ 	"fmt"
++ 	"io"
++ 	"os"
++ 	"path/filepath"
++ 	"sync"
++ 	"time"
++ )
++ // DefaultPath is where the harness logs unless told otherwise.
++ const DefaultPath = ".harness/events.jsonl"
++ // Event is one line of the log. Extra is flattened into the same JSON object so
++ // the file stays greppable with jq without nested lookups.
++ type Event struct {
++ 	Time    time.Time `json:"ts"`
++ 	Session string    `json:"session,omitempty"`
++ 	Type    string    `json:"type"`
++ 	// Tool dispatch.
++ 	Tool     string `json:"tool,omitempty"`
++ 	Duration int64  `json:"dur_ms,omitempty"`
++ 	OK       *bool  `json:"ok,omitempty"`
++ 	Err      string `json:"err,omitempty"`
++ 	// Turn accounting.
++ 	Turn       int    `json:"turn,omitempty"`
++ 	StopReason string `json:"stop_reason,omitempty"`
++ 	// Token accounting, mirroring the Messages API usage block.
++ 	InputTokens      int `json:"input_tokens,omitempty"`
++ 	OutputTokens     int `json:"output_tokens,omitempty"`
++ 	CacheReadTokens  int `json:"cache_read_input_tokens,omitempty"`
++ 	CacheWriteTokens int `json:"cache_creation_input_tokens,omitempty"`
++ 	Extra map[string]any `json:"-"`
++ }
++ // MarshalJSON flattens Extra alongside the declared fields. Keys in Extra never
++ // clobber a declared field.
++ func (e Event) MarshalJSON() ([]byte, error) {
++ 	type alias Event
++ 	b, err := json.Marshal(alias(e))
++ 	if err != nil {
++ 		return nil, err
++ 	}
++ 	if len(e.Extra) == 0 {
++ 		return b, nil
++ 	}
++ 	var m map[string]json.RawMessage
++ 	if err := json.Unmarshal(b, &m); err != nil {
++ 		return nil, err
++ 	}
++ 	for k, v := range e.Extra {
++ 		if _, taken := m[k]; taken {
++ 			continue
++ 		}
++ 		ev, err := json.Marshal(v)
++ 		if err != nil {
++ 			return nil, fmt.Errorf("metrics: extra field %q: %w", k, err)
++ 		}
++ 		m[k] = ev
++ 	}
++ 	return json.Marshal(m)
++ }
++ // Logger appends events to a writer. The zero value and a nil *Logger are both
++ // usable and simply drop events, so callers never need a nil check.
++ type Logger struct {
++ 	mu      sync.Mutex
++ 	w       io.Writer
++ 	closer  io.Closer
++ 	session string
++ }
++ // Open appends to path, creating parent directories as needed.
++ func Open(path, session string) (*Logger, error) {
++ 	if dir := filepath.Dir(path); dir != "." {
++ 		if err := os.MkdirAll(dir, 0o755); err != nil {
++ 			return nil, fmt.Errorf("metrics: %w", err)
++ 		}
++ 	}
++ 	f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
++ 	if err != nil {
++ 		return nil, fmt.Errorf("metrics: %w", err)
++ 	}
++ 	return &Logger{w: f, closer: f, session: session}, nil
++ }
++ // NewWriter logs to an arbitrary writer. Useful in tests and in the eval runner,
++ // which keeps a separate log per run.
++ func NewWriter(w io.Writer, session string) *Logger {
++ 	return &Logger{w: w, session: session}
++ }
++ // Session returns the session id stamped onto every event.
++ func (l *Logger) Session() string {
++ 	if l == nil {
++ 		return ""
++ 	}
++ 	return l.session
++ }
++ // Log writes one event. Errors are dropped: a broken metrics file must never
++ // take down a run.
++ func (l *Logger) Log(ev Event) {
++ 	if l == nil || l.w == nil {
++ 		return
++ 	}
++ 	if ev.Time.IsZero() {
++ 		ev.Time = time.Now().UTC()
++ 	}
++ 	if ev.Session == "" {
++ 		ev.Session = l.session
++ 	}
++ 	b, err := json.Marshal(ev)
++ 	if err != nil {
++ 		return
++ 	}
++ 	l.mu.Lock()
++ 	defer l.mu.Unlock()
++ 	l.w.Write(append(b, '\n'))
++ }
++ // ToolCall records one dispatch. err nil means success.
++ func (l *Logger) ToolCall(name string, d time.Duration, err error, extra map[string]any) {
++ 	ok := err == nil
++ 	ev := Event{
++ 		Type:     "tool_call",
++ 		Tool:     name,
++ 		Duration: d.Milliseconds(),
++ 		OK:       &ok,
++ 		Extra:    extra,
++ 	}
++ 	if err != nil {
++ 		ev.Err = err.Error()
++ 	}
++ 	l.Log(ev)
++ }
++ // Close closes the underlying file, if there is one.
++ func (l *Logger) Close() error {
++ 	if l == nil || l.closer == nil {
++ 		return nil
++ 	}
++ 	return l.closer.Close()
++ }

+148−0internal/metrics/log_test.goaddedView File

++ package metrics
++ import (
++ 	"bytes"
++ 	"encoding/json"
++ 	"errors"
++ 	"os"
++ 	"path/filepath"
++ 	"strings"
++ 	"sync"
++ 	"testing"
++ 	"time"
++ )
++ func decode(t *testing.T, line string) map[string]any {
++ 	t.Helper()
++ 	var m map[string]any
++ 	if err := json.Unmarshal([]byte(line), &m); err != nil {
++ 		t.Fatalf("line is not valid JSON (%v): %s", err, line)
++ 	}
++ 	return m
++ }
++ func TestLogWritesOneObjectPerLine(t *testing.T) {
++ 	var buf bytes.Buffer
++ 	l := NewWriter(&buf, "sess-1")
++ 	l.Log(Event{Type: "turn", Turn: 1, InputTokens: 900})
++ 	l.Log(Event{Type: "turn", Turn: 2, InputTokens: 1200})
++ 	lines := strings.Split(strings.TrimRight(buf.String(), "\n"), "\n")
++ 	if len(lines) != 2 {
++ 		t.Fatalf("got %d lines, want 2:\n%s", len(lines), buf.String())
++ 	}
++ 	first := decode(t, lines[0])
++ 	if first["session"] != "sess-1" {
++ 		t.Errorf("session = %v, want sess-1 stamped automatically", first["session"])
++ 	}
++ 	if first["ts"] == nil {
++ 		t.Error("ts must be filled in when the caller leaves it zero")
++ 	}
++ 	if first["input_tokens"].(float64) != 900 {
++ 		t.Errorf("input_tokens = %v, want 900", first["input_tokens"])
++ 	}
++ }
++ func TestExtraIsFlattened(t *testing.T) {
++ 	var buf bytes.Buffer
++ 	NewWriter(&buf, "s").Log(Event{
++ 		Type:  "edit",
++ 		Tool:  "edit_file",
++ 		Extra: map[string]any{"matches": 3, "path": "main.go"},
++ 	})
++ 	m := decode(t, buf.String())
++ 	if m["matches"].(float64) != 3 || m["path"] != "main.go" {
++ 		t.Errorf("Extra should sit alongside declared fields, got %v", m)
++ 	}
++ }
++ func TestExtraCannotClobberDeclaredFields(t *testing.T) {
++ 	var buf bytes.Buffer
++ 	NewWriter(&buf, "s").Log(Event{Type: "edit", Extra: map[string]any{"type": "spoofed"}})
++ 	if m := decode(t, buf.String()); m["type"] != "edit" {
++ 		t.Errorf("type = %v, want the declared value to win", m["type"])
++ 	}
++ }
++ func TestToolCallRecordsOutcome(t *testing.T) {
++ 	tests := []struct {
++ 		name   string
++ 		err    error
++ 		wantOK bool
++ 		wantEr string
++ 	}{
++ 		{name: "success", err: nil, wantOK: true},
++ 		{name: "failure", err: errors.New("no such file"), wantOK: false, wantEr: "no such file"},
++ 	}
++ 	for _, tc := range tests {
++ 		t.Run(tc.name, func(t *testing.T) {
++ 			var buf bytes.Buffer
++ 			NewWriter(&buf, "s").ToolCall("read_file", 12*time.Millisecond, tc.err, nil)
++ 			m := decode(t, buf.String())
++ 			if m["type"] != "tool_call" || m["tool"] != "read_file" {
++ 				t.Fatalf("unexpected event %v", m)
++ 			}
++ 			if m["ok"] != tc.wantOK {
++ 				t.Errorf("ok = %v, want %v", m["ok"], tc.wantOK)
++ 			}
++ 			if m["dur_ms"].(float64) != 12 {
++ 				t.Errorf("dur_ms = %v, want 12", m["dur_ms"])
++ 			}
++ 			if tc.wantEr != "" && m["err"] != tc.wantEr {
++ 				t.Errorf("err = %v, want %q", m["err"], tc.wantEr)
++ 			}
++ 		})
++ 	}
++ }
++ func TestNilLoggerIsANoOp(t *testing.T) {
++ 	var l *Logger
++ 	l.Log(Event{Type: "turn"})
++ 	l.ToolCall("x", 0, nil, nil)
++ 	if l.Session() != "" {
++ 		t.Error("nil logger should report an empty session")
++ 	}
++ 	if err := l.Close(); err != nil {
++ 		t.Errorf("Close on nil logger = %v, want nil", err)
++ 	}
++ }
++ func TestOpenCreatesParentDirsAndAppends(t *testing.T) {
++ 	path := filepath.Join(t.TempDir(), ".harness", "events.jsonl")
++ 	for i := 0; i < 2; i++ {
++ 		l, err := Open(path, "s")
++ 		if err != nil {
++ 			t.Fatalf("Open: %v", err)
++ 		}
++ 		l.Log(Event{Type: "run_start"})
++ 		l.Close()
++ 	}
++ 	b, err := os.ReadFile(path)
++ 	if err != nil {
++ 		t.Fatalf("ReadFile: %v", err)
++ 	}
++ 	if n := strings.Count(string(b), "\n"); n != 2 {
++ 		t.Errorf("got %d lines, want 2 (second Open must append, not truncate)", n)
++ 	}
++ }
++ func TestConcurrentLogLinesStayIntact(t *testing.T) {
++ 	var buf bytes.Buffer
++ 	l := NewWriter(&buf, "s")
++ 	var wg sync.WaitGroup
++ 	for i := 0; i < 50; i++ {
++ 		wg.Add(1)
++ 		go func(i int) {
++ 			defer wg.Done()
++ 			l.Log(Event{Type: "tool_call", Turn: i})
++ 		}(i)
++ 	}
++ 	wg.Wait()
++ 	lines := strings.Split(strings.TrimRight(buf.String(), "\n"), "\n")
++ 	if len(lines) != 50 {
++ 		t.Fatalf("got %d lines, want 50", len(lines))
++ 	}
++ 	for _, ln := range lines {
++ 		decode(t, ln)
++ 	}
++ }

+158−0internal/tools/registry.goaddedView File

++ // Package tools defines the tool interface the agent loop dispatches against.
++ //
++ // Tools return (string, error). The error is text for the model to read, not an
++ // exception for the harness to swallow: a failed edit is a normal event in the
++ // transcript.
++ package tools
++ import (
++ 	"context"
++ 	"encoding/json"
++ 	"fmt"
++ 	"sort"
++ 	"sync"
++ 	"time"
++ 	"harness/internal/metrics"
++ 	"github.com/invopop/jsonschema"
++ )
++ // Tool is everything the loop needs to advertise and run one tool.
++ type Tool interface {
++ 	Name() string
++ 	Description() string
++ 	InputSchema() json.RawMessage
++ 	Execute(ctx context.Context, raw json.RawMessage) (string, error)
++ }
++ // typed adapts a func taking a concrete argument struct to the Tool interface,
++ // so individual tools never touch map[string]any.
++ type typed[T any] struct {
++ 	name, desc string
++ 	schema     json.RawMessage
++ 	fn         func(context.Context, T) (string, error)
++ }
++ func (t typed[T]) Name() string                 { return t.name }
++ func (t typed[T]) Description() string          { return t.desc }
++ func (t typed[T]) InputSchema() json.RawMessage { return t.schema }
++ func (t typed[T]) Execute(ctx context.Context, raw json.RawMessage) (string, error) {
++ 	if len(raw) == 0 {
++ 		raw = json.RawMessage("{}")
++ 	}
++ 	var in T
++ 	if err := json.Unmarshal(raw, &in); err != nil {
++ 		return "", fmt.Errorf("invalid arguments: %w", err)
++ 	}
++ 	return t.fn(ctx, in)
++ }
++ // New builds a Tool whose input schema is derived from T's struct tags. It
++ // panics if T cannot be reflected, which is a programming error at wiring time.
++ func New[T any](name, desc string, fn func(context.Context, T) (string, error)) Tool {
++ 	return typed[T]{name: name, desc: desc, schema: SchemaFor[T](), fn: fn}
++ }
++ // reflector inlines everything: the Messages API wants a self-contained schema
++ // object, not one stitched together from $defs.
++ var reflector = &jsonschema.Reflector{
++ 	ExpandedStruct: true,
++ 	DoNotReference: true,
++ }
++ // SchemaFor derives a JSON Schema from T. Fields tagged omitempty are optional;
++ // everything else is required. Descriptions come from `jsonschema:"description=..."`.
++ func SchemaFor[T any]() json.RawMessage {
++ 	var zero T
++ 	b, err := json.Marshal(reflector.Reflect(zero))
++ 	if err != nil {
++ 		panic(fmt.Sprintf("tools: cannot reflect schema for %T: %v", zero, err))
++ 	}
++ 	var m map[string]any
++ 	if err := json.Unmarshal(b, &m); err != nil {
++ 		panic(fmt.Sprintf("tools: cannot reflect schema for %T: %v", zero, err))
++ 	}
++ 	// Metadata the API has no use for, and which would only add bytes to the
++ 	// cached prompt prefix.
++ 	delete(m, "$schema")
++ 	delete(m, "$id")
++ 	delete(m, "version")
++ 	if _, ok := m["type"]; !ok {
++ 		m["type"] = "object"
++ 	}
++ 	if _, ok := m["properties"]; !ok {
++ 		m["properties"] = map[string]any{}
++ 	}
++ 	out, err := json.Marshal(m)
++ 	if err != nil {
++ 		panic(fmt.Sprintf("tools: cannot reflect schema for %T: %v", zero, err))
++ 	}
++ 	return out
++ }
++ // Registry holds the tools available to one session and is the single place
++ // tool dispatch gets logged, so no individual tool has to remember to do it.
++ type Registry struct {
++ 	mu     sync.RWMutex
++ 	byName map[string]Tool
++ 	order  []string
++ 	log    *metrics.Logger
++ }
++ // NewRegistry returns an empty registry. log may be nil.
++ func NewRegistry(log *metrics.Logger) *Registry {
++ 	return &Registry{byName: map[string]Tool{}, log: log}
++ }
++ // Register adds a tool, panicking on a duplicate name.
++ func (r *Registry) Register(ts ...Tool) *Registry {
++ 	r.mu.Lock()
++ 	defer r.mu.Unlock()
++ 	for _, t := range ts {
++ 		if _, dup := r.byName[t.Name()]; dup {
++ 			panic(fmt.Sprintf("tools: duplicate registration of %q", t.Name()))
++ 		}
++ 		r.byName[t.Name()] = t
++ 		r.order = append(r.order, t.Name())
++ 	}
++ 	return r
++ }
++ // Get returns the named tool.
++ func (r *Registry) Get(name string) (Tool, bool) {
++ 	r.mu.RLock()
++ 	defer r.mu.RUnlock()
++ 	t, ok := r.byName[name]
++ 	return t, ok
++ }
++ // List returns the tools in a stable order, so the tool-definition block of the
++ // prompt stays byte-identical between turns and keeps the cache prefix warm.
++ func (r *Registry) List() []Tool {
++ 	r.mu.RLock()
++ 	defer r.mu.RUnlock()
++ 	names := append([]string(nil), r.order...)
++ 	sort.Strings(names)
++ 	out := make([]Tool, 0, len(names))
++ 	for _, n := range names {
++ 		out = append(out, r.byName[n])
++ 	}
++ 	return out
++ }
++ // Execute dispatches by name and logs the outcome. An unknown tool is reported
++ // back to the model as an error rather than failing the turn.
++ func (r *Registry) Execute(ctx context.Context, name string, raw json.RawMessage) (string, error) {
++ 	t, ok := r.Get(name)
++ 	if !ok {
++ 		err := fmt.Errorf("unknown tool %q", name)
++ 		r.log.ToolCall(name, 0, err, nil)
++ 		return "", err
++ 	}
++ 	start := time.Now()
++ 	out, err := t.Execute(ctx, raw)
++ 	r.log.ToolCall(name, time.Since(start), err, map[string]any{"output_bytes": len(out)})
++ 	return out, err
++ }

+195−0internal/tools/registry_test.goaddedView File

++ package tools
++ import (
++ 	"bytes"
++ 	"context"
++ 	"encoding/json"
++ 	"errors"
++ 	"strings"
++ 	"testing"
++ 	"harness/internal/metrics"
++ )
++ type echoArgs struct {
++ 	Text  string `json:"text" jsonschema:"description=Text to echo back"`
++ 	Times int    `json:"times,omitempty" jsonschema:"description=How many times to repeat it"`
++ }
++ func echoTool(fail bool) Tool {
++ 	return New("echo", "Echo text back", func(_ context.Context, in echoArgs) (string, error) {
++ 		if fail {
++ 			return "", errors.New("echo refused")
++ 		}
++ 		n := in.Times
++ 		if n == 0 {
++ 			n = 1
++ 		}
++ 		return strings.Repeat(in.Text, n), nil
++ 	})
++ }
++ func TestSchemaForReflectsStructTags(t *testing.T) {
++ 	var got struct {
++ 		Type       string   `json:"type"`
++ 		Required   []string `json:"required"`
++ 		Properties map[string]struct {
++ 			Type        string `json:"type"`
++ 			Description string `json:"description"`
++ 		} `json:"properties"`
++ 		AdditionalProperties *bool           `json:"additionalProperties"`
++ 		Schema               json.RawMessage `json:"$schema"`
++ 	}
++ 	if err := json.Unmarshal(SchemaFor[echoArgs](), &got); err != nil {
++ 		t.Fatalf("schema is not valid JSON: %v", err)
++ 	}
++ 	if got.Type != "object" {
++ 		t.Errorf("type = %q, want object", got.Type)
++ 	}
++ 	if len(got.Required) != 1 || got.Required[0] != "text" {
++ 		t.Errorf("required = %v, want [text] (omitempty fields must be optional)", got.Required)
++ 	}
++ 	if got.Properties["times"].Type != "integer" {
++ 		t.Errorf("times type = %q, want integer", got.Properties["times"].Type)
++ 	}
++ 	if got.Properties["text"].Description != "Text to echo back" {
++ 		t.Errorf("description not carried over: %q", got.Properties["text"].Description)
++ 	}
++ 	if got.Schema != nil {
++ 		t.Errorf("$schema should be stripped, got %s", got.Schema)
++ 	}
++ }
++ func TestSchemaForNoFields(t *testing.T) {
++ 	type empty struct{}
++ 	var got map[string]any
++ 	if err := json.Unmarshal(SchemaFor[empty](), &got); err != nil {
++ 		t.Fatalf("invalid JSON: %v", err)
++ 	}
++ 	if got["type"] != "object" {
++ 		t.Errorf("type = %v, want object", got["type"])
++ 	}
++ 	if _, ok := got["properties"]; !ok {
++ 		t.Error("properties must be present even when empty")
++ 	}
++ }
++ func TestTypedExecute(t *testing.T) {
++ 	tests := []struct {
++ 		name    string
++ 		raw     string
++ 		want    string
++ 		wantErr string
++ 	}{
++ 		{name: "decodes typed args", raw: `{"text":"ab","times":3}`, want: "ababab"},
++ 		{name: "omitted optional field", raw: `{"text":"x"}`, want: "x"},
++ 		{name: "empty input treated as {}", raw: ``, want: ""},
++ 		{name: "malformed JSON", raw: `{"text":`, wantErr: "invalid arguments"},
++ 		{name: "wrong field type", raw: `{"text":"x","times":"three"}`, wantErr: "invalid arguments"},
++ 	}
++ 	for _, tc := range tests {
++ 		t.Run(tc.name, func(t *testing.T) {
++ 			got, err := echoTool(false).Execute(context.Background(), json.RawMessage(tc.raw))
++ 			switch {
++ 			case tc.wantErr != "" && err == nil:
++ 				t.Fatalf("want error containing %q, got none (result %q)", tc.wantErr, got)
++ 			case tc.wantErr != "":
++ 				if !strings.Contains(err.Error(), tc.wantErr) {
++ 					t.Fatalf("error = %v, want it to contain %q", err, tc.wantErr)
++ 				}
++ 			case err != nil:
++ 				t.Fatalf("unexpected error: %v", err)
++ 			case got != tc.want:
++ 				t.Fatalf("result = %q, want %q", got, tc.want)
++ 			}
++ 		})
++ 	}
++ }
++ func TestRegistryDispatch(t *testing.T) {
++ 	r := NewRegistry(nil).Register(echoTool(false))
++ 	got, err := r.Execute(context.Background(), "echo", json.RawMessage(`{"text":"hi"}`))
++ 	if err != nil || got != "hi" {
++ 		t.Fatalf("Execute = (%q, %v), want (\"hi\", nil)", got, err)
++ 	}
++ 	if _, err := r.Execute(context.Background(), "nope", nil); err == nil {
++ 		t.Error("unknown tool must return an error the model can read")
++ 	} else if !strings.Contains(err.Error(), "nope") {
++ 		t.Errorf("error should name the tool, got %v", err)
++ 	}
++ }
++ func TestRegistryToolErrorIsReturnedNotSwallowed(t *testing.T) {
++ 	r := NewRegistry(nil).Register(echoTool(true))
++ 	out, err := r.Execute(context.Background(), "echo", json.RawMessage(`{"text":"hi"}`))
++ 	if err == nil {
++ 		t.Fatal("want the tool's error to surface")
++ 	}
++ 	if out != "" {
++ 		t.Errorf("output = %q, want empty on failure", out)
++ 	}
++ 	if err.Error() != "echo refused" {
++ 		t.Errorf("error = %q, want it passed through verbatim", err)
++ 	}
++ }
++ func TestRegistryListIsStable(t *testing.T) {
++ 	r := NewRegistry(nil).Register(
++ 		New("zulu", "", func(context.Context, echoArgs) (string, error) { return "", nil }),
++ 		New("alpha", "", func(context.Context, echoArgs) (string, error) { return "", nil }),
++ 	)
++ 	var names []string
++ 	for _, tl := range r.List() {
++ 		names = append(names, tl.Name())
++ 	}
++ 	if len(names) != 2 || names[0] != "alpha" || names[1] != "zulu" {
++ 		t.Errorf("List = %v, want sorted [alpha zulu] for a byte-stable prompt prefix", names)
++ 	}
++ }
++ func TestRegistryDuplicatePanics(t *testing.T) {
++ 	defer func() {
++ 		if recover() == nil {
++ 			t.Error("registering the same name twice must panic at wiring time")
++ 		}
++ 	}()
++ 	NewRegistry(nil).Register(echoTool(false), echoTool(false))
++ }
++ func TestExecuteRespectsContext(t *testing.T) {
++ 	blocked := New("block", "", func(ctx context.Context, _ echoArgs) (string, error) {
++ 		<-ctx.Done()
++ 		return "", ctx.Err()
++ 	})
++ 	ctx, cancel := context.WithCancel(context.Background())
++ 	cancel()
++ 	if _, err := NewRegistry(nil).Register(blocked).Execute(ctx, "block", nil); !errors.Is(err, context.Canceled) {
++ 		t.Errorf("error = %v, want context.Canceled", err)
++ 	}
++ }
++ func TestRegistryLogsEveryDispatch(t *testing.T) {
++ 	var buf bytes.Buffer
++ 	r := NewRegistry(metrics.NewWriter(&buf, "sess")).Register(echoTool(false))
++ 	r.Execute(context.Background(), "echo", json.RawMessage(`{"text":"hi"}`))
++ 	r.Execute(context.Background(), "missing", nil)
++ 	lines := strings.Split(strings.TrimRight(buf.String(), "\n"), "\n")
++ 	if len(lines) != 2 {
++ 		t.Fatalf("got %d log lines, want one per dispatch:\n%s", len(lines), buf.String())
++ 	}
++ 	var ok struct {
++ 		Tool string `json:"tool"`
++ 		OK   bool   `json:"ok"`
++ 		Out  int    `json:"output_bytes"`
++ 	}
++ 	json.Unmarshal([]byte(lines[0]), &ok)
++ 	if ok.Tool != "echo" || !ok.OK || ok.Out != 2 {
++ 		t.Errorf("success event = %+v, want echo/ok/2 bytes", ok)
++ 	}
++ 	if !strings.Contains(lines[1], `"ok":false`) {
++ 		t.Errorf("unknown-tool dispatch should log a failure, got %s", lines[1])
++ 	}
++ }