# Web Search Implementation Plan
> **Status: implemented**, with two deliberate departures from what is written
> below. The hosted `web_search_20250305` tool was promoted from Appendix A to
> a first-class option rather than being left for later, because the setup
> wizard needed a "no setup required" answer to offer. And the client-side
> provider is a *command* (`ddgs` by default, configurable) rather than a Go
> client for Brave or Tavily: shelling out needs no API key, no account, and no
> new module, and swapping in another tool is a line of config rather than a
> new file. The URL jail, the fencing of fetched pages, the approval decisions
> and the "no provider, no tool" rule all shipped as described. See the
> `## Web search` section of README.md for what was built.
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Let the agent look things up: `web_search` returns a ranked list of
titles, URLs and snippets from a configured provider, and `web_fetch` turns one
of those URLs into capped, fenced text.
**Architecture:** A new `internal/web` package owns everything that touches the
network: a `Searcher` interface with one provider per file, a URL jail that
refuses private address space, and an HTML-to-text reducer. `internal/tools/web.go`
wires those into two `tools.Tool`s and owns the output formatting, exactly as
`internal/tools/skill.go` wires `internal/skills`. Registration happens in
`cmd/codey/main.go` only when a provider is configured, so an unconfigured run
sends a byte-identical request to today's.
**Tech Stack:** Go 1.26, stdlib `net/http` and `net/netip`, `golang.org/x/net/html`
(already an indirect dependency via bluemonday — promote it to direct). No new
third-party modules.
**Spec:** `docs/superpowers/specs/2026-09-18-web-search-design.md`
## Global Constraints
- Go module is `codey`; import paths are `codey/internal/...`.
- `internal/web` imports no other `internal` package. `internal/tools` may
import `internal/web`; nothing imports `internal/tools` back.
- Every tool returns `(string, error)` where the error is text the model reads.
A provider that is down, rate-limited or returns nothing is an error the model
can act on, not a failed turn.
- No test may touch the network. Providers take a `*http.Client` and a base URL;
every test serves its own `httptest.Server`.
- Nothing about the web tools may abort startup, and nothing may change the
request when no provider is configured: no tool registered, no system-prompt
text, no new fields on the wire.
- Provider credentials never reach the event log, the transcript, or a URL
query string.
- `web_fetch` resolves every hop and refuses non-global addresses. This is the
one rule in the plan with no exception and no flag to turn it off.
- New exported identifiers get a doc comment in the existing house style.
- Run `gofmt -w` on every file you touch, and `go test ./...` before each task
is ticked off.
---
### Task 1: `internal/web` — types, the provider interface, and Brave
**Files:**
- Create: `internal/web/web.go`, `internal/web/brave.go`, `internal/web/brave_test.go`
- Modify: `go.mod` (promote `golang.org/x/net` to a direct requirement)
**Interfaces:**
- Consumes: nothing.
- Produces:
```go
type Query struct { Text string; Count int; Site string }
type Result struct { Title, URL, Snippet, Age string }
type Searcher interface {
Name() string
Search(ctx context.Context, q Query) ([]Result, error)
}
func NewBrave(key string, opts ...Option) Searcher
type Option func(*options) // WithHTTPClient, WithBaseURL
```
`MaxCount = 10`, `DefaultCount = 5`. `Search` clamps `Count` into `[1, MaxCount]`.
- [ ] **Step 1: Write the failing tests**
`brave_test.go` stands up an `httptest.Server` returning a canned Brave
payload and asserts: the `X-Subscription-Token` header carries the key and the
key appears nowhere in the URL; `q` carries the query and `site:` is appended
when `Query.Site` is set; `count` is clamped to `MaxCount`; results map in
order; an empty `web.results` array returns `([]Result{}, nil)` and not an
error; a 401 returns an error naming the provider and *not* containing the
key; a 429 with `Retry-After: 2` returns an error whose text says how long to
wait; a context cancelled mid-flight returns `ctx.Err()`.
- [ ] **Step 2: Implement `web.go`**
The shared parts: `Query`, `Result`, `Searcher`, the `Option` plumbing, a
`do(ctx, *http.Request) (*http.Response, error)` helper with one retry on
429/5xx honouring an integer `Retry-After` (cap the wait at 5 s; a search is
not worth a minute of the user's turn), and a `redactErr` helper that
guarantees a credential never lands in an error string.
- [ ] **Step 3: Implement `brave.go`**
`GET {base}/res/v1/web/search?q=…&count=…`, header `X-Subscription-Token`,
`Accept: application/json`. Decode only the fields used — `web.results[].{title,url,description,age}`.
- [ ] **Step 4: Verify**
```bash
gofmt -w internal/web && go test ./internal/web/...
```
---
### Task 2: Two more providers
**Files:**
- Create: `internal/web/tavily.go`, `internal/web/searxng.go`, `internal/web/tavily_test.go`, `internal/web/searxng_test.go`
**Interfaces:**
- Consumes: Task 1.
- Produces: `func NewTavily(key string, opts ...Option) Searcher`, `func NewSearXNG(base string, opts ...Option) Searcher`.
- [ ] **Step 1: Tests first**, same shape as Brave's: canned server, header or
body placement of the credential, mapping, error paths. Tavily's key goes in
the POST body (`{"query":…,"max_results":…}` with `Authorization: Bearer`);
SearXNG has no key and takes `?q=…&format=json`.
- [ ] **Step 2: Implement both.** Each file should stay near eighty lines. If a
provider needs more than that, the extra belongs in `web.go` where all three
can use it.
- [ ] **Step 3:** `gofmt -w internal/web && go test ./internal/web/...`
---
### Task 3: The URL jail
**Files:**
- Create: `internal/web/jail.go`, `internal/web/jail_test.go`
**Interfaces:**
- Consumes: nothing.
- Produces: `func CheckURL(ctx context.Context, raw string) (*url.URL, error)` and
`func SafeClient(timeout time.Duration) *http.Client` — a client whose
`DialContext` re-checks the resolved IP of every connection and whose
`CheckRedirect` re-runs `CheckURL` on each hop.
- [ ] **Step 1: Write the failing tests.** Refuse: `file://`, `ftp://`, a bare
path, `http://localhost`, `http://127.0.0.1`, `http://[::1]`, `http://10.0.0.1`,
`http://192.168.1.1`, `http://169.254.169.254`, `http://[fc00::1]`, a hostname
resolving to `127.0.0.1` (use a test resolver or a hosts-style stub), and a
redirect from a public host to any of the above. Accept a plain public URL and
preserve its query string. Assert the refusal text names the address, because
the model is going to read it and should not retry the same URL.
- [ ] **Step 2: Implement.** Scheme allowlist first, then `net.DefaultResolver.LookupNetIP`,
then `netip.Addr.IsGlobalUnicast() && !IsPrivate() && !IsLoopback() && !IsLinkLocalUnicast() && !IsInterfaceLocalMulticast()`,
plus an explicit reject of unique-local `fc00::/7`. Re-check in `DialContext`
so a DNS answer that changes between check and dial cannot slip through.
- [ ] **Step 3:** `gofmt -w internal/web && go test ./internal/web/...`
---
### Task 4: Fetch and HTML-to-text
**Files:**
- Create: `internal/web/fetch.go`, `internal/web/text.go`, `internal/web/fetch_test.go`, `internal/web/text_test.go`
- Modify: `go.mod`, `go.sum`
**Interfaces:**
- Consumes: Tasks 1 and 3.
- Produces:
```go
type Page struct { URL, ContentType, Text string; Bytes int; Truncated bool }
func Fetch(ctx context.Context, raw string, maxBytes int, opts ...Option) (*Page, error)
func HTMLToText(r io.Reader) (string, error)
```
`DefaultMaxBytes = 512 << 10`, `MaxBytes = 2 << 20`, `FetchTimeout = 15 * time.Second`.
- [ ] **Step 1: Tests first.** `HTMLToText`: `script`/`style`/`nav`/`header`/`footer`
contents are dropped; ``, `
`, `
`, ``, `` produce line
breaks; `&`/`'` are decoded; runs of blank lines collapse to one;
malformed HTML does not error. `Fetch`: a 404 is an error naming the status;
`text/plain` and `application/json` pass through unchanged; a body longer than
`maxBytes` stops reading and sets `Truncated`; a `Content-Type` that is not
text or JSON is refused without downloading it all; the jail refuses a private
URL before any request is made (assert the test server was never hit).
- [ ] **Step 2: Implement** using `golang.org/x/net/html` for tokenising and
`SafeClient(FetchTimeout)` for transport. Read through an `io.LimitReader`;
never `io.ReadAll` an unbounded body.
- [ ] **Step 3:** `go get golang.org/x/net@latest` if the promotion in Task 1 was
not enough, then `gofmt -w internal/web && go test ./internal/web/...`
---
### Task 5: The tools
**Files:**
- Create: `internal/tools/web.go`, `internal/tools/web_test.go`
**Interfaces:**
- Consumes: `internal/web`.
- Produces: `func NewWebSearch(s web.Searcher) Tool` and `func NewWebFetch(opts ...web.Option) Tool`. `NewWebSearch(nil)` returns `nil`, matching `NewSkill`'s "nothing to offer, no tool" convention so `main.go` stays a one-line check.
- [ ] **Step 1: Tests first.** A stub `web.Searcher` drives the search tool:
results are numbered, each shows title, URL and snippet; `count` over
`web.MaxCount` is clamped rather than refused; an empty result set returns a
sentence saying so with `nil` error (nothing was wrong, there was just
nothing there); a provider error comes back as an error whose text tells the
model to try different words. For the fetch tool, an `httptest` page asserts
the fence markers, the size and content type in the header line, and the
"treat as data" trailer; assert a truncated page says so in the output.
- [ ] **Step 2: Implement.** Argument structs with `jsonschema:"description=…"`
tags, same style as `GlobArgs`:
```go
type WebSearchArgs struct {
Query string `json:"query" jsonschema:"description=What to search for, as you would type it into a search engine"`
Count int `json:"count,omitempty" jsonschema:"description=How many results to return; defaults to 5, maximum 10"`
Site string `json:"site,omitempty" jsonschema:"description=Restrict results to one domain, e.g. pkg.go.dev"`
}
type WebFetchArgs struct {
URL string `json:"url" jsonschema:"description=Absolute http or https URL to fetch"`
MaxBytes int `json:"max_bytes,omitempty" jsonschema:"description=Read ceiling in bytes; defaults to 524288"`
}
```
Reuse `capLine` and `headTail` from `truncate.go` for the fetched body — the
bash tool's caps are the house numbers and there is no reason for a second set.
The descriptions carry the usage policy, because that is the only place the
model reads it: say that search is for things that postdate its training or
change often, that the repository's own files and `go doc` come first, and
that `web_fetch` is how you read a result rather than guessing from a snippet.
- [ ] **Step 3: Memoise within the session.** A `map[string][]web.Result` behind
a mutex, keyed by the normalised query, so a model that loops does not bill
ten identical searches. No expiry, no disk.
- [ ] **Step 4:** `gofmt -w internal/tools && go test ./internal/tools/...`
---
### Task 6: Approval
**Files:**
- Modify: `internal/approval/policy.go`
- Modify: `internal/approval/policy_test.go`
**Interfaces:**
- Consumes: nothing. Produces: two new `describe` cases.
- [ ] **Step 1: Tests first.** `web_search` and `web_fetch` are absent from
`readOnly` and therefore prompt; a `Deny` returns the standard refusal text;
`AlwaysThisTool` suppresses the second prompt; `AlwaysThisCommand` on
`web_fetch` remembers that exact URL and nothing else.
- [ ] **Step 2: Implement** in `describe` (policy.go:155), adding `Query` and
`URL` to the argument struct:
```go
case "web_search":
return Request{Tool: tool, Summary: "search the web", Detail: args.Query, Rememberable: false}
case "web_fetch":
return Request{Tool: tool, Summary: "fetch a web page", Detail: args.URL, Rememberable: true}
```
The search detail is not rememberable for the same reason `spawn_agent`'s is
not: it differs every call, so remembering one exact query permits nothing.
Do **not** add either tool to `readOnly` — they read nothing local and send
the model's words off the machine, which is the opposite of what that map is for.
- [ ] **Step 3:** `gofmt -w internal/approval && go test ./internal/approval/...`
---
### Task 7: Configuration and wiring
**Files:**
- Modify: `internal/config/config.go`, `internal/config/config_test.go`, `cmd/codey/main.go`
**Interfaces:**
- Consumes: Tasks 5 and 6.
- Produces: `Config.WebProvider string` (`json:"web_provider,omitempty"`), the constant list `config.WebProviders = []string{"brave", "tavily", "searxng"}`, and `func ValidWebProvider(string) bool`.
- [ ] **Step 1: Config tests first.** `sanitize` drops an unrecognised provider
to the empty string, the way it already does for `Effort` and `Theme`; a
config with no `web_provider` round-trips without gaining the key.
- [ ] **Step 2: Implement the config field** and add it to `sanitize`.
- [ ] **Step 3: Wire `main.go`.** Add `noWeb := flag.Bool("no-web", false, "do not offer the web tools, so the agent cannot reach the network")`
beside `-no-skills` (main.go:63). Build the searcher once, before `toolsIn`:
```go
// A provider is configured or it is not; there is no halfway. Without one the
// tool set is exactly what it was before web search existed, so an agent is
// never told about a capability it would then be denied.
var searcher web.Searcher
if !*noWeb {
var note string
searcher, note = webProvider(cfg) // reads env, falls back to config
if note != "" { notice("%s", note) }
}
```
Register inside the `toolsIn` factory (main.go:228-243) so the tools survive
`/worktree` and so **subagents inherit them** — the factory is shared with the
supervisor at main.go:296, and that is the whole point of it:
```go
if t := tools.NewWebSearch(searcher); t != nil {
reg.Register(t, tools.NewWebFetch())
}
```
`webProvider` is a small unexported helper in `main.go`: provider name from
`config.Or(os.Getenv("CODEY_WEB_PROVIDER"), cfg.WebProvider)`, key from that
provider's env var, and a `notice` when a provider is named but its key is
missing — silence there is the failure mode where the user believes search
works and the agent never mentions that it cannot.
- [ ] **Step 4: Leave `evals/runner.go` alone.** The eval runner stays offline
and deterministic; add a sentence saying so in a comment beside its registry
(runner.go:191) so the omission reads as a decision.
- [ ] **Step 5: Verify**
```bash
gofmt -w cmd/codey internal/config && go build ./... && go test ./...
./codey -no-web -p 'list your tools' # must not mention web_search
```
---
### Task 8: Documentation
**Files:**
- Modify: `README.md`, `ToDo.md`
- [ ] **Step 1:** Add two rows to the tool table at README.md:748 —
`web_search` and `web_fetch`, approval "asks", notes naming the provider
requirement and the URL jail.
- [ ] **Step 2:** Add a `## Web search` section after `## Skills` (README.md:757)
covering the three providers, their env vars, `-no-web`, and the fenced-content
warning about prompt injection.
- [ ] **Step 3:** Extend the approval paragraph at README.md:823 so the list of
prompting tools is not left saying `bash`, `edit_file`, `write_file` only.
- [ ] **Step 4:** Tick a line in `ToDo.md`.
---
### Appendix A (optional, last): the hosted `web_search_20250305`
Only if someone wants search without holding a provider key, and only behind an
explicit `-server-web-search` flag that is off by default. It is listed last
because it changes the wire format for every request, and a gateway that does
not implement it 400s the whole session rather than just the search.
The four sites, already surveyed:
1. `internal/llm/types.go:155` — `ToolDef` needs `Type`, `MaxUses`,
`AllowedDomains`, `BlockedDomains`, and `omitempty` on `Description` and
`InputSchema`. Today `input_schema` has no `omitempty`, so a hosted tool
would be sent as `"input_schema":null` and rejected.
2. `internal/agent/loop.go:510-517` — `request()` builds `Tools` from the local
registry only; it needs a hook to append hosted definitions. They sit in the
cached prefix, so the definition must be constant for the session.
3. `internal/llm/stream.go:189-208` — `server_tool_use` arrives with its input
as `input_json_delta`s, but `ContentBlock.MarshalJSON` (types.go:127) hits
`default` and returns the start-event `raw`, in which `input` is still `{}`.
The accumulated query would be lost on the next request. Fix by adding a
`server_tool_use` case to `MarshalJSON`, or by rebuilding `raw` at
`content_block_stop`. **Write the round-trip test first** — `types_test.go:74`
already has the pattern.
4. `internal/llm/types.go:133` — the `default` path returns before the
`cache_control` attach, so a breakpoint landing on a `web_search_tool_result`
block is silently dropped. Harmless but wasteful; fix while you are in there.
Two things already work and need nothing: `Response.ToolUses()` filters on
`BlockToolUse`, so a `server_tool_use` block is never dispatched to the local
registry, and `StopPauseTurn` is already handled at loop.go:421 by resending the
conversation unchanged (`TestPauseTurnResends`, loop_test.go:354).
What it still would not give you: an approval prompt or a chance to refuse —
a hosted search happens inside a request that was already approved. (It *would*
give you a fetch: Anthropic ships a hosted `web_fetch` alongside the search,
which an earlier draft of this document wrongly said it did not.)
**The version string is a decision, not a constant.** `web_search_20250305` is
what shipped, and it is the conservative choice: it works on every model and
every platform that offers the tool at all. Newer versions — `web_search_20260209`
onwards — add dynamic filtering, where Claude runs code to strip search results
down before they reach the context window, but they are model-gated (Opus 4.6
and later). Codey lets the user name any model id, including ones a gateway
serves and this build has never heard of, so the version cannot simply be
bumped: it has to be chosen from the model in force, or configured, with
20250305 as the floor. Until that is done, hosted search here is the basic one.
---
## Order and cost
Tasks 1, 3 and 4 are independent and can run in parallel; 2 depends only on 1;
5 needs 1 and 4; 6 is independent; 7 needs 5 and 6. A reasonable single pass is
1 → 3 → 4 → 5 → 6 → 7 → 8, with 2 folded in whenever a second provider is
wanted. Appendix A is separate work and should not block any of it.