Status: implemented, with two deliberate departures from what is written below. The hosted
web_search_20250305tool was promoted from Appendix A to a first-class option rather than being left for later, because the setup wizard needed a "no setup required" answer to offer. And the client-side provider is a command (ddgsby default, configurable) rather than a Go client for Brave or Tavily: shelling out needs no API key, no account, and no new module, and swapping in another tool is a line of config rather than a new file. The URL jail, the fencing of fetched pages, the approval decisions and the "no provider, no tool" rule all shipped as described. See the## Web searchsection of README.md for what was built.
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Let the agent look things up: web_search returns a ranked list of
titles, URLs and snippets from a configured provider, and web_fetch turns one
of those URLs into capped, fenced text.
Architecture: A new internal/web package owns everything that touches the
network: a Searcher interface with one provider per file, a URL jail that
refuses private address space, and an HTML-to-text reducer. internal/tools/web.go
wires those into two tools.Tools and owns the output formatting, exactly as
internal/tools/skill.go wires internal/skills. Registration happens in
cmd/codey/main.go only when a provider is configured, so an unconfigured run
sends a byte-identical request to today's.
Tech Stack: Go 1.26, stdlib net/http and net/netip, golang.org/x/net/html
(already an indirect dependency via bluemonday — promote it to direct). No new
third-party modules.
Spec: docs/superpowers/specs/2026-09-18-web-search-design.md
codey; import paths are codey/internal/....internal/web imports no other internal package. internal/tools may
import internal/web; nothing imports internal/tools back.(string, error) where the error is text the model reads.
A provider that is down, rate-limited or returns nothing is an error the model
can act on, not a failed turn.*http.Client and a base URL;
every test serves its own httptest.Server.web_fetch resolves every hop and refuses non-global addresses. This is the
one rule in the plan with no exception and no flag to turn it off.gofmt -w on every file you touch, and go test ./... before each task
is ticked off.internal/web — types, the provider interface, and BraveFiles:
internal/web/web.go, internal/web/brave.go, internal/web/brave_test.gogo.mod (promote golang.org/x/net to a direct requirement)Interfaces:
Consumes: nothing.
Produces:
type Query struct { Text string; Count int; Site string }
type Result struct { Title, URL, Snippet, Age string }
type Searcher interface {
Name() string
Search(ctx context.Context, q Query) ([]Result, error)
}
func NewBrave(key string, opts ...Option) Searcher
type Option func(*options) // WithHTTPClient, WithBaseURL
MaxCount = 10, DefaultCount = 5. Search clamps Count into [1, MaxCount].
Step 1: Write the failing tests
brave_test.go stands up an httptest.Server returning a canned Brave
payload and asserts: the X-Subscription-Token header carries the key and the
key appears nowhere in the URL; q carries the query and site: is appended
when Query.Site is set; count is clamped to MaxCount; results map in
order; an empty web.results array returns ([]Result{}, nil) and not an
error; a 401 returns an error naming the provider and not containing the
key; a 429 with Retry-After: 2 returns an error whose text says how long to
wait; a context cancelled mid-flight returns ctx.Err().
Step 2: Implement web.go
The shared parts: Query, Result, Searcher, the Option plumbing, a
do(ctx, *http.Request) (*http.Response, error) helper with one retry on
429/5xx honouring an integer Retry-After (cap the wait at 5 s; a search is
not worth a minute of the user's turn), and a redactErr helper that
guarantees a credential never lands in an error string.
Step 3: Implement brave.go
GET {base}/res/v1/web/search?q=…&count=…, header X-Subscription-Token,
Accept: application/json. Decode only the fields used — web.results[].{title,url,description,age}.
Step 4: Verify
gofmt -w internal/web && go test ./internal/web/...
Files:
internal/web/tavily.go, internal/web/searxng.go, internal/web/tavily_test.go, internal/web/searxng_test.goInterfaces:
Consumes: Task 1.
Produces: func NewTavily(key string, opts ...Option) Searcher, func NewSearXNG(base string, opts ...Option) Searcher.
Step 1: Tests first, same shape as Brave's: canned server, header or
body placement of the credential, mapping, error paths. Tavily's key goes in
the POST body ({"query":…,"max_results":…} with Authorization: Bearer);
SearXNG has no key and takes ?q=…&format=json.
Step 2: Implement both. Each file should stay near eighty lines. If a
provider needs more than that, the extra belongs in web.go where all three
can use it.
Step 3: gofmt -w internal/web && go test ./internal/web/...
Files:
internal/web/jail.go, internal/web/jail_test.goInterfaces:
Consumes: nothing.
Produces: func CheckURL(ctx context.Context, raw string) (*url.URL, error) and
func SafeClient(timeout time.Duration) *http.Client — a client whose
DialContext re-checks the resolved IP of every connection and whose
CheckRedirect re-runs CheckURL on each hop.
Step 1: Write the failing tests. Refuse: file://, ftp://, a bare
path, http://localhost, http://127.0.0.1, http://[::1], http://10.0.0.1,
http://192.168.1.1, http://169.254.169.254, http://[fc00::1], a hostname
resolving to 127.0.0.1 (use a test resolver or a hosts-style stub), and a
redirect from a public host to any of the above. Accept a plain public URL and
preserve its query string. Assert the refusal text names the address, because
the model is going to read it and should not retry the same URL.
Step 2: Implement. Scheme allowlist first, then net.DefaultResolver.LookupNetIP,
then netip.Addr.IsGlobalUnicast() && !IsPrivate() && !IsLoopback() && !IsLinkLocalUnicast() && !IsInterfaceLocalMulticast(),
plus an explicit reject of unique-local fc00::/7. Re-check in DialContext
so a DNS answer that changes between check and dial cannot slip through.
Step 3: gofmt -w internal/web && go test ./internal/web/...
Files:
internal/web/fetch.go, internal/web/text.go, internal/web/fetch_test.go, internal/web/text_test.gogo.mod, go.sumInterfaces:
Consumes: Tasks 1 and 3.
Produces:
type Page struct { URL, ContentType, Text string; Bytes int; Truncated bool }
func Fetch(ctx context.Context, raw string, maxBytes int, opts ...Option) (*Page, error)
func HTMLToText(r io.Reader) (string, error)
DefaultMaxBytes = 512 << 10, MaxBytes = 2 << 20, FetchTimeout = 15 * time.Second.
Step 1: Tests first. HTMLToText: script/style/nav/header/footer
contents are dropped; <p>, <br>, <li>, <h1..h6>, <tr> produce line
breaks; &/' are decoded; runs of blank lines collapse to one;
malformed HTML does not error. Fetch: a 404 is an error naming the status;
text/plain and application/json pass through unchanged; a body longer than
maxBytes stops reading and sets Truncated; a Content-Type that is not
text or JSON is refused without downloading it all; the jail refuses a private
URL before any request is made (assert the test server was never hit).
Step 2: Implement using golang.org/x/net/html for tokenising and
SafeClient(FetchTimeout) for transport. Read through an io.LimitReader;
never io.ReadAll an unbounded body.
Step 3: go get golang.org/x/net@latest if the promotion in Task 1 was
not enough, then gofmt -w internal/web && go test ./internal/web/...
Files:
internal/tools/web.go, internal/tools/web_test.goInterfaces:
Consumes: internal/web.
Produces: func NewWebSearch(s web.Searcher) Tool and func NewWebFetch(opts ...web.Option) Tool. NewWebSearch(nil) returns nil, matching NewSkill's "nothing to offer, no tool" convention so main.go stays a one-line check.
Step 1: Tests first. A stub web.Searcher drives the search tool:
results are numbered, each shows title, URL and snippet; count over
web.MaxCount is clamped rather than refused; an empty result set returns a
sentence saying so with nil error (nothing was wrong, there was just
nothing there); a provider error comes back as an error whose text tells the
model to try different words. For the fetch tool, an httptest page asserts
the fence markers, the size and content type in the header line, and the
"treat as data" trailer; assert a truncated page says so in the output.
Step 2: Implement. Argument structs with jsonschema:"description=…"
tags, same style as GlobArgs:
type WebSearchArgs struct {
Query string `json:"query" jsonschema:"description=What to search for, as you would type it into a search engine"`
Count int `json:"count,omitempty" jsonschema:"description=How many results to return; defaults to 5, maximum 10"`
Site string `json:"site,omitempty" jsonschema:"description=Restrict results to one domain, e.g. pkg.go.dev"`
}
type WebFetchArgs struct {
URL string `json:"url" jsonschema:"description=Absolute http or https URL to fetch"`
MaxBytes int `json:"max_bytes,omitempty" jsonschema:"description=Read ceiling in bytes; defaults to 524288"`
}
Reuse capLine and headTail from truncate.go for the fetched body — the
bash tool's caps are the house numbers and there is no reason for a second set.
The descriptions carry the usage policy, because that is the only place the
model reads it: say that search is for things that postdate its training or
change often, that the repository's own files and go doc come first, and
that web_fetch is how you read a result rather than guessing from a snippet.
Step 3: Memoise within the session. A map[string][]web.Result behind
a mutex, keyed by the normalised query, so a model that loops does not bill
ten identical searches. No expiry, no disk.
Step 4: gofmt -w internal/tools && go test ./internal/tools/...
Files:
internal/approval/policy.gointernal/approval/policy_test.goInterfaces:
Consumes: nothing. Produces: two new describe cases.
Step 1: Tests first. web_search and web_fetch are absent from
readOnly and therefore prompt; a Deny returns the standard refusal text;
AlwaysThisTool suppresses the second prompt; AlwaysThisCommand on
web_fetch remembers that exact URL and nothing else.
Step 2: Implement in describe (policy.go:155), adding Query and
URL to the argument struct:
case "web_search":
return Request{Tool: tool, Summary: "search the web", Detail: args.Query, Rememberable: false}
case "web_fetch":
return Request{Tool: tool, Summary: "fetch a web page", Detail: args.URL, Rememberable: true}
The search detail is not rememberable for the same reason spawn_agent's is
not: it differs every call, so remembering one exact query permits nothing.
Do not add either tool to readOnly — they read nothing local and send
the model's words off the machine, which is the opposite of what that map is for.
Step 3: gofmt -w internal/approval && go test ./internal/approval/...
Files:
internal/config/config.go, internal/config/config_test.go, cmd/codey/main.goInterfaces:
Consumes: Tasks 5 and 6.
Produces: Config.WebProvider string (json:"web_provider,omitempty"), the constant list config.WebProviders = []string{"brave", "tavily", "searxng"}, and func ValidWebProvider(string) bool.
Step 1: Config tests first. sanitize drops an unrecognised provider
to the empty string, the way it already does for Effort and Theme; a
config with no web_provider round-trips without gaining the key.
Step 2: Implement the config field and add it to sanitize.
Step 3: Wire main.go. Add noWeb := flag.Bool("no-web", false, "do not offer the web tools, so the agent cannot reach the network")
beside -no-skills (main.go:63). Build the searcher once, before toolsIn:
// A provider is configured or it is not; there is no halfway. Without one the
// tool set is exactly what it was before web search existed, so an agent is
// never told about a capability it would then be denied.
var searcher web.Searcher
if !*noWeb {
var note string
searcher, note = webProvider(cfg) // reads env, falls back to config
if note != "" { notice("%s", note) }
}
Register inside the toolsIn factory (main.go:228-243) so the tools survive
/worktree and so subagents inherit them — the factory is shared with the
supervisor at main.go:296, and that is the whole point of it:
if t := tools.NewWebSearch(searcher); t != nil {
reg.Register(t, tools.NewWebFetch())
}
webProvider is a small unexported helper in main.go: provider name from
config.Or(os.Getenv("CODEY_WEB_PROVIDER"), cfg.WebProvider), key from that
provider's env var, and a notice when a provider is named but its key is
missing — silence there is the failure mode where the user believes search
works and the agent never mentions that it cannot.
Step 4: Leave evals/runner.go alone. The eval runner stays offline
and deterministic; add a sentence saying so in a comment beside its registry
(runner.go:191) so the omission reads as a decision.
Step 5: Verify
gofmt -w cmd/codey internal/config && go build ./... && go test ./...
./codey -no-web -p 'list your tools' # must not mention web_search
Files:
Modify: README.md, ToDo.md
Step 1: Add two rows to the tool table at README.md:748 —
web_search and web_fetch, approval "asks", notes naming the provider
requirement and the URL jail.
Step 2: Add a ## Web search section after ## Skills (README.md:757)
covering the three providers, their env vars, -no-web, and the fenced-content
warning about prompt injection.
Step 3: Extend the approval paragraph at README.md:823 so the list of
prompting tools is not left saying bash, edit_file, write_file only.
Step 4: Tick a line in ToDo.md.
web_search_20250305Only if someone wants search without holding a provider key, and only behind an
explicit -server-web-search flag that is off by default. It is listed last
because it changes the wire format for every request, and a gateway that does
not implement it 400s the whole session rather than just the search.
The four sites, already surveyed:
internal/llm/types.go:155 — ToolDef needs Type, MaxUses,
AllowedDomains, BlockedDomains, and omitempty on Description and
InputSchema. Today input_schema has no omitempty, so a hosted tool
would be sent as "input_schema":null and rejected.internal/agent/loop.go:510-517 — request() builds Tools from the local
registry only; it needs a hook to append hosted definitions. They sit in the
cached prefix, so the definition must be constant for the session.internal/llm/stream.go:189-208 — server_tool_use arrives with its input
as input_json_deltas, but ContentBlock.MarshalJSON (types.go:127) hits
default and returns the start-event raw, in which input is still {}.
The accumulated query would be lost on the next request. Fix by adding a
server_tool_use case to MarshalJSON, or by rebuilding raw at
content_block_stop. Write the round-trip test first — types_test.go:74
already has the pattern.internal/llm/types.go:133 — the default path returns before the
cache_control attach, so a breakpoint landing on a web_search_tool_result
block is silently dropped. Harmless but wasteful; fix while you are in there.Two things already work and need nothing: Response.ToolUses() filters on
BlockToolUse, so a server_tool_use block is never dispatched to the local
registry, and StopPauseTurn is already handled at loop.go:421 by resending the
conversation unchanged (TestPauseTurnResends, loop_test.go:354).
What it still would not give you: an approval prompt or a chance to refuse —
a hosted search happens inside a request that was already approved. (It would
give you a fetch: Anthropic ships a hosted web_fetch alongside the search,
which an earlier draft of this document wrongly said it did not.)
The version string is a decision, not a constant. web_search_20250305 is
what shipped, and it is the conservative choice: it works on every model and
every platform that offers the tool at all. Newer versions — web_search_20260209
onwards — add dynamic filtering, where Claude runs code to strip search results
down before they reach the context window, but they are model-gated (Opus 4.6
and later). Codey lets the user name any model id, including ones a gateway
serves and this build has never heard of, so the version cannot simply be
bumped: it has to be chosen from the model in force, or configured, with
20250305 as the floor. Until that is done, hosted search here is the basic one.
Tasks 1, 3 and 4 are independent and can run in parallel; 2 depends only on 1; 5 needs 1 and 4; 6 is independent; 7 needs 5 and 6. A reasonable single pass is 1 → 3 → 4 → 5 → 6 → 7 → 8, with 2 folded in whenever a second provider is wanted. Appendix A is separate work and should not block any of it.