# Spec: inline UI, diagrams and mini apps in the transcript

Status: ready to build. Written for an agent that has not seen the discussion behind it.
Repo: Vectrum Code (`/home/joni/Documents/DATA/Work/Coding/rust/code`), a GPUI desktop client
for Claude Code, Pi and jcode. Read `README.md` first for the app's shape.

## 1. Goal

Agents can put rich, live output into their answers, like Claude's artifacts or
[OpenUI](https://www.openui.com/): diagrams, charts, cards, tables, small forms and test UIs,
and whole HTML mini apps. The agent writes them as fenced code blocks in its normal Markdown
answer. The app renders them natively in the transcript (MetalUI look, no webview), and
interactive ones can talk back to the agent.

Four block kinds, all plain fenced code blocks, so they degrade to readable code anywhere else:

| Fence | Renders as | Priority |
|---|---|---|
| ` ```mermaid ` | Diagram image (flowchart, sequence, class, state, ER, pie, gantt, mindmap...) | P0 |
| ` ```openui ` | Native interactive UI from an OpenUI Lang program (section 5) | P0 |
| ` ```html ` | Code block as today, plus a **Preview** button that opens it as a mini app | P1 |
| ` ```svg ` | The SVG drawn as an image, with a "Show source" toggle | P2 |

Every rendered block keeps a small header row with **Copy source** and **Show source**
(toggles back to the plain code block), so a bad render never hides the content.

Non-goals: arbitrary JS in the transcript, OpenUI `Query`/`Mutation` (tool calls from UI),
React/Zod component libraries, the OpenUI Gateway or Cloud.

## 2. Deliver in phases, commit after each

1. **Mermaid** (smallest visible win, proves the markdown hook).
2. **OpenUI Lang parser + evaluator**, pure Rust, no GPUI, with unit tests.
3. **OpenUI renderer** (static components first, then state, inputs, actions).
4. **HTML preview** and **SVG**.
5. **Agent instructions + setting + README.**

Each phase must compile (`cargo check`), pass `cargo test`, and be committed with a message
in the repo's style ("Render mermaid blocks in answers as diagrams"). Do not batch phases.

## 3. Environment and build facts

- Toolchain is not on PATH: `export PATH=$HOME/.cargo/bin:$PATH`.
- Cold `cargo check` takes about 7 minutes on this laptop (i7-8550U). Incremental checks are
  much faster. Run long builds in the background and keep working.
- Binary: `cargo run` (debug) or `./build.sh` (release). The desktop is Hyprland (Omarchy),
  `grim` can screenshot for visual checks.
- Tests: GPUI exports its own `test` attribute. Don't glob-import into test modules, or use
  `#[::core::prelude::v1::test]` as `src/ui/workspace.rs` does.
- Style: doc comments in plain prose, explain why. No em dashes in comments, docs or UI text.
- Deps: `gpui-kit = "=0.7.1"` (re-exports gpui, `gpui_kit::component` = gpui-component,
  `gpui_kit::base` = gpui-base). Sources are in `~/.cargo/registry/src/*/gpui-*-0.7.1/` and
  `gpui-pre-0.3.8` for core GPUI. Read them when unsure; don't guess APIs.

## 4. Where it hooks in (verified)

### 4.1 The Markdown view

- Answers render through `pub(super) fn markdown(state, p) -> TextView` in
  `src/ui/chat.rs` (around line 1993). It already adds a copy button per code block with
  `.code_block_actions(...)`. Its only transcript caller is `render_assistant` in
  `src/ui/chat/timeline.rs` (around line 576/598). Plans use `TextView::markdown(...)` at
  timeline.rs ~917 and ~1165 (optional to hook).
- The state is `Entity<TextViewState>` per answer, created in `src/session.rs`
  (`push_assistant_text` ~3779, `item_from_saved` ~4210). Streaming text is pushed with
  `AssistantText::show`.
- `gpui_kit::component::text::TextView` has the hooks needed:
  - `.markdown_block_parser(|node: &markdown_ast::Node, cx: &MarkdownParseContext| -> Option<MarkdownNode>)`
  - `.markdown_block_renderer(name, |node: &MarkdownNode, &mut Window, &mut App| -> impl IntoElement)`
  - or a `MarkdownPlugin` impl with `is_block() -> true` applied via `.plugin(...)`.
  - Types are re-exported from `gpui_kit::component::text`: `MarkdownNode`,
    `MarkdownParseContext`, `MarkdownPlugin`, `MarkdownExtensions`, `markdown_ast` (mdast).
- The parser sees mdast nodes. A fence is `markdown_ast::Node::Code(Code { value, lang, meta, .. })`.
  Match on `lang` (`mermaid`, `openui`, `svg`). Build the node with
  `MarkdownNode::new("openui", data).text(source).markdown(source)` where `source` is
  `cx.node_source(node)`, so selection and copy still yield the original fence. Read the
  data back in the renderer with `node.data::<T>()`. `node.source_range()` gives the byte
  range in the document.
- Constraints (from `gpui-base-0.7.1/src/text/markdown_ext.rs`):
  - Parsers and renderers must be `Send + Sync + 'static`. Parsers run on a background
    thread with no `Window`/`App`: do pure parsing only (the OpenUI parse is ideal here,
    Mermaid layout too).
  - Hooks are re-registered every frame by the builder. That is fine: the document is only
    reparsed when the *parser configuration* changes (number of parsers/renderers, their
    names, `parser_revision`). Keep the set of parsers and names constant.
  - Custom blocks are wrapped by the view in `div().pb(margin)`.
- Renderers get no context about which thread they're in. Pass it in: change `markdown()`
  to also take the session's `EntityId` (Copy, Send) and capture it in the renderer closure.
  `render_item` in timeline.rs has the `session: &Entity<Session>`.
- Streaming: with the default "paragraphs" streaming (`Streaming::Paragraphs`,
  `finished_prefix` in session.rs) whole code blocks tend to arrive at once, but with token
  streaming the fence arrives partially (an unclosed fence parses as code to the end).
  Renderers must accept partial input: render what resolves, show skeletons for the rest,
  never panic, and only show a parse error once the fence is closed.
- The transcript is a virtual list (`MessageScroller`, `src/ui/chat.rs` ~1452). If a block
  changes height after an interaction or an async render, call
  `scroller.remeasure(cx)` (see timeline.rs ~542 for the pattern), or the next frame may
  jump.

### 4.2 Per-block state and re-rendering

- Do not rely on `window.use_keyed_state` alone for values: virtual list rows that scroll
  out of view drop their element state. Keep UI values in a GPUI `Global`, e.g.
  `InlineUiStore { blocks: HashMap<(EntityId, u64 /* hash of source range + source */), BlockState> }`.
  `use_keyed_state` is fine for widget entities (`InputState`, `SliderState`) as long as
  they are re-seeded from the store when recreated.
- After a store change in an event handler, call `window.refresh()` so the custom block
  re-renders. If the `TextViewState` view is cached and does not repaint, also
  `cx.notify(text_view_state_entity_id)`.
- Optional: persist block state with the thread (`SavedItem` in session.rs) so forms keep
  values after restart. Not required for v1.

### 4.3 Talking back to the agent and opening previews

- A renderer only has `&mut App`. Add a `Global` hub, set up once in `Workspace::new`
  (`src/ui/workspace.rs`), holding `WeakEntity<Workspace>`, and a map
  `EntityId -> WeakEntity<Session>` (or look sessions up through the workspace).
- Sending a message: mirror `ChatView::send` (`src/ui/chat.rs` ~520-550): if the session is
  working, `queue_message` vs `send_message` per the `follow_up` setting, then scroll to end.
  `Session::send_message(text, Vec::new(), cx)` is at session.rs ~1253,
  `queue_message` ~1444.
- Message format from UI actions should be explicit so the agent knows it came from its UI,
  e.g. `[UI] Submitted "signup": {"name":"Ada","email":"ada@x.io"}` or the literal text of
  `@ToAssistant("...")`.
- Browser panel: `Workspace::open_surface(Surface::Browser, window, cx)`
  (`src/ui/workspace/surfaces.rs` ~62), panels made by `shown_browser` (workspace.rs ~1786),
  `BrowserPanel` (`src/ui/browser.rs`) owns a `BrowserSession` with `open(url)`
  (`src/browser.rs` ~522). Needs `agent-browser` installed (`browser::find_binary()`).
  Fallback: `workspace::open_url(url)` (xdg-open, workspace.rs ~2362).

### 4.4 Telling the agents

- Claude and Pi: `AgentProcess::spawn` in `src/claude.rs` (~174-200) passes
  `--append-system-prompt <prompt>` only for the browser prompt today. Build one combined
  prompt (browser prompt if any + inline UI prompt if enabled) and pass the flag once.
- jcode: system prompt can't be extended. Instructions ride along with the first message as
  `system_reminder`, see `src/session.rs` ~1600 (`orchestrator::INSTRUCTIONS` and
  `browser_note` are joined there). Add the inline UI prompt to that list.
- Put the prompt text in a `pub const INSTRUCTIONS: &str` in the new module (pattern:
  `src/orchestrator.rs` ~44). Keep it short, under ~60 lines: when to use each block,
  the component signatures, two examples, and the rule "plain Markdown first, UI only when
  it helps".

### 4.5 Theming

- `metal::palette(cx) -> &'static Palette` (`src/ui/metal.rs`): `page, s_hi, s, s_lo,
  well_top, well_bot, ink, ink2, ink3, rule, primary, green_ink, red, amber, blue_ink,
  tints[6], dark`, shadow recipes. Surface helpers via `MetalStyled`: `.plate(p)`,
  `.raised(p)`, `.well(p)`, `.region(p)`, `.engraved(p)`. Also `metal::tag`, `metal::rule`,
  `metal::led`, `metal::toggle`, `metal::segmented`, `MetalButton::new(id)
  .primary()/.ghost()/.danger()/.compact()/.mini()/.label()/.icon()/.on_click()`, and
  `.dropdown_menu(|menu, _, _| menu.item(PopupMenuItem::new(..).checked(..).on_click(..)))`.
- Fonts: `metal::SANS` ("Geist"), `metal::MONO` ("Geist Mono").
- Code wells look: see `markdown_style` in chat.rs ~1973. UI blocks should sit on a
  `.raised(p)` or `.region(p)` card with 14px radius so they read as objects, not prose.

## 5. Block kinds in detail

### 5.1 Mermaid (P0)

- Dependency: `mermaid-rs-renderer = { version = "0.3.1", default-features = false }`
  (pure Rust, 23 diagram types; default features pull clap and a second resvg version, so
  keep them off).
- API: `mermaid_rs_renderer::render_with_options(src, RenderOptions { theme, layout })
  -> anyhow::Result<String>` (SVG text). `Theme` has public `String` color fields
  (`font_family, font_size, primary_color, primary_text_color, primary_border_color,
  line_color, secondary_color, tertiary_color, edge_label_background, cluster_background,
  cluster_border, background, text_color, sequence_*, pie_colors[12], ...`). Start from
  `Theme::modern()` (light) or `Theme::dark()` and override from the palette (convert
  `Hsla` to hex via `gpui::Rgba::from(hsla)`). Make the background transparent or the
  palette's surface color.
- Fonts: the crate measures text with system fonts (fontdb), and GPUI rasterizes SVG with
  system fonts plus bundled assets. Use a family that exists on the system for both (e.g.
  `"DejaVu Sans, sans-serif"`), not Geist, or labels overflow their boxes.
- Rendering: do parse + layout + SVG generation off the main thread
  (`cx.background_spawn`), the first call loads the system font DB and is slow. Rasterize
  with `cx.svg_renderer().render_single_frame(svg_bytes, window.scale_factor())`
  -> `Arc<RenderImage>`, and draw with `img(render_image)`. `ImageFormat::Svg` through
  `Image::from_bytes` renders at scale 1.0 and looks blurry on HiDPI, avoid it.
- Cache rendered images in a `Global` keyed by `(hash(src), dark, scale)`. `RenderImage`
  ids are unique per render, so never re-render per frame. Drop evicted images with
  `window.drop_image(...)`/`cx.drop_image(...)`.
- Size: intrinsic width from the SVG (`width`/`height` attrs or `mermaid_rs_renderer::measure`),
  `max_w_full()`, keep aspect ratio (`.aspect_ratio(w / h)`). Center small diagrams. Wide
  diagrams: horizontal scroll, or click to open full size in an overlay (nice to have).
- While rendering: skeleton of a guessed height. On error: the code block plus a one-line
  error in `p.red`.

### 5.2 OpenUI Lang (P0)

Follow OpenUI Lang v0.5 (spec: https://www.openui.com/docs/openui-lang/specification-v05.mdx,
syntax: https://www.openui.com/docs/openui-lang/syntax.mdx) so models that know OpenUI
write valid blocks. Implement a subset.

**Module layout** (keep the language pure, it gets the unit tests):

- `src/inline_ui/lang.rs`: lexer + parser -> `Program { statements: Vec<(Ident, Expr)> }`.
- `src/inline_ui/eval.rs`: evaluator -> `Value` tree, given state.
- `src/ui/inline/mod.rs`: markdown hooks, globals (store, hub, cache).
- `src/ui/inline/mermaid.rs`, `src/ui/inline/openui.rs`, `src/ui/inline/html.rs`.

**Grammar** (one statement per line, `identifier = Expression`):

- Statements: component/value `name = Expr`, state `$name = default`.
- Expressions: component call `Type(args...)` (positional only), builtin `@Name(args)`,
  string `"..."` (with `\"` `\n` escapes), number, `true`/`false`/`null`, array `[..]`,
  object `{key: value}`, reference `ident`, state ref `$ident`, member access `a.b.c`
  (on arrays it plucks the field from every element), ternary `c ? a : b`, binary
  `+ - * / % == != > < >= <= && ||`, unary `! -`. Precedence as in C/JS.
- `//` line comments. Blank lines ignored. Forward references allowed.
- Root: the statement named `root` if present, else the first statement.
- Streaming tolerance: drop the last line if it doesn't parse and the fence is still open.
  An unresolved reference evaluates to `Value::Pending` and renders as a skeleton.
  Cycles evaluate to an error value, not a stack overflow (depth limit ~64).

**Values**: `Null, Bool, Number(f64), String, Array(Vec), Object(BTreeMap),
Component { name, args: Vec<Value> }, Action(Vec<Step>), State(name), Pending, Error(msg)`.
Evaluate the whole program on every render. Programs are small, keep it simple.

**Builtins**: `@Count @Sum @Avg @Min @Max @First @Last @Filter(array, field, op, value)
@Sort(array, field, dir?) @Round(n, decimals?) @Abs @Floor @Ceil
@Each(array, varName, template)`.

**Actions**: `Action([steps])` where steps are `@Set($var, value)`, `@Reset($a, $b)`,
`@ToAssistant("message")`, `@OpenUrl("url")`. Also accept v0.1 string actions on
buttons: `"submit:<form>"` (send the form's values to the agent), `"action:<name>"`
(send `[UI] action: <name>` to the agent). `Query`, `Mutation` and `@Run` are not
supported: render a small "not supported here" notice in their place.

**Components** (positional args in this order, optional ones may be omitted from the end).
Names and orders match OpenUI's default library where it has them:

| Component | Signature | Render |
|---|---|---|
| `Stack` | `(children, direction?="column"\|"row", gap?="s"\|"m"\|"l", align?, justify?, wrap?)` | flex |
| `Card` | `(children, variant?="raised"\|"well"\|"plain", direction?, gap?)` | MetalUI card |
| `CardHeader` | `(title, subtitle?)` | |
| `TextContent` | `(text, size?="small"\|"default"\|"large"\|"large-heavy")` | |
| `Markdown` | `(text)` | nested TextView |
| `Callout` | `(variant="info"\|"success"\|"warning"\|"error", title, description?)` | |
| `Tag` | `(text, variant?)` | `metal::tag` |
| `Separator` | `()` | `metal::rule` |
| `Progress` | `(value 0..1 or 0..100, label?)` | |
| `Stat` | `(label, value, delta?)` | KPI tile |
| `Image` | `(src, alt?)` | http(s) or data URI only |
| `CodeBlock` | `(language, code)` | |
| `Table` | `(columns)` with `Col(label, data, type?="text"\|"number"\|"tag")` | rows = longest column |
| `BarChart` | `(labels, series, variant?="grouped"\|"stacked", xLabel?, yLabel?)` with `Series(name, values)` | gpui chart |
| `LineChart` / `AreaChart` | `(labels, series, xLabel?, yLabel?)` | |
| `PieChart` | `(labels, values)` | |
| `Tabs` | `(items)` with `TabItem(value, trigger, content)` | `metal::segmented` + content |
| `Form` | `(name, fields, buttons)` | collects field values by `name` |
| `FormControl` | `(label, input, hint?)` | |
| `Input` | `(name or $var, placeholder?, type?="text"\|"email"\|"number"\|"password", rules?)` | |
| `TextArea` | `(name or $var, placeholder?, rows?)` | |
| `Select` | `(name or $var, items, placeholder?)`, items = strings or `{label, value}` | MetalButton + dropdown |
| `Checkbox` / `Switch` | `(name or $var, label)` | |
| `Slider` | `(name or $var, min, max, step?)` | |
| `Buttons` | `(buttons, direction?)` | |
| `Button` | `(label, action, variant?="primary"\|"secondary"\|"danger"\|"ghost")` | MetalButton |

Binding: a `$var` in an input's first argument binds it two-way; a plain string name keeps the
value in the enclosing `Form`. Expressions referencing `$vars` re-evaluate on change.

Unknown component: render its name as a tag and its children, so new names degrade softly.

**GPUI building blocks** (verified to exist in 0.7.1):

- Charts: `gpui_kit::component::chart::{BarChart, LineChart, AreaChart, PieChart}`.
  `BarChart::new(data).band(|d| label).value(|d| v)`, `AreaChart` takes several `.y(..)`
  for multi-series, `PieChart::new(data).value(..).color(..).label(..)`. `PlotValue` is
  implemented for `f32`/`f64`. Charts default their `ElementId` to the call site, so
  several charts share hover state: always call `.id(...)` with a per-block id.
  Give charts a fixed height (e.g. 220px).
- Inputs: `gpui_kit::component::input::{Input, InputState, InputEvent, Textarea, TextareaState}`.
  `InputState::new(window, cx).placeholder(..).default_value(..)`, `.value()`,
  `.set_value(v, window, cx)`, subscribe to `InputEvent` (`PressEnter`, change). The app
  draws inputs inside its own well with `Input::new(&state).appearance(false)`
  (`src/ui/browser.rs` ~393).
- Slider: `gpui_kit::component::slider::{Slider, SliderState, SliderEvent, SliderValue}`.
  `SliderState::new().min(..).max(..).step(..).default_value(..)`,
  `SliderEvent::Change(SliderValue::Single(f32))`.
- Checkbox/Switch: `gpui_kit::component::checkbox::Checkbox::new(id).label(..).checked(..)
  .on_click(|checked: &bool, window, cx| ..)`, or `metal::toggle(...)` for the app's look.

**Safety**: a block can only compose these components, open http(s) URLs after a click,
and send text to the agent after a click. No file or network access on render. Images from
remote URLs load only on click ("Load image") to avoid tracking pixels. Cap: 500 statements,
2000 rendered nodes, 10k table cells; beyond that, render an error.

### 5.3 HTML mini apps (P1)

- Extend the existing `code_block_actions` in `markdown()` (chat.rs): for `lang == "html"`
  add a **Preview** button next to Copy.
- On click: write the code to `settings::data_dir()/artifacts/<sha-or-hash>.html`
  (`settings::data_dir()` is in `src/settings.rs`), then open `file://...` in the thread's
  browser panel (section 4.3), falling back to `open_url` if agent-browser isn't installed.
- Preview is user-driven in v1. Don't tell the agent about the artifacts folder.

### 5.4 SVG (P2)

`lang == "svg"`: rasterize with `cx.svg_renderer().render_single_frame` like Mermaid, cap
size at 1MB of source, show source on error.

## 6. Setting

- `inline_ui: bool`, default `true`, in `Settings` (`src/settings.rs`, struct ~438, defaults
  ~577, the changed-settings labels list ~681). Device preference, not an environment key.
- Row in Settings → General → Behavior next to "Rich text composer"
  (`src/ui/settings/general.rs` ~384, use `row(...)`/`switch(...)`/`reset(...)` like it),
  search entry in `src/ui/settings/search.rs` (~83).
- Off: no agent instructions, and blocks render as plain code blocks.
- Description: "Let agents draw diagrams, charts and small interactive UIs in their answers.
  Applies to the agent instructions when a thread's agent next starts."

## 7. Agent instructions (draft, refine while testing)

~~~~text
## Inline UI in Vectrum Code

Your Markdown answers are shown in Vectrum Code, which renders some fenced blocks live:
- ```mermaid: diagrams (flowchart, sequenceDiagram, classDiagram, stateDiagram-v2, erDiagram, gantt, pie, mindmap).
- ```openui: native UI written in OpenUI Lang. One statement per line, `name = Component(args)`, positional args, `root = ...` first. State: `$name = default`. Actions: `Button("Go", Action([@ToAssistant("...")]))`, `@Set($x, v)`, `@Reset($x)`, `@OpenUrl("https://...")`.
  Components: Stack, Card, CardHeader, TextContent, Markdown, Callout, Tag, Separator, Progress, Stat, Image, CodeBlock, Table/Col, BarChart/LineChart/AreaChart(labels, [Series(name, values)]), PieChart(labels, values), Tabs/TabItem, Form, FormControl, Input, TextArea, Select, Checkbox, Switch, Slider, Buttons, Button.
- ```html: a self-contained page (inline CSS/JS); the user can open it with Preview.
Use them when a picture, table or control says it better than prose: architecture and flows, comparisons, results data, choices for the user to make. Keep prose for everything else. When the user acts on your UI you receive a message starting with [UI].

Example:
```openui
root = Card([header, form])
header = CardHeader("Deploy", "Pick a target")
form = Form("deploy", [FormControl("Environment", Select("env", ["staging", "production"]))], Buttons([Button("Deploy", "submit:deploy", "primary")]))
```
~~~~

## 8. Acceptance checks

Run the app (`cargo run`), start a Claude thread, and send each prompt. Screenshot each
result (`grim`) in both Bone and Graphite themes and look at it.

1. "Draw a mermaid flowchart of a login flow with 6 steps." -> crisp diagram, readable
   labels, colors fit the theme, no blur at the display scale.
2. "Show our 4 quarterly revenues as an openui bar chart and a table." -> chart + table in
   a card.
3. "Give me an openui form to pick a deploy target and confirm." -> select + button; clicking
   sends a `[UI] Submitted "deploy": {...}` message and the agent replies.
4. "Make an openui counter with + and - buttons using state." -> clicks update the number
   without a message to the agent; scrolling it away and back keeps the value.
5. "Write a small HTML snake game." -> Preview opens it in the browser panel (or the
   system browser without agent-browser).
6. Token streaming on (Settings → General → Response streaming by token): blocks build up
   without flicker, errors, or panics while the fence is incomplete.
7. A deliberately broken openui and mermaid block -> source shown with a one-line error.
8. Setting off -> plain code blocks, and the next agent start has no UI instructions.
9. Old threads restored from disk with these blocks render them too.

Unit tests (pure, in `src/inline_ui`): parse every example in this file and in the OpenUI
syntax page; operator precedence; member pluck on arrays; builtins; forward refs; cycles;
partial last line; `@Each`; actions; error values.

## 9. Done means

All phases committed, `cargo test` green, the 9 checks pass with screenshots looked at,
README "Features" gains a bullet describing inline diagrams, UI blocks and HTML previews.