Showing 1 changed file with 364 additions and 0 deletions.
| @@ -0,0 +1,364 @@ | ||
| 1 | + # Spec: inline UI, diagrams and mini apps in the transcript | |
| 2 | + | |
| 3 | + Status: ready to build. Written for an agent that has not seen the discussion behind it. | |
| 4 | + Repo: Vectrum Code (`/home/joni/Documents/DATA/Work/Coding/rust/code`), a GPUI desktop client | |
| 5 | + for Claude Code, Pi and jcode. Read `README.md` first for the app's shape. | |
| 6 | + | |
| 7 | + ## 1. Goal | |
| 8 | + | |
| 9 | + Agents can put rich, live output into their answers, like Claude's artifacts or | |
| 10 | + [OpenUI](https://www.openui.com/): diagrams, charts, cards, tables, small forms and test UIs, | |
| 11 | + and whole HTML mini apps. The agent writes them as fenced code blocks in its normal Markdown | |
| 12 | + answer. The app renders them natively in the transcript (MetalUI look, no webview), and | |
| 13 | + interactive ones can talk back to the agent. | |
| 14 | + | |
| 15 | + Four block kinds, all plain fenced code blocks, so they degrade to readable code anywhere else: | |
| 16 | + | |
| 17 | + | Fence | Renders as | Priority | | |
| 18 | + |---|---|---| | |
| 19 | + | ` ```mermaid ` | Diagram image (flowchart, sequence, class, state, ER, pie, gantt, mindmap...) | P0 | | |
| 20 | + | ` ```openui ` | Native interactive UI from an OpenUI Lang program (section 5) | P0 | | |
| 21 | + | ` ```html ` | Code block as today, plus a **Preview** button that opens it as a mini app | P1 | | |
| 22 | + | ` ```svg ` | The SVG drawn as an image, with a "Show source" toggle | P2 | | |
| 23 | + | |
| 24 | + Every rendered block keeps a small header row with **Copy source** and **Show source** | |
| 25 | + (toggles back to the plain code block), so a bad render never hides the content. | |
| 26 | + | |
| 27 | + Non-goals: arbitrary JS in the transcript, OpenUI `Query`/`Mutation` (tool calls from UI), | |
| 28 | + React/Zod component libraries, the OpenUI Gateway or Cloud. | |
| 29 | + | |
| 30 | + ## 2. Deliver in phases, commit after each | |
| 31 | + | |
| 32 | + 1. **Mermaid** (smallest visible win, proves the markdown hook). | |
| 33 | + 2. **OpenUI Lang parser + evaluator**, pure Rust, no GPUI, with unit tests. | |
| 34 | + 3. **OpenUI renderer** (static components first, then state, inputs, actions). | |
| 35 | + 4. **HTML preview** and **SVG**. | |
| 36 | + 5. **Agent instructions + setting + README.** | |
| 37 | + | |
| 38 | + Each phase must compile (`cargo check`), pass `cargo test`, and be committed with a message | |
| 39 | + in the repo's style ("Render mermaid blocks in answers as diagrams"). Do not batch phases. | |
| 40 | + | |
| 41 | + ## 3. Environment and build facts | |
| 42 | + | |
| 43 | + - Toolchain is not on PATH: `export PATH=$HOME/.cargo/bin:$PATH`. | |
| 44 | + - Cold `cargo check` takes about 7 minutes on this laptop (i7-8550U). Incremental checks are | |
| 45 | + much faster. Run long builds in the background and keep working. | |
| 46 | + - Binary: `cargo run` (debug) or `./build.sh` (release). The desktop is Hyprland (Omarchy), | |
| 47 | + `grim` can screenshot for visual checks. | |
| 48 | + - Tests: GPUI exports its own `test` attribute. Don't glob-import into test modules, or use | |
| 49 | + `#[::core::prelude::v1::test]` as `src/ui/workspace.rs` does. | |
| 50 | + - Style: doc comments in plain prose, explain why. No em dashes in comments, docs or UI text. | |
| 51 | + - Deps: `gpui-kit = "=0.7.1"` (re-exports gpui, `gpui_kit::component` = gpui-component, | |
| 52 | + `gpui_kit::base` = gpui-base). Sources are in `~/.cargo/registry/src/*/gpui-*-0.7.1/` and | |
| 53 | + `gpui-pre-0.3.8` for core GPUI. Read them when unsure; don't guess APIs. | |
| 54 | + | |
| 55 | + ## 4. Where it hooks in (verified) | |
| 56 | + | |
| 57 | + ### 4.1 The Markdown view | |
| 58 | + | |
| 59 | + - Answers render through `pub(super) fn markdown(state, p) -> TextView` in | |
| 60 | + `src/ui/chat.rs` (around line 1993). It already adds a copy button per code block with | |
| 61 | + `.code_block_actions(...)`. Its only transcript caller is `render_assistant` in | |
| 62 | + `src/ui/chat/timeline.rs` (around line 576/598). Plans use `TextView::markdown(...)` at | |
| 63 | + timeline.rs ~917 and ~1165 (optional to hook). | |
| 64 | + - The state is `Entity<TextViewState>` per answer, created in `src/session.rs` | |
| 65 | + (`push_assistant_text` ~3779, `item_from_saved` ~4210). Streaming text is pushed with | |
| 66 | + `AssistantText::show`. | |
| 67 | + - `gpui_kit::component::text::TextView` has the hooks needed: | |
| 68 | + - `.markdown_block_parser(|node: &markdown_ast::Node, cx: &MarkdownParseContext| -> Option<MarkdownNode>)` | |
| 69 | + - `.markdown_block_renderer(name, |node: &MarkdownNode, &mut Window, &mut App| -> impl IntoElement)` | |
| 70 | + - or a `MarkdownPlugin` impl with `is_block() -> true` applied via `.plugin(...)`. | |
| 71 | + - Types are re-exported from `gpui_kit::component::text`: `MarkdownNode`, | |
| 72 | + `MarkdownParseContext`, `MarkdownPlugin`, `MarkdownExtensions`, `markdown_ast` (mdast). | |
| 73 | + - The parser sees mdast nodes. A fence is `markdown_ast::Node::Code(Code { value, lang, meta, .. })`. | |
| 74 | + Match on `lang` (`mermaid`, `openui`, `svg`). Build the node with | |
| 75 | + `MarkdownNode::new("openui", data).text(source).markdown(source)` where `source` is | |
| 76 | + `cx.node_source(node)`, so selection and copy still yield the original fence. Read the | |
| 77 | + data back in the renderer with `node.data::<T>()`. `node.source_range()` gives the byte | |
| 78 | + range in the document. | |
| 79 | + - Constraints (from `gpui-base-0.7.1/src/text/markdown_ext.rs`): | |
| 80 | + - Parsers and renderers must be `Send + Sync + 'static`. Parsers run on a background | |
| 81 | + thread with no `Window`/`App`: do pure parsing only (the OpenUI parse is ideal here, | |
| 82 | + Mermaid layout too). | |
| 83 | + - Hooks are re-registered every frame by the builder. That is fine: the document is only | |
| 84 | + reparsed when the *parser configuration* changes (number of parsers/renderers, their | |
| 85 | + names, `parser_revision`). Keep the set of parsers and names constant. | |
| 86 | + - Custom blocks are wrapped by the view in `div().pb(margin)`. | |
| 87 | + - Renderers get no context about which thread they're in. Pass it in: change `markdown()` | |
| 88 | + to also take the session's `EntityId` (Copy, Send) and capture it in the renderer closure. | |
| 89 | + `render_item` in timeline.rs has the `session: &Entity<Session>`. | |
| 90 | + - Streaming: with the default "paragraphs" streaming (`Streaming::Paragraphs`, | |
| 91 | + `finished_prefix` in session.rs) whole code blocks tend to arrive at once, but with token | |
| 92 | + streaming the fence arrives partially (an unclosed fence parses as code to the end). | |
| 93 | + Renderers must accept partial input: render what resolves, show skeletons for the rest, | |
| 94 | + never panic, and only show a parse error once the fence is closed. | |
| 95 | + - The transcript is a virtual list (`MessageScroller`, `src/ui/chat.rs` ~1452). If a block | |
| 96 | + changes height after an interaction or an async render, call | |
| 97 | + `scroller.remeasure(cx)` (see timeline.rs ~542 for the pattern), or the next frame may | |
| 98 | + jump. | |
| 99 | + | |
| 100 | + ### 4.2 Per-block state and re-rendering | |
| 101 | + | |
| 102 | + - Do not rely on `window.use_keyed_state` alone for values: virtual list rows that scroll | |
| 103 | + out of view drop their element state. Keep UI values in a GPUI `Global`, e.g. | |
| 104 | + `InlineUiStore { blocks: HashMap<(EntityId, u64 /* hash of source range + source */), BlockState> }`. | |
| 105 | + `use_keyed_state` is fine for widget entities (`InputState`, `SliderState`) as long as | |
| 106 | + they are re-seeded from the store when recreated. | |
| 107 | + - After a store change in an event handler, call `window.refresh()` so the custom block | |
| 108 | + re-renders. If the `TextViewState` view is cached and does not repaint, also | |
| 109 | + `cx.notify(text_view_state_entity_id)`. | |
| 110 | + - Optional: persist block state with the thread (`SavedItem` in session.rs) so forms keep | |
| 111 | + values after restart. Not required for v1. | |
| 112 | + | |
| 113 | + ### 4.3 Talking back to the agent and opening previews | |
| 114 | + | |
| 115 | + - A renderer only has `&mut App`. Add a `Global` hub, set up once in `Workspace::new` | |
| 116 | + (`src/ui/workspace.rs`), holding `WeakEntity<Workspace>`, and a map | |
| 117 | + `EntityId -> WeakEntity<Session>` (or look sessions up through the workspace). | |
| 118 | + - Sending a message: mirror `ChatView::send` (`src/ui/chat.rs` ~520-550): if the session is | |
| 119 | + working, `queue_message` vs `send_message` per the `follow_up` setting, then scroll to end. | |
| 120 | + `Session::send_message(text, Vec::new(), cx)` is at session.rs ~1253, | |
| 121 | + `queue_message` ~1444. | |
| 122 | + - Message format from UI actions should be explicit so the agent knows it came from its UI, | |
| 123 | + e.g. `[UI] Submitted "signup": {"name":"Ada","email":"ada@x.io"}` or the literal text of | |
| 124 | + `@ToAssistant("...")`. | |
| 125 | + - Browser panel: `Workspace::open_surface(Surface::Browser, window, cx)` | |
| 126 | + (`src/ui/workspace/surfaces.rs` ~62), panels made by `shown_browser` (workspace.rs ~1786), | |
| 127 | + `BrowserPanel` (`src/ui/browser.rs`) owns a `BrowserSession` with `open(url)` | |
| 128 | + (`src/browser.rs` ~522). Needs `agent-browser` installed (`browser::find_binary()`). | |
| 129 | + Fallback: `workspace::open_url(url)` (xdg-open, workspace.rs ~2362). | |
| 130 | + | |
| 131 | + ### 4.4 Telling the agents | |
| 132 | + | |
| 133 | + - Claude and Pi: `AgentProcess::spawn` in `src/claude.rs` (~174-200) passes | |
| 134 | + `--append-system-prompt <prompt>` only for the browser prompt today. Build one combined | |
| 135 | + prompt (browser prompt if any + inline UI prompt if enabled) and pass the flag once. | |
| 136 | + - jcode: system prompt can't be extended. Instructions ride along with the first message as | |
| 137 | + `system_reminder`, see `src/session.rs` ~1600 (`orchestrator::INSTRUCTIONS` and | |
| 138 | + `browser_note` are joined there). Add the inline UI prompt to that list. | |
| 139 | + - Put the prompt text in a `pub const INSTRUCTIONS: &str` in the new module (pattern: | |
| 140 | + `src/orchestrator.rs` ~44). Keep it short, under ~60 lines: when to use each block, | |
| 141 | + the component signatures, two examples, and the rule "plain Markdown first, UI only when | |
| 142 | + it helps". | |
| 143 | + | |
| 144 | + ### 4.5 Theming | |
| 145 | + | |
| 146 | + - `metal::palette(cx) -> &'static Palette` (`src/ui/metal.rs`): `page, s_hi, s, s_lo, | |
| 147 | + well_top, well_bot, ink, ink2, ink3, rule, primary, green_ink, red, amber, blue_ink, | |
| 148 | + tints[6], dark`, shadow recipes. Surface helpers via `MetalStyled`: `.plate(p)`, | |
| 149 | + `.raised(p)`, `.well(p)`, `.region(p)`, `.engraved(p)`. Also `metal::tag`, `metal::rule`, | |
| 150 | + `metal::led`, `metal::toggle`, `metal::segmented`, `MetalButton::new(id) | |
| 151 | + .primary()/.ghost()/.danger()/.compact()/.mini()/.label()/.icon()/.on_click()`, and | |
| 152 | + `.dropdown_menu(|menu, _, _| menu.item(PopupMenuItem::new(..).checked(..).on_click(..)))`. | |
| 153 | + - Fonts: `metal::SANS` ("Geist"), `metal::MONO` ("Geist Mono"). | |
| 154 | + - Code wells look: see `markdown_style` in chat.rs ~1973. UI blocks should sit on a | |
| 155 | + `.raised(p)` or `.region(p)` card with 14px radius so they read as objects, not prose. | |
| 156 | + | |
| 157 | + ## 5. Block kinds in detail | |
| 158 | + | |
| 159 | + ### 5.1 Mermaid (P0) | |
| 160 | + | |
| 161 | + - Dependency: `mermaid-rs-renderer = { version = "0.3.1", default-features = false }` | |
| 162 | + (pure Rust, 23 diagram types; default features pull clap and a second resvg version, so | |
| 163 | + keep them off). | |
| 164 | + - API: `mermaid_rs_renderer::render_with_options(src, RenderOptions { theme, layout }) | |
| 165 | + -> anyhow::Result<String>` (SVG text). `Theme` has public `String` color fields | |
| 166 | + (`font_family, font_size, primary_color, primary_text_color, primary_border_color, | |
| 167 | + line_color, secondary_color, tertiary_color, edge_label_background, cluster_background, | |
| 168 | + cluster_border, background, text_color, sequence_*, pie_colors[12], ...`). Start from | |
| 169 | + `Theme::modern()` (light) or `Theme::dark()` and override from the palette (convert | |
| 170 | + `Hsla` to hex via `gpui::Rgba::from(hsla)`). Make the background transparent or the | |
| 171 | + palette's surface color. | |
| 172 | + - Fonts: the crate measures text with system fonts (fontdb), and GPUI rasterizes SVG with | |
| 173 | + system fonts plus bundled assets. Use a family that exists on the system for both (e.g. | |
| 174 | + `"DejaVu Sans, sans-serif"`), not Geist, or labels overflow their boxes. | |
| 175 | + - Rendering: do parse + layout + SVG generation off the main thread | |
| 176 | + (`cx.background_spawn`), the first call loads the system font DB and is slow. Rasterize | |
| 177 | + with `cx.svg_renderer().render_single_frame(svg_bytes, window.scale_factor())` | |
| 178 | + -> `Arc<RenderImage>`, and draw with `img(render_image)`. `ImageFormat::Svg` through | |
| 179 | + `Image::from_bytes` renders at scale 1.0 and looks blurry on HiDPI, avoid it. | |
| 180 | + - Cache rendered images in a `Global` keyed by `(hash(src), dark, scale)`. `RenderImage` | |
| 181 | + ids are unique per render, so never re-render per frame. Drop evicted images with | |
| 182 | + `window.drop_image(...)`/`cx.drop_image(...)`. | |
| 183 | + - Size: intrinsic width from the SVG (`width`/`height` attrs or `mermaid_rs_renderer::measure`), | |
| 184 | + `max_w_full()`, keep aspect ratio (`.aspect_ratio(w / h)`). Center small diagrams. Wide | |
| 185 | + diagrams: horizontal scroll, or click to open full size in an overlay (nice to have). | |
| 186 | + - While rendering: skeleton of a guessed height. On error: the code block plus a one-line | |
| 187 | + error in `p.red`. | |
| 188 | + | |
| 189 | + ### 5.2 OpenUI Lang (P0) | |
| 190 | + | |
| 191 | + Follow OpenUI Lang v0.5 (spec: https://www.openui.com/docs/openui-lang/specification-v05.mdx, | |
| 192 | + syntax: https://www.openui.com/docs/openui-lang/syntax.mdx) so models that know OpenUI | |
| 193 | + write valid blocks. Implement a subset. | |
| 194 | + | |
| 195 | + **Module layout** (keep the language pure, it gets the unit tests): | |
| 196 | + | |
| 197 | + - `src/inline_ui/lang.rs`: lexer + parser -> `Program { statements: Vec<(Ident, Expr)> }`. | |
| 198 | + - `src/inline_ui/eval.rs`: evaluator -> `Value` tree, given state. | |
| 199 | + - `src/ui/inline/mod.rs`: markdown hooks, globals (store, hub, cache). | |
| 200 | + - `src/ui/inline/mermaid.rs`, `src/ui/inline/openui.rs`, `src/ui/inline/html.rs`. | |
| 201 | + | |
| 202 | + **Grammar** (one statement per line, `identifier = Expression`): | |
| 203 | + | |
| 204 | + - Statements: component/value `name = Expr`, state `$name = default`. | |
| 205 | + - Expressions: component call `Type(args...)` (positional only), builtin `@Name(args)`, | |
| 206 | + string `"..."` (with `\"` `\n` escapes), number, `true`/`false`/`null`, array `[..]`, | |
| 207 | + object `{key: value}`, reference `ident`, state ref `$ident`, member access `a.b.c` | |
| 208 | + (on arrays it plucks the field from every element), ternary `c ? a : b`, binary | |
| 209 | + `+ - * / % == != > < >= <= && ||`, unary `! -`. Precedence as in C/JS. | |
| 210 | + - `//` line comments. Blank lines ignored. Forward references allowed. | |
| 211 | + - Root: the statement named `root` if present, else the first statement. | |
| 212 | + - Streaming tolerance: drop the last line if it doesn't parse and the fence is still open. | |
| 213 | + An unresolved reference evaluates to `Value::Pending` and renders as a skeleton. | |
| 214 | + Cycles evaluate to an error value, not a stack overflow (depth limit ~64). | |
| 215 | + | |
| 216 | + **Values**: `Null, Bool, Number(f64), String, Array(Vec), Object(BTreeMap), | |
| 217 | + Component { name, args: Vec<Value> }, Action(Vec<Step>), State(name), Pending, Error(msg)`. | |
| 218 | + Evaluate the whole program on every render. Programs are small, keep it simple. | |
| 219 | + | |
| 220 | + **Builtins**: `@Count @Sum @Avg @Min @Max @First @Last @Filter(array, field, op, value) | |
| 221 | + @Sort(array, field, dir?) @Round(n, decimals?) @Abs @Floor @Ceil | |
| 222 | + @Each(array, varName, template)`. | |
| 223 | + | |
| 224 | + **Actions**: `Action([steps])` where steps are `@Set($var, value)`, `@Reset($a, $b)`, | |
| 225 | + `@ToAssistant("message")`, `@OpenUrl("url")`. Also accept v0.1 string actions on | |
| 226 | + buttons: `"submit:<form>"` (send the form's values to the agent), `"action:<name>"` | |
| 227 | + (send `[UI] action: <name>` to the agent). `Query`, `Mutation` and `@Run` are not | |
| 228 | + supported: render a small "not supported here" notice in their place. | |
| 229 | + | |
| 230 | + **Components** (positional args in this order, optional ones may be omitted from the end). | |
| 231 | + Names and orders match OpenUI's default library where it has them: | |
| 232 | + | |
| 233 | + | Component | Signature | Render | | |
| 234 | + |---|---|---| | |
| 235 | + | `Stack` | `(children, direction?="column"\|"row", gap?="s"\|"m"\|"l", align?, justify?, wrap?)` | flex | | |
| 236 | + | `Card` | `(children, variant?="raised"\|"well"\|"plain", direction?, gap?)` | MetalUI card | | |
| 237 | + | `CardHeader` | `(title, subtitle?)` | | | |
| 238 | + | `TextContent` | `(text, size?="small"\|"default"\|"large"\|"large-heavy")` | | | |
| 239 | + | `Markdown` | `(text)` | nested TextView | | |
| 240 | + | `Callout` | `(variant="info"\|"success"\|"warning"\|"error", title, description?)` | | | |
| 241 | + | `Tag` | `(text, variant?)` | `metal::tag` | | |
| 242 | + | `Separator` | `()` | `metal::rule` | | |
| 243 | + | `Progress` | `(value 0..1 or 0..100, label?)` | | | |
| 244 | + | `Stat` | `(label, value, delta?)` | KPI tile | | |
| 245 | + | `Image` | `(src, alt?)` | http(s) or data URI only | | |
| 246 | + | `CodeBlock` | `(language, code)` | | | |
| 247 | + | `Table` | `(columns)` with `Col(label, data, type?="text"\|"number"\|"tag")` | rows = longest column | | |
| 248 | + | `BarChart` | `(labels, series, variant?="grouped"\|"stacked", xLabel?, yLabel?)` with `Series(name, values)` | gpui chart | | |
| 249 | + | `LineChart` / `AreaChart` | `(labels, series, xLabel?, yLabel?)` | | | |
| 250 | + | `PieChart` | `(labels, values)` | | | |
| 251 | + | `Tabs` | `(items)` with `TabItem(value, trigger, content)` | `metal::segmented` + content | | |
| 252 | + | `Form` | `(name, fields, buttons)` | collects field values by `name` | | |
| 253 | + | `FormControl` | `(label, input, hint?)` | | | |
| 254 | + | `Input` | `(name or $var, placeholder?, type?="text"\|"email"\|"number"\|"password", rules?)` | | | |
| 255 | + | `TextArea` | `(name or $var, placeholder?, rows?)` | | | |
| 256 | + | `Select` | `(name or $var, items, placeholder?)`, items = strings or `{label, value}` | MetalButton + dropdown | | |
| 257 | + | `Checkbox` / `Switch` | `(name or $var, label)` | | | |
| 258 | + | `Slider` | `(name or $var, min, max, step?)` | | | |
| 259 | + | `Buttons` | `(buttons, direction?)` | | | |
| 260 | + | `Button` | `(label, action, variant?="primary"\|"secondary"\|"danger"\|"ghost")` | MetalButton | | |
| 261 | + | |
| 262 | + Binding: a `$var` in an input's first argument binds it two-way; a plain string name keeps the | |
| 263 | + value in the enclosing `Form`. Expressions referencing `$vars` re-evaluate on change. | |
| 264 | + | |
| 265 | + Unknown component: render its name as a tag and its children, so new names degrade softly. | |
| 266 | + | |
| 267 | + **GPUI building blocks** (verified to exist in 0.7.1): | |
| 268 | + | |
| 269 | + - Charts: `gpui_kit::component::chart::{BarChart, LineChart, AreaChart, PieChart}`. | |
| 270 | + `BarChart::new(data).band(|d| label).value(|d| v)`, `AreaChart` takes several `.y(..)` | |
| 271 | + for multi-series, `PieChart::new(data).value(..).color(..).label(..)`. `PlotValue` is | |
| 272 | + implemented for `f32`/`f64`. Charts default their `ElementId` to the call site, so | |
| 273 | + several charts share hover state: always call `.id(...)` with a per-block id. | |
| 274 | + Give charts a fixed height (e.g. 220px). | |
| 275 | + - Inputs: `gpui_kit::component::input::{Input, InputState, InputEvent, Textarea, TextareaState}`. | |
| 276 | + `InputState::new(window, cx).placeholder(..).default_value(..)`, `.value()`, | |
| 277 | + `.set_value(v, window, cx)`, subscribe to `InputEvent` (`PressEnter`, change). The app | |
| 278 | + draws inputs inside its own well with `Input::new(&state).appearance(false)` | |
| 279 | + (`src/ui/browser.rs` ~393). | |
| 280 | + - Slider: `gpui_kit::component::slider::{Slider, SliderState, SliderEvent, SliderValue}`. | |
| 281 | + `SliderState::new().min(..).max(..).step(..).default_value(..)`, | |
| 282 | + `SliderEvent::Change(SliderValue::Single(f32))`. | |
| 283 | + - Checkbox/Switch: `gpui_kit::component::checkbox::Checkbox::new(id).label(..).checked(..) | |
| 284 | + .on_click(|checked: &bool, window, cx| ..)`, or `metal::toggle(...)` for the app's look. | |
| 285 | + | |
| 286 | + **Safety**: a block can only compose these components, open http(s) URLs after a click, | |
| 287 | + and send text to the agent after a click. No file or network access on render. Images from | |
| 288 | + remote URLs load only on click ("Load image") to avoid tracking pixels. Cap: 500 statements, | |
| 289 | + 2000 rendered nodes, 10k table cells; beyond that, render an error. | |
| 290 | + | |
| 291 | + ### 5.3 HTML mini apps (P1) | |
| 292 | + | |
| 293 | + - Extend the existing `code_block_actions` in `markdown()` (chat.rs): for `lang == "html"` | |
| 294 | + add a **Preview** button next to Copy. | |
| 295 | + - On click: write the code to `settings::data_dir()/artifacts/<sha-or-hash>.html` | |
| 296 | + (`settings::data_dir()` is in `src/settings.rs`), then open `file://...` in the thread's | |
| 297 | + browser panel (section 4.3), falling back to `open_url` if agent-browser isn't installed. | |
| 298 | + - Preview is user-driven in v1. Don't tell the agent about the artifacts folder. | |
| 299 | + | |
| 300 | + ### 5.4 SVG (P2) | |
| 301 | + | |
| 302 | + `lang == "svg"`: rasterize with `cx.svg_renderer().render_single_frame` like Mermaid, cap | |
| 303 | + size at 1MB of source, show source on error. | |
| 304 | + | |
| 305 | + ## 6. Setting | |
| 306 | + | |
| 307 | + - `inline_ui: bool`, default `true`, in `Settings` (`src/settings.rs`, struct ~438, defaults | |
| 308 | + ~577, the changed-settings labels list ~681). Device preference, not an environment key. | |
| 309 | + - Row in Settings → General → Behavior next to "Rich text composer" | |
| 310 | + (`src/ui/settings/general.rs` ~384, use `row(...)`/`switch(...)`/`reset(...)` like it), | |
| 311 | + search entry in `src/ui/settings/search.rs` (~83). | |
| 312 | + - Off: no agent instructions, and blocks render as plain code blocks. | |
| 313 | + - Description: "Let agents draw diagrams, charts and small interactive UIs in their answers. | |
| 314 | + Applies to the agent instructions when a thread's agent next starts." | |
| 315 | + | |
| 316 | + ## 7. Agent instructions (draft, refine while testing) | |
| 317 | + | |
| 318 | + ~~~~text | |
| 319 | + ## Inline UI in Vectrum Code | |
| 320 | + | |
| 321 | + Your Markdown answers are shown in Vectrum Code, which renders some fenced blocks live: | |
| 322 | + - ```mermaid: diagrams (flowchart, sequenceDiagram, classDiagram, stateDiagram-v2, erDiagram, gantt, pie, mindmap). | |
| 323 | + - ```openui: native UI written in OpenUI Lang. One statement per line, `name = Component(args)`, positional args, `root = ...` first. State: `$name = default`. Actions: `Button("Go", Action([@ToAssistant("...")]))`, `@Set($x, v)`, `@Reset($x)`, `@OpenUrl("https://...")`. | |
| 324 | + Components: Stack, Card, CardHeader, TextContent, Markdown, Callout, Tag, Separator, Progress, Stat, Image, CodeBlock, Table/Col, BarChart/LineChart/AreaChart(labels, [Series(name, values)]), PieChart(labels, values), Tabs/TabItem, Form, FormControl, Input, TextArea, Select, Checkbox, Switch, Slider, Buttons, Button. | |
| 325 | + - ```html: a self-contained page (inline CSS/JS); the user can open it with Preview. | |
| 326 | + Use them when a picture, table or control says it better than prose: architecture and flows, comparisons, results data, choices for the user to make. Keep prose for everything else. When the user acts on your UI you receive a message starting with [UI]. | |
| 327 | + | |
| 328 | + Example: | |
| 329 | + ```openui | |
| 330 | + root = Card([header, form]) | |
| 331 | + header = CardHeader("Deploy", "Pick a target") | |
| 332 | + form = Form("deploy", [FormControl("Environment", Select("env", ["staging", "production"]))], Buttons([Button("Deploy", "submit:deploy", "primary")])) | |
| 333 | + ``` | |
| 334 | + ~~~~ | |
| 335 | + | |
| 336 | + ## 8. Acceptance checks | |
| 337 | + | |
| 338 | + Run the app (`cargo run`), start a Claude thread, and send each prompt. Screenshot each | |
| 339 | + result (`grim`) in both Bone and Graphite themes and look at it. | |
| 340 | + | |
| 341 | + 1. "Draw a mermaid flowchart of a login flow with 6 steps." -> crisp diagram, readable | |
| 342 | + labels, colors fit the theme, no blur at the display scale. | |
| 343 | + 2. "Show our 4 quarterly revenues as an openui bar chart and a table." -> chart + table in | |
| 344 | + a card. | |
| 345 | + 3. "Give me an openui form to pick a deploy target and confirm." -> select + button; clicking | |
| 346 | + sends a `[UI] Submitted "deploy": {...}` message and the agent replies. | |
| 347 | + 4. "Make an openui counter with + and - buttons using state." -> clicks update the number | |
| 348 | + without a message to the agent; scrolling it away and back keeps the value. | |
| 349 | + 5. "Write a small HTML snake game." -> Preview opens it in the browser panel (or the | |
| 350 | + system browser without agent-browser). | |
| 351 | + 6. Token streaming on (Settings → General → Response streaming by token): blocks build up | |
| 352 | + without flicker, errors, or panics while the fence is incomplete. | |
| 353 | + 7. A deliberately broken openui and mermaid block -> source shown with a one-line error. | |
| 354 | + 8. Setting off -> plain code blocks, and the next agent start has no UI instructions. | |
| 355 | + 9. Old threads restored from disk with these blocks render them too. | |
| 356 | + | |
| 357 | + Unit tests (pure, in `src/inline_ui`): parse every example in this file and in the OpenUI | |
| 358 | + syntax page; operator precedence; member pluck on arrays; builtins; forward refs; cycles; | |
| 359 | + partial last line; `@Each`; actions; error values. | |
| 360 | + | |
| 361 | + ## 9. Done means | |
| 362 | + | |
| 363 | + All phases committed, `cargo test` green, the 9 checks pass with screenshots looked at, | |
| 364 | + README "Features" gains a bullet describing inline diagrams, UI blocks and HTML previews. | |