drawgent: your coding agent on a live Excalidraw canvas#
drawgent connects your own Claude Code, Codex or opencode (your install, login, config and
repo) to an Excalidraw whiteboard. Ask for a diagram in the chat panel, or write AGENT: …
next to the part of a drawing you want changed. The agent looks at the canvas (screenshot +
scene), edits it live, checks the result, and marks the note DONE.
Quick start#
Prerequisite: one of claude, codex or opencode installed and logged in. The Claude and
Codex bridges also need Node.js ≥ 18 (npm).
drawgent setup claude # once per agent: claude | codex | opencode
cd ~/my-repo
drawgent up # new agent session in this repo + canvas in your browser
drawgent up --attach # or: pick one of your running sessions and connect the canvas to it
drawgent setup <agent>#
Checks everything once, fails with the exact fix when something is missing, and writes
~/.config/drawgent/config.toml:
-
The agent CLI. Your claude/codex/opencodeon PATH.
-
Login. claude auth status,codex login statusoropencode auth list.
-
ACP bridge. - opencode speaks ACP itself (opencode acp).
- Claude Code and Codex use the official ACP adapters. They are installed once into
~/.cache/drawgent/adapters(~60 MB), without their bundled agent binaries, and pointed at your CLI (CLAUDE_CODE_EXECUTABLE,CODEX_PATH).
- Setup then verifies the ACP handshake.
- opencode speaks ACP itself (
-
Canvas tools for attached sessions. Only Codex needs a change: codex mcp add drawgent -- drawgent mcp. Claude and opencode get the tools at attach time.
-
Headless Chrome for the renderer. Setup uses your Chrome/Chromium if you have one. Otherwise it proposes: - downloading Chrome Headless Shell (Chrome for Testing, ~120 MB, no sudo) into
~/.cache/drawgent/chrome, and telling you exactly which system libraries are missing, if any;
- installing Chromium with your package manager (apt,snap,dnf,pacman,zypper,apk,brew, ornixwithout sudo).
Non-interactive: --chrome download | system | /path/to/chrome.
- downloading Chrome Headless Shell (Chrome for Testing, ~120 MB, no sudo) into
drawgent up refuses to start until setup succeeded for that agent, or if something setup
recorded disappeared.
drawgent up#
Runs in the current directory (the workspace):
- starts the editor + API on 127.0.0.1:7300(next free port if taken);
- starts a new session of your agent over ACP, working in the workspace;
- opens your browser. On a headless box it prints the ssh -Lcommand instead.
The scene is kept in .drawgent/scene.json, which is git-ignored automatically.
drawgent up --attach [id]#
Connects the canvas to a session you already run. Without an id it lists the sessions it finds (current directory first) and lets you pick one:
Using the canvas#
- Chat panel (right side): send a request, watch replies and tool calls stream in, approve permission prompts, press Stop to cancel a turn.
- On the canvas: write a text starting with AGENT:next to or inside a shape, or draw an arrow from the note to a shape. It fires about 2.5 s after you stop typing, with its position, what it points at and what is nearby. The agent resolves it into a greenDONE: …note. Edit it back toAGENT:to send it again.
- Prompts are queued and run one at a time. A queued note that was already handled is skipped.
excalidraw.com rooms#
drawgent up --room 'https://excalidraw.com/#room=<id>,<key>'
- drawgent joins the room as a collaborator (🤖 Agent, with a cursor that follows its edits). Humans can stay on excalidraw.com: theirAGENT:notes reach your agent, and its edits appear there live.
- Traffic is end-to-end encrypted with the room key. Empty rooms are loaded from and saved to excalidraw's Firestore storage.
- The local editor mirrors the room and still has the chat panel.
Other commands#
- drawgent mcp: stdio MCP server with the canvas tools. Agents launch it; it finds the running- drawgent upby itself.
- drawgent serve: low-level server with explicit agents (- --agent claude,codex,opencodeas set up, or- name=commandfor any ACP agent); for scripts and containers.
- Options for up/serve:--port,--room,--token(API bearer +?token=in the URL),--permissions canvas|ask|all(defaultcanvas: drawing tools auto-approved, anything else asked in the chat panel),--data,--settle-ms.
Optional: Docker#
docker compose up --build runs a canvas server (drawgent + Chromium, no agents), e.g. to
host a shared canvas or a room bridge on a server. Agents are never bundled: they always run
with your own setup.
Build from source#
npm ci && npm run build # editor + renderer pages -> dist/ (embedded into the binary)
cargo install --path . # or: cargo build --release
Agent tools (MCP)#
get_scene, get_screenshot (vision; zoom with element_ids), add_elements (Excalidraw
skeletons; arrows bind by id and are routed edge-to-edge), add_mermaid (auto-layout),
update_elements (labels follow shapes, bound arrows re-route), delete_elements,
clear_canvas, list_instructions, resolve_instruction, set_status.
API#
GET /api/health · GET /api/scene · GET /api/screenshot?ids=&padding=&max= ·
POST|PATCH|DELETE /api/elements · POST /api/mermaid · POST /api/clear ·
GET /api/instructions · POST /api/instructions/{id}/resolve · POST /api/status ·
POST /api/chat {agent?, text} · WS /ws (browser sync + chat events)
Tests#
cargo test # fractional indices, room crypto/framing
node scripts/smoke.mjs [url] # chat turn + AGENT: note against a running drawgent
node scripts/e2e-browser.mjs [url] # real browser: chat panel + note typed on the canvas
node scripts/room-e2e.mjs # fresh excalidraw.com room ↔ drawgent, both directions
Layout#
src/ (Rust):
- main.rs: CLI.
- setup.rs,- config.rs,- chrome.rs: setup, config, renderer install.
- attach.rs: session discovery and picker.
- agents.rs+- acp.rs: ACP driver (new / fork sessions).
- live.rs: live opencode / Codex drivers.
- hub.rs: routing, notes, chat log.
- scene.rs: store and edit operations.
- renderer.rs: Chrome over CDP.
- mcp.rs: MCP server.
- room.rs: excalidraw.com client.
- fractional.rs,- geometry.rs,- el.rs: helpers.
web/: editor (main.jsx, chat.jsx) and renderer page (render.jsx).
Limits#
- The renderer needs Chrome (a native renderer is planned).
- Claude "attach" is a fork, because Claude Code has no public way to inject into a running terminal session.
- Codex live attach is implemented, but was not yet tested against a logged-in Codex.
- One scene per workspace. Images/files are not synced.