emrg: GUI image input via clipboard paste (Cmd+V) and file drag-drop - #1109
Conversation
Render chain for GUI image attachments, aligned with the TUI's [📷 label]
placeholder semantics so both ends interoperate in the same session:
- Composer (tiptap): handlePaste detects clipboard image/* files (inserts
accompanying text/plain first, TUI behavior); handleDrop accepts dragged
image files; each image is base64-sent to the new emrg:saveImage IPC,
then a literal [📷 label] placeholder is inserted at the cursor and the
attachment recorded in pending state.
- main.js: new emrg:saveImage handler writes into the active session's
images directory ({safeLabel}_{hash8}.{ext}, TUI naming convention,
extension allowlist png/jpg/gif/webp/bmp/svg, dedup by existence);
sendMessage now validates and forwards an images array to sendTask
(daemon already supported it).
- preload.js exposes saveImage (invoke API 53 -> 54, guard test updated).
- composer.ts helpers (pure, unit-tested): imagePlaceholder/toSafeImageLabel
mirror TUI safe_label; normalizePlaceholders reverts tiptap-markdown's
backslash-escaped [📷 ...]; resolveSendImages keeps only attachments whose
placeholder is still in the text and computes character positions
(deleting the placeholder drops the image, TUI semantics). Images survive
the busy queue/requeue paths in both Composer and daemonBridge.
- daemon.py: _build_user_content now uses images[].mime for the data URL
instead of hardcoding image/png (GUI drag-in JPEG sent wrong mime);
defaults to png for TUI/history records. 2 new daemon tests.
Verification: pytest 1224 passed/1 skipped (1225 collected), renderer
typecheck clean + 508 tests/45 files, GUI node suite 93 pass/0 fail,
doc-count guard 5/5, import + CLI green. Agent.md counts synced.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle (review pass, 1/3)
Reviewed the full diff (10 files, +440/−17) against the rant's acceptance items and re-verified the chain end-to-end:
- Renderer collection (Composer.tsx handlePaste/handleDrop): clipboard image/* → preventDefault with accompanying text/plain inserted first (TUI parity); drag-drop filtered to image/*; attach flow inserts a literal [📷 label] placeholder at the cursor and records pending (path/label/mime). TDZ handled via ref-bridge into the one-shot tiptap closure; sid-switch drops pending (TUI semantics); failed saveImage skips the ghost placeholder.
- saveImage IPC (main.js): session-id anchored (no traversal), extension allowlist png/jpg/gif/webp/bmp/svg matching the renderer SUPPORTED_IMAGE_MIME set, label sanitized to the TUI safe_label rule, dedup by existence. sendMessage images validated (≤12, path/label/position/mime shape) before passthrough.
- Pure helpers (composer.ts): I verified against prosemirror-markdown's esc() that the serializer escapes BOTH brackets (regex includes [ and ]) so normalizePlaceholders' [📷 …] revert matches actual output; resolveSendImages drops placeholders deleted from text (TUI semantics) and computes positions via indexOf. Images survive the busy queue/requeue path in both Composer (queuedRef) and daemonBridge (queuedSends).
- Daemon mime fix: data URL now uses images[].mime with png default for TUI/history records; +2 tests cover mime-metadata and the non-vision degrade path.
Verification (all run locally this authoring cycle): pytest 1224 passed/1 skipped (1225 collected), renderer typecheck clean + vitest 508/45, GUI node suite 93 pass/0 fail (54-member preload contract), doc-count guard 5/5, import + CLI green. CI on this PR: test + test-windows both PASS (33605058214).
Note documented in code: the filename hash prefix differs between TUI (Python blake2b digest_size=4) and GUI (blake2b-512 first 8 hex) — same image pasted once in each client stores two files; self-consistent dedup per client, harmless. Not blocking.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle (independent review pass, 2/3)
Fresh pass this cycle (different cycle from the 1/3 reviewer): branch head 06c4549 unchanged since authoring, single clean commit on master b9e170c, CI test + test-windows both PASS (33605058214), MERGEABLE/CLEAN.
Re-verified the key acceptance items against the rant:
- Paste (Cmd+V) + drag-drop image collection in Composer with text-first insert parity to TUI; placeholder semantics [📷 label] identical (interop with TUI sessions preserved — normalizePlaceholders reverts the serializer's escaped [📷 …] form, matching prosemirror esc() behavior).
- saveImage IPC writes to the session images dir with TUI naming convention, session-id anchored (no traversal), extension allowlist consistent across main.js and renderer.
- Deleting the placeholder drops the image at send time (TUI semantics); images survive the busy-queue/requeue paths.
- daemon mime fix (images[].mime drives data URL, png default for TUI/history) with +2 tests; local pytest 1224 passed/1 skipped, renderer 508/45 + typecheck clean, GUI node suite 93 pass/0 fail.
No issues found. 2/3 — one more independent cycle LGTM required before merge.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle (independent review pass, 3/3)
Third distinct cycle review: head 06c4549 unchanged since authoring, CI test + test-windows PASS (33605058214), MERGEABLE/CLEAN, no ❌ in the review history (1/3 cyc155034, 2/3 cyc155801, 3/3 this cycle).
Re-confirmed the acceptance chain remains intact: Composer paste/drag image collection with TUI-identical [📷 label] placeholder semantics, saveImage IPC (session-anchored path, allowlisted extensions, dedup), images forwarded through sendMessage/sendTask with queue/requeue survival, and the daemon mime fix (data URL from images[].mime, png default) with its +2 tests. Local matrix green at authoring (pytest 1224/1 skipped, renderer 508/45 + typecheck, GUI 93/0).
3/3 reached — eligible for squash merge.
Summary
Add image input to the GUI composer — clipboard paste (Cmd+V) and drag-drop of image files — matching the TUI's established
[📷 label]placeholder semantics so both clients interoperate inside the same session (identical text sent to the LLM, interoperable history records).Implementation (bottom of the chain already worked; this adds the missing upper links)
Composer.tsx(tiptap)handlePaste: clipboard items containingimage/*→ preventDefault; accompanyingtext/plainis inserted first (TUI behavior: text paste then image); each image file goes through the attach flow.handleDrop+ containeronDragOver/onDropguards: dragged image files are attached.File → base64 (FileReader)→window.emrg.saveImageIPC → literal[📷 label]placeholder inserted at the cursor → attachment recorded in pending state (path/label/mime).main.js/preload.jsemrg:saveImageIPC: validates session id / base64 / label / mime (extension allowlist png/jpg/gif/webp/bmp/svg), writes into the active session's images directory as{safeLabel}_{hash8}.{ext}following the TUI naming convention (dedup by existence, session-id anchored — no traversal).emrg:sendMessagenow validates and forwards animagesarray tosendTask(daemon already supported it).preload.jsexposessaveImage— invoke API 53 → 54 (guard test updated).composer.ts(unit-tested, TUI-identical semantics)imagePlaceholder/toSafeImageLabel: mirror the TUI's placeholder literal andsafe_labelsanitization.normalizePlaceholders: reverts tiptap-markdown's backslash-escaped\[📷 …\]before sending so the daemon receives the exact same text as the TUI.resolveSendImages: keeps only attachments whose placeholder is still in the text and computes character positions (deleting the placeholder drops the image — same as TUI); images survive the busy queue/requeue paths in both Composer and daemonBridge.daemon.py_build_user_contenthardcodeddata:image/png;base64— a GUI drag-in JPEG would send the wrong MIME. It now readsimages[].mime(defaultimage/pngfor TUI/history records). 2 new daemon tests cover both mime-metadata and the non-vision degradation path.Verification
Note: the hash prefix differs across TUI (Python
blake2b(digest_size=4)) and GUI (blake2b-512 first 8 hex, Node crypto limitation) — dedup stays self-consistent per client; the same image pasted in both clients within one session creates two files (harmless, storage only). This is documented in the code comment.Visual interaction (paste shows placeholder, bubble echoes placeholder text) is host-verifiable; the placeholder/echo behavior itself is unit-tested.