Skip to content

emrg: GUI image input via clipboard paste (Cmd+V) and file drag-drop - #1109

Merged
argszero merged 1 commit into
masterfrom
feature/gui-image-input
Sep 2, 2026
Merged

emrg: GUI image input via clipboard paste (Cmd+V) and file drag-drop#1109
argszero merged 1 commit into
masterfrom
feature/gui-image-input

Conversation

@argszero

@argszero argszero commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

Add image input to the GUI composer — clipboard paste (Cmd+V) and drag-drop of image files — matching the TUI's established [📷 label] placeholder semantics so both clients interoperate inside the same session (identical text sent to the LLM, interoperable history records).

Implementation (bottom of the chain already worked; this adds the missing upper links)

  • Renderer collection — Composer.tsx (tiptap)
    • handlePaste: clipboard items containing image/* → preventDefault; accompanying text/plain is inserted first (TUI behavior: text paste then image); each image file goes through the attach flow.
    • handleDrop + container onDragOver/onDrop guards: dragged image files are attached.
    • Attach flow: File → base64 (FileReader)window.emrg.saveImage IPC → literal [📷 label] placeholder inserted at the cursor → attachment recorded in pending state (path/label/mime).
  • Disk + passthrough — main.js / preload.js
    • New emrg:saveImage IPC: validates session id / base64 / label / mime (extension allowlist png/jpg/gif/webp/bmp/svg), writes into the active session's images directory as {safeLabel}_{hash8}.{ext} following the TUI naming convention (dedup by existence, session-id anchored — no traversal).
    • emrg:sendMessage now validates and forwards an images array to sendTask (daemon already supported it).
    • preload.js exposes saveImage — invoke API 53 → 54 (guard test updated).
  • Pure helpers — composer.ts (unit-tested, TUI-identical semantics)
    • imagePlaceholder / toSafeImageLabel: mirror the TUI's placeholder literal and safe_label sanitization.
    • normalizePlaceholders: reverts tiptap-markdown's backslash-escaped \[📷 …\] before sending so the daemon receives the exact same text as the TUI.
    • resolveSendImages: keeps only attachments whose placeholder is still in the text and computes character positions (deleting the placeholder drops the image — same as TUI); images survive the busy queue/requeue paths in both Composer and daemonBridge.
  • Daemon latent-bug fix — daemon.py
    • _build_user_content hardcoded data:image/png;base64 — a GUI drag-in JPEG would send the wrong MIME. It now reads images[].mime (default image/png for TUI/history records). 2 new daemon tests cover both mime-metadata and the non-vision degradation path.

Verification

  • pytest: 1224 passed, 1 skipped (1225 collected; +2 daemon tests)
  • renderer typecheck clean; vitest 508 passed / 45 files (+9 composer helper tests)
  • GUI node suite: 93 pass / 0 fail (incl. the 54-member preload contract)
  • doc-count guard 5/5 (Agent.md counts synced); import + CLI green

Note: the hash prefix differs across TUI (Python blake2b(digest_size=4)) and GUI (blake2b-512 first 8 hex, Node crypto limitation) — dedup stays self-consistent per client; the same image pasted in both clients within one session creates two files (harmless, storage only). This is documented in the code comment.

Visual interaction (paste shows placeholder, bubble echoes placeholder text) is host-verifiable; the placeholder/echo behavior itself is unit-tested.

Render chain for GUI image attachments, aligned with the TUI's [📷 label]
placeholder semantics so both ends interoperate in the same session:

- Composer (tiptap): handlePaste detects clipboard image/* files (inserts
  accompanying text/plain first, TUI behavior); handleDrop accepts dragged
  image files; each image is base64-sent to the new emrg:saveImage IPC,
  then a literal [📷 label] placeholder is inserted at the cursor and the
  attachment recorded in pending state.
- main.js: new emrg:saveImage handler writes into the active session's
  images directory ({safeLabel}_{hash8}.{ext}, TUI naming convention,
  extension allowlist png/jpg/gif/webp/bmp/svg, dedup by existence);
  sendMessage now validates and forwards an images array to sendTask
  (daemon already supported it).
- preload.js exposes saveImage (invoke API 53 -> 54, guard test updated).
- composer.ts helpers (pure, unit-tested): imagePlaceholder/toSafeImageLabel
  mirror TUI safe_label; normalizePlaceholders reverts tiptap-markdown's
  backslash-escaped [📷 ...]; resolveSendImages keeps only attachments whose
  placeholder is still in the text and computes character positions
  (deleting the placeholder drops the image, TUI semantics). Images survive
  the busy queue/requeue paths in both Composer and daemonBridge.
- daemon.py: _build_user_content now uses images[].mime for the data URL
  instead of hardcoding image/png (GUI drag-in JPEG sent wrong mime);
  defaults to png for TUI/history records. 2 new daemon tests.

Verification: pytest 1224 passed/1 skipped (1225 collected), renderer
typecheck clean + 508 tests/45 files, GUI node suite 93 pass/0 fail,
doc-count guard 5/5, import + CLI green. Agent.md counts synced.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (review pass, 1/3)

Reviewed the full diff (10 files, +440/−17) against the rant's acceptance items and re-verified the chain end-to-end:

  • Renderer collection (Composer.tsx handlePaste/handleDrop): clipboard image/* → preventDefault with accompanying text/plain inserted first (TUI parity); drag-drop filtered to image/*; attach flow inserts a literal [📷 label] placeholder at the cursor and records pending (path/label/mime). TDZ handled via ref-bridge into the one-shot tiptap closure; sid-switch drops pending (TUI semantics); failed saveImage skips the ghost placeholder.
  • saveImage IPC (main.js): session-id anchored (no traversal), extension allowlist png/jpg/gif/webp/bmp/svg matching the renderer SUPPORTED_IMAGE_MIME set, label sanitized to the TUI safe_label rule, dedup by existence. sendMessage images validated (≤12, path/label/position/mime shape) before passthrough.
  • Pure helpers (composer.ts): I verified against prosemirror-markdown's esc() that the serializer escapes BOTH brackets (regex includes [ and ]) so normalizePlaceholders' [📷 …] revert matches actual output; resolveSendImages drops placeholders deleted from text (TUI semantics) and computes positions via indexOf. Images survive the busy queue/requeue path in both Composer (queuedRef) and daemonBridge (queuedSends).
  • Daemon mime fix: data URL now uses images[].mime with png default for TUI/history records; +2 tests cover mime-metadata and the non-vision degrade path.

Verification (all run locally this authoring cycle): pytest 1224 passed/1 skipped (1225 collected), renderer typecheck clean + vitest 508/45, GUI node suite 93 pass/0 fail (54-member preload contract), doc-count guard 5/5, import + CLI green. CI on this PR: test + test-windows both PASS (33605058214).

Note documented in code: the filename hash prefix differs between TUI (Python blake2b digest_size=4) and GUI (blake2b-512 first 8 hex) — same image pasted once in each client stores two files; self-consistent dedup per client, harmless. Not blocking.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (independent review pass, 2/3)

Fresh pass this cycle (different cycle from the 1/3 reviewer): branch head 06c4549 unchanged since authoring, single clean commit on master b9e170c, CI test + test-windows both PASS (33605058214), MERGEABLE/CLEAN.

Re-verified the key acceptance items against the rant:

  • Paste (Cmd+V) + drag-drop image collection in Composer with text-first insert parity to TUI; placeholder semantics [📷 label] identical (interop with TUI sessions preserved — normalizePlaceholders reverts the serializer's escaped [📷 …] form, matching prosemirror esc() behavior).
  • saveImage IPC writes to the session images dir with TUI naming convention, session-id anchored (no traversal), extension allowlist consistent across main.js and renderer.
  • Deleting the placeholder drops the image at send time (TUI semantics); images survive the busy-queue/requeue paths.
  • daemon mime fix (images[].mime drives data URL, png default for TUI/history) with +2 tests; local pytest 1224 passed/1 skipped, renderer 508/45 + typecheck clean, GUI node suite 93 pass/0 fail.

No issues found. 2/3 — one more independent cycle LGTM required before merge.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (independent review pass, 3/3)

Third distinct cycle review: head 06c4549 unchanged since authoring, CI test + test-windows PASS (33605058214), MERGEABLE/CLEAN, no ❌ in the review history (1/3 cyc155034, 2/3 cyc155801, 3/3 this cycle).

Re-confirmed the acceptance chain remains intact: Composer paste/drag image collection with TUI-identical [📷 label] placeholder semantics, saveImage IPC (session-anchored path, allowlisted extensions, dedup), images forwarded through sendMessage/sendTask with queue/requeue survival, and the daemon mime fix (data URL from images[].mime, png default) with its +2 tests. Local matrix green at authoring (pytest 1224/1 skipped, renderer 508/45 + typecheck, GUI 93/0).

3/3 reached — eligible for squash merge.

@argszero
argszero merged commit e46c160 into master Sep 2, 2026
2 checks passed
argszero added a commit that referenced this pull request Sep 2, 2026
… line, #1109 — GUI image input, #1110 — GUI stop button, #1111 — journal_prompt quality bar) (#1113)

Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant