Skip to content

emrg: GUI image input via clipboard paste (Cmd+V) and file drag-drop #1109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
argszero merged 1 commit into from
Sep 2, 2026
Merged

emrg: GUI image input via clipboard paste (Cmd+V) and file drag-drop #1109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Agent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,9 @@ Community needs voiced in HN agent-UI discussions map directly to EMRG's design:
pkill -f "emrg.server"; rm -f ~/.emrg/emrgd.token; python -m emrg
```

Python: `uv run pytest tests/ -v` (1223) — import check: `uv run python -c "from emrg.client.app import run_client"`
Python: `uv run pytest tests/ -v` (1225) — import check: `uv run python -c "from emrg.client.app import run_client"`
GUI: `cd emrg/gui && npm test` (100: 44 daemon_client + 20 conn-manager + 7 integration + 7 nav-policy + 7 gui-state + 6 build-config + 4 boot-contract + 3 preload-api + 2 theme-guard) — syntax: `node --check main.js preload.js daemon_client.js`
Renderer: `cd emrg/gui/renderer && npm run typecheck && npm test` (499: 5 snapshot-store + 9 utils + 3 ErrorBoundary + 2 App smoke + 11 commands + 4 copywriting + 11 i18n + 11 markdown + 21 transcript + 10 TranscriptView + 15 history + 22 composer + 38 Composer + 6 LinkDialog + 16 sidebar + 17 Sidebar + 9 fileTree + 9 FileTree + 16 resultPanel + 8 ResultPanel + 27 workspaceView + 10 WorkspaceView + 10 dialog + 6 Dialog + 9 ConfirmDialog + 9 RenameDialog + 10 dialogLists + 3 HelpDialog + 9 MemoryDialog + 6 SkillsDialog + 8 openSession + 6 WelcomeDialog + 9 OpenSessionDialog + 7 NewSessionDialog + 7 rewind + 8 RewindDialog + 7 GithubDeviceDialog + 18 daemonBridge + 7 DaemonBridgeProvider + 30 Shell + 15 DialogHost + 20 SettingsPanel + 6 TaskFormDialog + 5 RantDialog + 4 vendorMarkdown) + `npm run build` → `renderer/dist/`
Renderer: `cd emrg/gui/renderer && npm run typecheck && npm test` (508: 5 snapshot-store + 9 utils + 3 ErrorBoundary + 2 App smoke + 11 commands + 4 copywriting + 11 i18n + 11 markdown + 21 transcript + 10 TranscriptView + 15 history + 31 composer + 38 Composer + 6 LinkDialog + 16 sidebar + 17 Sidebar + 9 fileTree + 9 FileTree + 16 resultPanel + 8 ResultPanel + 27 workspaceView + 10 WorkspaceView + 10 dialog + 6 Dialog + 9 ConfirmDialog + 9 RenameDialog + 10 dialogLists + 3 HelpDialog + 9 MemoryDialog + 6 SkillsDialog + 8 openSession + 6 WelcomeDialog + 9 OpenSessionDialog + 7 NewSessionDialog + 7 rewind + 8 RewindDialog + 7 GithubDeviceDialog + 18 daemonBridge + 7 DaemonBridgeProvider + 30 Shell + 15 DialogHost + 20 SettingsPanel + 6 TaskFormDialog + 5 RantDialog + 4 vendorMarkdown) + `npm run build` → `renderer/dist/`
CI: `uv run pytest` (ubuntu + **windows-2025 matrix** — Windows pytest 回归在 PR CI 即失败，v0.2.29 教训 #725) + GUI tests + **actionlint workflow lint** (`rhysd/actionlint@v1.7.12` gate, #444 — workflow 解析错误在 PR CI 即失败，如 `if:` secrets 上下文)
Re-trigger: `scripts/re-trigger-ci.sh [branch]` (workflow_dispatch, #527 — 替代空 commit 重触发：Actions outage 会整段丢弃 push 事件，dispatch 走 API 路径不受影响)
Git-over-https 兜底: `python scripts/sync-master-from-api.py [--repo owner/name] [--ref master]` — 受限网络下 github.com:443 不可达而 api.github.com 可达时，用 Git Data API 的 verification payload + signature 字节级重建上游 commit（含 web-flow GPG 签名 squash merge，reconstruct_commit 经 hermetic 测试验证 sha 一致）并推进本地 refs；内容对象缺失时 fail-loud 提示改用 git fetch（10+ 周期实证的恢复路径）
Expand Down
57 changes: 55 additions & 2 deletions emrg/gui/main.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const os = require("os");
const path = require("path");
const { pathToFileURL } = require("url");
const { spawn } = require("child_process");
const crypto = require("crypto");
const { parse: parseToml, stringify: stringifyToml } = require("smol-toml");
const { generateSessionId, SESSION_ID_RE } = require("./daemon_client");
const { ConnManager } = require("./conn-manager");
Expand Down Expand Up @@ -279,6 +280,25 @@ vision = false
return typeof t === "string" && t.length > 0 && t.length <= 20000;
}

// rant 2026-09-02T15:23:53：images 透传护栏（{path,label?,position?,mime?}[]）
function validateImages(imgs) {
if (imgs === null || imgs === undefined) return true;
if (!Array.isArray(imgs) || imgs.length > 12) return false;
for (const it of imgs) {
if (!it || typeof it !== "object") return false;
if (typeof it.path !== "string" || !it.path || it.path.length > 1024) return false;
if (it.label !== undefined && (typeof it.label !== "string" || it.label.length > 200)) return false;
if (
it.position !== undefined &&
(typeof it.position !== "number" || !Number.isInteger(it.position) || it.position < 0 || it.position > 20000)
) {
return false;
}
if (it.mime !== undefined && (typeof it.mime !== "string" || !/^image\/[\w.+-]+$/.test(it.mime))) return false;
}
return true;
}

function validateConfig(c) {
// 设计 §7.1：直接接收所需字段 + 基本类型检查（防写坏 config.toml 的健壮性，非安全设计）
const out = {};
Expand Down Expand Up @@ -336,9 +356,10 @@ vision = false
};
});

ipcMain.handle("emrg:sendMessage", async (_e, { sessionId, text, requestId, sandbox }) => {
ipcMain.handle("emrg:sendMessage", async (_e, { sessionId, text, requestId, sandbox, images }) => {
if (!validateSessionId(sessionId)) throw new Error("invalid session_id");
if (!validateText(text)) throw new Error("invalid text");
if (!validateImages(images)) throw new Error("invalid images"); // rant 2026-09-02T15:23:53
if (requestId !== undefined && (typeof requestId !== "string" || requestId.length < 8 || requestId.length > 64)) {
throw new Error("invalid request_id"); // G143：renderer 预生成 id 的格式护栏
}
Expand All @@ -355,14 +376,46 @@ vision = false
// G143：renderer 预生成 requestId（send 前标记自有流，消除 IPC 往返竞态窗口）
// Rant 2026-08-20T18:18：sandbox 档位（read-only / workspace-write / danger-full-access）
// G65：conn.sendTask 内部标记 ownStream（每连接独立锁）
rid = conn.sendTask({ sessionId, cwd: sessionCwd, prompt: text, requestId, sandbox });
// rant 2026-09-02T15:23:53：images 透传（daemon_client.sendTask 早已支持）
rid = conn.sendTask({ sessionId, cwd: sessionCwd, prompt: text, requestId, sandbox, images: images ?? null });
} catch (e) {
conn._releaseOwnStream(); // sendTask 抛异常（ws.send 失败）→ 释放锁，防 G65 锁泄漏
throw e;
}
return { ok: true, requestId: rid }; // G124：回传 requestId → renderer 识别自有流
});

// rant 2026-09-02T15:23:53：图片落盘——<projectRoot>/.emrg/sessions/<sid>/images/
// {safeLabel}_{blake2b4}.{ext}（TUI 同款命名约定：label 清洗 + 8-hex hash 前缀，
// 同图重复粘贴去重）。⚠️ 哈希算法与 TUI 不同：TUI 用 Python blake2b(digest_size=4)
// （Node crypto 不支持变长 blake2b），GUI 取 blake2b-512 前 8 hex —— 两端同字节
// 文件名 hash 不同（同一图在 TUI/GUI 各贴一次会各存一份，功能无碍，仅存储），
// GUI 自身去重自洽（同字节 → 同文件名 → 跳过写盘）。
ipcMain.handle("emrg:saveImage", async (_e, { sessionId, data, label, mime } = {}) => {
if (!validateSessionId(sessionId)) throw new Error("invalid session_id");
if (typeof data !== "string" || !data || data.length > 28_000_000) throw new Error("invalid image data"); // ~21MB base64
if (typeof label !== "string" || !label || label.length > 200) throw new Error("invalid label");
const mm = /^image\/([\w.+-]+)$/.exec(mime || "image/png");
if (!mm) throw new Error("invalid mime");
// 扩展名白名单（与 Composer SUPPORTED_IMAGE_MIME 一致）
const EXT_BY_SUB = { png: "png", jpeg: "jpg", gif: "gif", webp: "webp", bmp: "bmp", "svg+xml": "svg" };
const ext = EXT_BY_SUB[mm[1].toLowerCase()];
if (!ext) throw new Error("unsupported image type");
const buf = Buffer.from(data, "base64");
if (buf.length === 0) throw new Error("invalid image data");
const projectRoot = resolveSessionCwd(sessionId) || DEFAULT_CWD;
const imagesDir = path.join(projectRoot, ".emrg", "sessions", sessionId, "images");
fs.mkdirSync(imagesDir, { recursive: true });
// 文件名字面清洗镜像 TUI safe_label（. 保留以便 .png 类后缀友好，去尾部 ._）
const safeLabel = label.replace(/[^a-zA-Z0-9._-]/g, "_").slice(0, 40).replace(/[._]+$/, "") || "image";
const h = crypto.createHash("blake2b512").update(buf).digest("hex").slice(0, 8);
const filename = `${safeLabel}_${h}.${ext}`;
const finalPath = path.join(imagesDir, filename);
if (!fs.existsSync(finalPath)) fs.writeFileSync(finalPath, buf); // 同名去重
logger.info(`[gui:saveImage] ${filename} (${buf.length} bytes)`);
return { path: finalPath, mime: `image/${mm[1].toLowerCase()}` };
});

ipcMain.handle("emrg:listSessions", async () => listSessions());

// Rant 2026-08-21T12:44:34：一键"重启生效"。旧实现只发 shutdown —— daemon 会被
Expand Down
2 changes: 2 additions & 0 deletions emrg/gui/preload.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ const { contextBridge, ipcRenderer } = require("electron");
const api = {
init: () => ipcRenderer.invoke("emrg:init"),
sendMessage: (payload) => ipcRenderer.invoke("emrg:sendMessage", payload),
// rant 2026-09-02T15:23:53：图片落盘（粘贴/拖拽 → base64 → main 写 <cwd>/.emrg/sessions/<sid>/images/）
saveImage: (payload) => ipcRenderer.invoke("emrg:saveImage", payload),
listSessions: () => ipcRenderer.invoke("emrg:listSessions"),
restartDaemon: () => ipcRenderer.invoke("emrg:restartDaemon"),
relaunchGui: () => ipcRenderer.invoke("emrg:relaunchGui"),
Expand Down
Loading
Loading