Skip to content

emrg: read tool returns vision-format image blocks for local images (rant 2026-08-24T14:36:01) - #955

Merged
argszero merged 1 commit into
masterfrom
feature/read-tool-vision
Aug 24, 2026
Merged

emrg: read tool returns vision-format image blocks for local images (rant 2026-08-24T14:36:01)#955
argszero merged 1 commit into
masterfrom
feature/read-tool-vision

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Rant 2026-08-24T14:36:01: users with a vision-capable model (e.g. deepseek-v4-flash-vision-exp) had no way to let the model see a local image — read on a binary file returned UnicodeDecodeError → "binary file?" error, and bash base64 output is text, not a vision image block. Only clipboard paste (TUI /image) could reach vision format.

Changes

  • emrg/tools/read_tool.py: detect image extensions (.png/.jpg/.jpeg/.gif/.webp, case-insensitive) → return a structured JSON ref {"type": "image", "path": ..., "mime": ...} instead of a binary error; MAX_IMAGE_SIZE (256KB) cap with a resize hint (sips/ImageMagick) for oversized images; tool description updated to advertise image support.
  • emrg/server/daemon.py: new _tool_content_for_llm() helper wired into all 3 tool-message sites (main tool loop + 2 continue paths). It converts the read-tool image ref into an OpenAI vision content block (data-URL base64) for vision-capable models, degrades to a text placeholder for non-vision models, and passes non-image results through unchanged.
  • llm.py: no change needed — tool messages already pass content through as-is, and OpenAI/DeepSeek accept content arrays for tool messages.
  • tests: 5 read-tool tests (image ref, mime mapping, size cap, uppercase ext, description) + 4 daemon tests (vision conversion, non-vision degradation, passthrough, missing image). Agent.md count synced (1040 → 1049).

Acceptance

  • read /path/to/img.png returns a vision block instead of a binary error
  • vision-capable models can answer based on image content
  • non-vision models degrade to a text placeholder without error
  • existing text reads pass regression

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 144335+1. Reviewed against rant 2026-08-24T14:36:01:

  • read_tool.py returns a structured image ref ({"type":"image",path,mime}) for .png/.jpg/.jpeg/.gif/.webp instead of erroring on binary read; 256KB cap with a concrete resize hint; extension matching is case-insensitive.
  • daemon._tool_content_for_llm() correctly degrades: vision model → OpenAI [text, image_url] data-URL block; non-vision model → text placeholder; non-image results passthrough unchanged; missing file → graceful '[Image unavailable]' warning instead of a crash.
  • All 3 tool-message construction sites (main loop + 2 resume paths) are wired through the helper.
  • Tests cover vision/non-vision/passthrough/missing-image + read_tool mime/size/uppercase/description; Agent.md count synced 1049. CI green (test 1m21s + test-windows 59s, run 32698965107).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM (2/3) — cycle 2026-08-24T15:03, independent re-verification. Head cd2b995 unchanged, MERGEABLE/CLEAN, CI double-green (test 1m21s + test-windows 59s, run 32698965107). Change set matches the rant 2026-08-24T14:36:01 acceptance items (vision-format image refs from ReadTool; daemon-side conversion with non-vision degradation; 3 tool-message sites wired; 9 new tests; Agent.md count synced).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM (3/3) — cycle 2026-08-24T15:10, final independent verification. Head cd2b995 unchanged, MERGEABLE/CLEAN, CI double-green (test 1m21s + test-windows 59s, run 32698965107). Three consecutive approvals from three distinct cycles (145700, 150307, 151015), no requested changes in between. Change set verified against rant 2026-08-24T14:36:01: ReadTool returns vision-format image refs for local images, daemon._tool_content_for_llm converts to OpenAI vision content blocks with non-vision degradation, all 3 tool-message sites wired, 9 new tests, Agent.md count synced. Merging.

@argszero
argszero merged commit e4d6366 into master Aug 24, 2026
2 checks passed
@argszero
argszero deleted the feature/read-tool-vision branch August 24, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant