Skip to content

Implement: multimodal MCP ContentBlock passthrough to vision-capable LLM providers #6229

Description

@bug-ops

Spec

See specs/072-multimodal-mcp-passthrough/ (added in PR #6228 on branch feat/issue-5366/multimodal-mcp-passthrough).

Summary

Implements true multimodal passthrough of MCP ContentBlock::Image content to vision-capable LLM providers. Currently zeph_mcp::render_content_blocks renders images only as a [kind: mime, N bytes] text placeholder. This adds an opt-in (per MCP server, default OFF) path that decodes, validates, and attaches the image as a sibling MessagePart::Image so vision-capable providers can actually see it — gated on vision-capable tier routing (degrades to the existing text placeholder rather than a 400/422 when unavailable), with binary content validated by a new MediaSanitizer and stripped from all persistence/embedding surfaces before it reaches SQLite, Qdrant, or the durable JSONL replay log.

Audio/blob passthrough is explicitly out of scope (deferred — adding a MessagePart variant is an Ask-First decision per system invariant #4).

Acceptance criteria

Full testable list in specs/072-multimodal-mcp-passthrough/spec.md §6 (AC-1 through AC-15). Highlights:

  • AC-5: no Image part bytes appear in SQLite parts_json, Qdrant embeddings, or the durable JSONL replay log — verified by an integration test asserting all three surfaces.
  • AC-3: ToolOutput/ImageData custom redacting Debug impl — no raw bytes in logs/debug-dumps.
  • AC-6: vision-tier routing gate — media-bearing turns select a vision-capable tier or degrade to the text placeholder; never a runtime 400/422.
  • AC-9: MediaSanitizer enforces magic-byte sniff, format allowlist, and size/dimension/pixel caps (5 MiB / 8192px / 64MP / 4-per-result / 8-per-turn defaults, tunable via config).
  • AC-15: the three pre-assembly passes (run_causal_ipi_post_probe, record_shadow_event, apply_acon_compression) remain Image-part-safe under refactor (invariant C5).

Per project rules, this touches provider request builders and MessagePart construction — a live cascade+MCP-image session test is mandatory before merge (LLM Serialization Gate, .claude/rules/continuous-improvement.md).

Two open questions remain in the spec (OQ-1, OQ-2) — resolve or explicitly re-affirm during implementation.

Suggested next step

Run /rust-agents:team-develop new-feature on this issue. The architect and developer will pick up specs/072-multimodal-mcp-passthrough/ automatically.

Metadata

Metadata

Assignees

Labels

llmzeph-llm crate (Ollama, Claude)

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions