Skip to content

[Feature]: Support DataBlock in all provider message converters #1932

Description

@Buktal

Is your feature request related to a problem? Please describe.

DataBlock is documented as the forward-looking unified container for all binary modalities (image / audio / video / file), replacing the legacy ImageBlock / AudioBlock / VideoBlock subclasses. However, none of the provider message converters currently handle it:

  • DashScopeMessageConverterconvertToMultimodalContent() and hasMediaContent() skip DataBlock
  • GeminiMessageConverter — no DataBlock branch; extractSourceFromBlock() throws on DataBlock
  • AnthropicMessageConverter — no DataBlock branch
  • OpenAIMessageConverter — no DataBlock branch

As a result, when a user builds a message with DataBlock, every provider silently drops it. The LLM receives no media content and responds as if no resource was attached.

There is also a related gap in URLSource: it only stores a URL string, with no field to carry a MIME type hint. When the URL has no file extension (CDN URLs, signed URLs, API-generated links), MediaUtils.determineMediaType() falls back to application/octet-stream, making it impossible for converters to route the block to the correct image / audio / video slot.

Describe the solution you'd like

  1. Add an optional mimeType field to URLSource — fully backward-compatible (null means fall back to extension-based inference). This lets callers explicitly declare the media type when the URL has no extension.

  2. Add DataBlock support to all provider converters — using the following resolution order for MIME type:

    • Base64Source → use getMediaType() directly
    • URLSource → use mimeType field if set, otherwise fall back to MediaUtils.determineMediaType(url)
    • If MIME type cannot be resolved, throw a clear exception instead of silently skipping

    Route to the appropriate provider-native slot based on the resolved MIME prefix (image/*, audio/*, video/*).

Describe alternatives you've considered

  • Adding a mimeType field directly to DataBlock — rejected because Base64Source already carries mediaType and URLSource can carry it too; duplicating the field on DataBlock would be redundant.
  • Keeping DataBlock as an internal-only type and requiring callers to use legacy block types — contradicts the documented intent ("new code should prefer DataBlock").

Additional context

Affected files:

  • agentscope-core/.../message/URLSource.java — needs optional mimeType field
  • agentscope-core/.../formatter/dashscope/DashScopeMediaConverter.java
  • agentscope-core/.../formatter/dashscope/DashScopeMessageConverter.java
  • agentscope-core/.../formatter/gemini/GeminiMediaConverter.java
  • agentscope-core/.../formatter/gemini/GeminiMessageConverter.java
  • agentscope-core/.../formatter/anthropic/AnthropicMediaConverter.java
  • agentscope-core/.../formatter/anthropic/AnthropicMessageConverter.java
  • agentscope-extensions/.../openai/formatter/OpenAIMessageConverter.java

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions