Add multimodal input support for images and files

[wu-changxing]

Description

This PR adds comprehensive multimodal input support to the Connectonion framework, allowing agents to accept images and files alongside text prompts across all connection types (HTTP, WebSocket, and relay). Includes file upload validation with configurable size limits and detailed documentation.

Type of Change

• New feature (non-breaking change which adds functionality)
• Documentation update

Changes Made

Core Features

• Agent multimodal input: Extended Agent.input() to accept images (list of base64 data URLs) and files (list of dicts with name and base64 data)
• Message formatting: Automatically converts multimodal inputs to proper LLM message format (text + image_url + file objects)
• File validation: Added validate_files() function in config.py to enforce configurable file size and count limits
• Host configuration: Added max_file_size (MB) and max_files_per_request parameters to host() function

API Endpoints

• POST /input: Now accepts optional images and files fields in request body
• WebSocket /ws: Supports images and files in INPUT messages
• GET /info: Returns new accepted_inputs field describing supported input types and file limits
• Error handling: Returns 400 status with descriptive error messages when file validation fails

Client Library

• connect.input(): Added images and files parameters for remote agent calls
• connect.input_async(): Async version also supports multimodal inputs
• Message building: Properly constructs WebSocket messages with multimodal content

Documentation

• Added "Multimodal Input (Images & Files)" section with examples for HTTP, WebSocket, and Python client
• Updated /info endpoint documentation with accepted_inputs schema
• Added file size limit configuration examples in host.yaml and code
• Updated interactive docs UI with image/file input fields and file attachment support

Testing

• Added unit tests for Agent.input() with images, files, and combined multimodal inputs
• Added tests for multiple files in single request
• Added HTTP endpoint tests verifying images/files are passed to handlers
• Added file validation tests (oversized files, too many files, edge cases)
• Added info_handler tests verifying accepted_inputs field with custom config

Example Usage

from connectonion import Agent, host

# Local agent with multimodal input
agent = Agent(name="analyzer")

# Send with images
result = agent.input("Describe this image", images=["data:image/png;base64,iVBORw0KGgo..."])

# Send with files
result = agent.input("Summarize this document", files=[
    {"name": "report.pdf", "data": "data:application/pdf;base64,JVBERi..."}
])

# Send with both
result = agent.input("Analyze", images=[...], files=[...])

# Configure file limits when hosting
host(create_agent, max_file_size=50, max_files_per_request=5)

# HTTP request with images
curl -X POST http://localhost:8000/input \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What do you see?",
    "images": ["data:image/png;base64,iVBORw0KGgo..."]
  }'

# HTTP request with files
curl -X POST http://localhost:8000/input \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Summarize this",
    "files": [{"name": "doc.pdf", "data": "data:application/pdf;base64,JVBERi..."}]
  }'

Testing

• Added unit tests for Agent multimodal input (images, files, combined)
• Added HTTP endpoint tests for image/file parameter passing and validation
• Added file validation tests (size limits, file count limits, edge cases)
• Added info_handler tests for accepted_inputs

https://claude.ai/code/session_019ktNJoNpDXEgiiZEf4ataq