agent.input() and /ws endpoint do not support file passing

[wu-changxing]

Problem

The agent.input() method and WebSocket /ws endpoint do not currently support passing files as input.

Context

When hosting agents via agent.host() or using co ai, users can only send text prompts through:

• agent.input(prompt) - Python API
• POST /input - HTTP endpoint
• /ws - WebSocket endpoint

There is no way to pass files (images, documents, data files) to the agent.

Expected Behavior

Users should be able to pass files to agents for processing:

• Vision tasks: Send images for analysis
• Document processing: Upload PDFs, text files
• Data analysis: Pass CSV, JSON files
• Multi-modal inputs: Combine text prompt + file(s)

Current Workaround

Users must:

1. Save files to disk first
2. Use file path in prompt
3. Agent reads via read_file tool

This doesn't work for remote/hosted agents where client and server have different filesystems.

Suggested Solution

Extend the input protocol to support file attachments:

Python API:

agent.input(
    "Analyze this image",
    files=[{"path": "screenshot.png", "data": base64_data}]
)

HTTP endpoint:

POST /input
Content-Type: multipart/form-data

prompt=Analyze+this+image
file=@screenshot.png

WebSocket:

{
  "type": "input",
  "prompt": "Analyze this image",
  "files": [{
    "name": "screenshot.png",
    "data": "base64...",
    "mime_type": "image/png"
  }]
}

Impact

• Cannot use vision models with hosted agents
• Cannot process user-uploaded files in remote sessions
• Limits usefulness of agent hosting feature

Related Components

• connectonion/core/agent.py - Agent.input() method
• connectonion/network/host.py - HTTP/WebSocket endpoints
• connectonion/cli/co_ai/ - co ai relay connections

[wu-changxing]

🔍 Summary

Add file passing support to agent.input(), POST /input, and WS /ws so users can send files (images, PDFs, CSVs, etc.) to agents — especially critical for remote/hosted agents where client and server have different filesystems.

Status: Core implementation is COMPLETE in the working tree. This plan documents the architecture, what's done, and remaining gaps.

📊 Current vs Proposed

CURRENT (Problem):
┌──────────┐                          ┌───────────┐
│  Client   │ ── text prompt only ──▶ │   Agent   │
└──────────┘                          └───────────┘
       │                                     │
       │  ❌ No way to send files            │  Agent has read_file tool
       │  ❌ Different filesystems            │  but can't reach client's files
       │                                     │
       └── Workaround: save to disk ────────▶│  Only works locally

PROPOSED (Solution):
┌──────────┐                          ┌───────────┐
│  Client   │ ── prompt + files[] ──▶ │   Agent   │
└──────────┘                          └───────────┘
       │                                     │
       │  ✅ base64 data URLs                │  Files saved to .co/uploads/
       │  ✅ Validated (size/count)          │  System reminder tells LLM to
       │  ✅ Works over HTTP + WS            │  read files with tools
       │                                     │
       └── Works for remote agents ─────────▶│  Files on server filesystem

📊 Data Flow — File Upload Lifecycle

Client sends files
       │
       ▼
┌──────────────────────┐
│  HTTP POST /input    │  files: [{name, data}]   OR
│  WS INPUT message    │  images: [data_url]
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  validate_files()    │  ← config.py
│  • count ≤ max_files │
│  • size ≤ max_size   │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  input_handler()     │  ← routes.py
│  • create fresh agent│
│  • merge session     │
│  • call agent.input()│
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  agent.input()       │  ← core/agent.py
│  1. Save files to    │
│     .co/uploads/     │  Path traversal protection
│  2. Build multimodal │
│     message:         │
│     • text prompt    │
│     • image_url[]    │  (images passed inline)
│     • system-reminder│  (file paths for LLM)
│  3. Run LLM loop     │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  LLM sees message:   │
│  "Analyze this doc"  │
│  + <system-reminder> │
│  uploaded files:     │
│  - .co/uploads/x.pdf │
│  Use read_file tool  │
└──────────────────────┘

📁 Files Changed

connectonion/
├── core/
│   └── agent.py              ← MODIFIED: input() accepts files=[], saves to .co/uploads/,
│                                builds multimodal message with system-reminder
├── network/
│   ├── asgi/
│   │   ├── http.py           ← MODIFIED: POST /input extracts files from JSON body
│   │   └── websocket.py      ← MODIFIED: INPUT message extracts files
│   └── host/
│       ├── routes.py         ← MODIFIED: input_handler() passes files to agent.input()
│       ├── server.py         ← MODIFIED: _create_route_handlers() wires validate_files()
│       └── config.py         ← MODIFIED: DEFAULT_FILE_LIMITS, validate_files()
tests/
├── unit/
│   ├── test_agent.py         ← MODIFIED: 4 new tests (single file, multi file,
│   │                            images+files, path traversal)
│   └── test_host_routes.py   ← MODIFIED: 5 new tests (validation: valid, oversized,
│                                too many, none, empty)

🏗️ Architecture — Message Format

Plain text input (no files):
┌────────────────────────────────────┐
│ { "role": "user",                  │
│   "content": "Hello" }             │
└────────────────────────────────────┘

Multimodal input (images + files):
┌────────────────────────────────────┐
│ { "role": "user",                  │
│   "content": [                     │
│     { "type": "text",              │
│       "text": "Analyze these" },   │  ← User's prompt
│     { "type": "image_url",         │
│       "image_url": { "url":        │  ← Inline image (vision models)
│         "data:image/png;base64,…"  │
│       }                            │
│     },                             │
│     { "type": "text",              │
│       "text": "<system-reminder>   │  ← File path reference
│         Uploaded files:            │     (LLM reads via tools)
│         - .co/uploads/report.pdf   │
│         Use read_file tool…        │
│       </system-reminder>"          │
│     }                              │
│   ] }                              │
└────────────────────────────────────┘

Key design: Images go inline (vision API). Non-image files are saved
to disk and referenced by path (LLM uses read_file tool to access).

🔄 Protocol — Client Message Formats

HTTP POST /input (JSON body):
{
  "prompt": "Analyze this document",
  "payload": { "timestamp": 1234 },
  "from": "0x...",
  "signature": "...",
  "images": ["data:image/png;base64,..."],     ← optional
  "files": [                                    ← optional
    { "name": "report.pdf",
      "data": "data:application/pdf;base64,..." }
  ],
  "session": { "session_id": "..." }           ← optional
}

WebSocket INPUT message:
{
  "type": "INPUT",
  "prompt": "Analyze this document",
  "images": ["data:image/png;base64,..."],     ← optional
  "files": [                                    ← optional
    { "name": "report.pdf",
      "data": "data:application/pdf;base64,..." }
  ],
  "session": { ... }                            ← optional
}

Python API:
agent.input(
    "Analyze this document",
    images=["data:image/png;base64,..."],       # optional
    files=[{                                    # optional
        "name": "report.pdf",
        "data": "data:application/pdf;base64,..."
    }]
)

🔒 Security — File Validation Pipeline

Incoming files[]
       │
       ├──▶ Count check: len(files) ≤ max_files_per_request (default: 10)
       │    ❌ ValueError: "Too many files: N (max: M)"
       │
       ├──▶ Size check (per file): len(data) ≤ max_file_size MB (default: 10MB)
       │    ❌ ValueError: "File too large: name.pdf (X.YMB, max: ZMB)"
       │
       ├──▶ Path traversal: Path(name).name strips ../../ prefixes
       │    "../../etc/passwd" → saved as "passwd" in .co/uploads/
       │
       └──▶ Configurable via .co/host.yaml:
            max_file_size: 10        # MB per file
            max_files_per_request: 10

📝 Implementation Steps

✅ Already Done (in working tree)

1. agent.input() file support — core/agent.py:226-296

   • Accepts files: list[dict] parameter
   • Saves to .co/uploads/ with path traversal protection
   • Builds multimodal message with system-reminder for LLM

2. HTTP endpoint — asgi/http.py:284-285

   • Extracts files from JSON body
   • Passes through to input_handler()

3. WebSocket endpoint — asgi/websocket.py:268

   • Extracts files from INPUT message
   • Passes through to ws_input handler

4. Route handlers — host/routes.py:27-29, 90

   • input_handler() accepts and passes files parameter
   • handle_input and handle_ws_input call validate_files()

5. Validation + config — host/config.py:24-153

   • DEFAULT_FILE_LIMITS with max_file_size (10MB) and max_files_per_request (10)
   • validate_files() checks count and size limits
   • Configurable via host.yaml

6. /info endpoint — host/routes.py:159-166

   • Advertises accepted_inputs.files with limits

7. Tests — 9 new tests

   • test_agent.py: single file, multiple files, images+files, path traversal
   • test_host_routes.py: valid files, oversized, too many, None, empty list

🔲 Potential Future Enhancements (not blocking)

1. Python API convenience for local paths

   • Accept files=["report.pdf", "/path/to/data.csv"] (string paths)
   • Auto-read and base64-encode local files
   • Only useful for local agent usage (not remote)

2. HTTP multipart/form-data

   • Current: JSON body only (base64 encoded)
   • Enhancement: Accept multipart uploads for large files
   • Would reduce memory usage (no base64 inflation)

3. Image-type file auto-detection

   • Files with image MIME types could be sent inline as image_url
   • Instead of saving to disk and referencing by path
   • Would enable vision model analysis without read_file tool

⚠️ Risks & Considerations

• Base64 overhead: Files are ~33% larger when base64 encoded. The 10MB limit effectively means ~7.5MB actual file size. This is fine for most use cases.
• Disk usage: Files saved to .co/uploads/ accumulate. No automatic cleanup yet. May want TTL-based cleanup in the future.
• Filename collisions: Multiple uploads with same filename overwrite each other. Could add timestamp/UUID prefix if needed.

🧪 Testing Strategy

• Unit tests (✅ done): File save, message format, path traversal, validation
• Integration tests: End-to-end HTTP POST with files, WebSocket INPUT with files
• Manual verification: Send file via chat.openonion.ai, verify agent reads it