fix(lark): ingest inbound images and videos as chat attachments (MUL-4934)

[beastpu]

What does this PR do?

Delivers inbound Feishu/Lark images and videos to agents as real Multica chat attachments instead of leaving only [Image] or [Video] placeholders. It covers standalone media messages and media embedded in rich-text post messages.

Thinking path:

1. The adapter already preserves a human-readable placeholder, but the normalized channel envelope has no persisted media reference.
2. Routing, dedup, group-addressing, identity checks, and placeholder persistence remain synchronous; remote media I/O is queued only after the message and dedup mark are durable, keeping it off the connector ACK path.
3. Media jobs are ordered per chat session. The tail job binds attachments and then schedules the debounced agent run, so later messages cannot overtake earlier media.
4. The resolver applies Lark's 100 MiB resource limit and a detached 45-second timeout. Known-length bodies stream to S3 with an explicit content length and unsigned SigV4 payload; unknown-length bodies use the bounded buffered upload path.
5. Download, upload, or attachment-binding failures remain non-fatal and retain the original placeholder text.

Related Issue

Closes #5579

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Refactor / code improvement (no behavior change)
• Documentation update
• Tests (adding or improving test coverage)
• CI / infrastructure

Changes Made

• Add an optional channel-engine MediaResolver seam and detach media processing from the connector ACK path.
• Preserve per-session message/media ordering before scheduling the agent run.
• Split durable message append from the short database-only attachment-binding transaction.
• Download Feishu/Lark message resources through the authenticated HTTP client with a 45-second timeout and 100 MiB size safeguard.
• Support standalone image/video messages plus embedded img and media spans in rich-text posts.
• Stream known-length uploads through local or S3 storage; use fixed-length, non-seekable S3 requests and bounded buffering when the upstream length is unknown.
• Add regression coverage for ACK latency, ordered media processing, timeout fallback, HTTP downloads, S3 streaming, message decoding, and attachment binding.

How to Test

1. Run make setup-worktree.
2. Run cd server && go test ./internal/integrations/channel/engine ./internal/integrations/lark ./internal/integrations/slack ./internal/storage -count=1.
3. Run cd server && go test -race ./internal/integrations/channel/engine ./internal/integrations/lark ./internal/storage -count=1.
4. Run cd server && go vet ./....
5. Send standalone and rich-text embedded images/videos through a bound Feishu/Lark bot and verify the resulting chat message contains attachments without delaying the connector ACK.

Checklist

• I have included a thinking path that traces from project context to this change
• I have run tests locally and they pass
• I have added or updated tests where applicable
• I have considered and documented risks above
• I will address all reviewer comments before requesting merge

Documentation and UI screenshots are not applicable because this changes the server-side inbound adapter without adding configuration or UI.

AI Disclosure

AI tool used: OpenAI Codex

Prompt / approach: Ported a proven Feishu/Lark media-ingestion fix onto the latest upstream main, traced the adapter-to-channel-to-attachment path, then revised it from maintainer feedback to keep remote media I/O off the ACK path and make non-seekable S3 uploads explicit and testable.

[vercel]

@beastpu is attempting to deploy a commit to the IndexLabs Team on Vercel.

A member of the Team first needs to authorize it.

[beastpu]

Addressed the retry-safety and test-coverage review in 88651c3a2:

• made media object keys deterministic per workspace, installation, message, resource type, and resource key, so an append failure/redelivery overwrites the same object instead of leaking a new upload
• added Router tests proving media resolution runs for valid messages and is skipped for revoked installations, duplicates, unaddressed group messages, unbound users, and non-members
• added graceful-degradation tests for download failure, upload failure, and partial rich-post failure
• added declared-size, unknown-length overflow/exact-boundary, and timeout-cancellation tests
• added database-backed tests for attachment persistence/linking and transaction rollback on link failure

Validation: make test, targeted repeated boundary tests, targeted race tests, and go vet ./... all pass locally.

[Bohan-J]

Thanks for this — the product direction is right (delivering inbound images/videos as real chat attachments, with a graceful placeholder fallback on failure), and the retry-safe object keying, the "resolve media only after routing/dedup/identity/session checks" ordering, the in-tx attachment binding, and the engine/http-layer test coverage are all solid. The whole-message media budget (MediaTimeout capping ResolveMedia) also nicely closes the earlier multi-attachment vs. dedup-claim-window concern.

Two blocking issues remain before merge, both around where/how the download happens.

1. Media resolution runs on the synchronous pre-ACK path and can breach the Lark ~3s ACK deadline

The inbound path is synchronous on the single connection receive goroutine: WSLongConnConnector reads the frame → Enricher.Enrich (bounded by EnrichTimeout, default 2s, explicitly documented to stay "well under Lark's ~3s ACK window") → emit → Router.Handle → dispatch → processClaimed. This PR inserts resolveMedia (up to MediaTimeout = 2.5s) synchronously between EnsureSession and AppendMessage, and the ACK frame (NewAckFrame(frame, true)) is only written after Handle returns.

So worst-case pre-ACK latency becomes enrich (≤2s) + media (≤2.5s) + identity/session/append DB time, which can exceed 3s. Even a standalone image with no enrichment is media 2.5s + DB, right at the edge. When the ACK is late, Lark redelivers the event; the 60s dedup guard keeps that from double-processing, but repeated late ACKs risk long-connection churn/reconnects — the exact thing the codebase avoids elsewhere by detaching the reply/typing calls.

Note the 2.5s default matches ReplyTimeout, but ReplyTimeout runs on a detached goroutine off the ACK path, whereas media runs on it — they can't share the same budget. Suggested direction: move media resolution off the pre-ACK path (e.g. append the message first so it's durable, then download/upload asynchronously and link the attachment to the persisted message before the debounced run flush), or bound it strictly within the ACK headroom left after enrichment + DB and reconcile it with EnrichTimeout.

2. S3Storage.UploadStream streams an unknown-length, non-seekable reader into PutObject — likely rejected by real S3/MinIO, and untested

UploadStream passes the download body straight into s3.PutObject as Body with no ContentLength set (the existing Upload uses a seekable, known-length bytes.NewReader). With aws-sdk-go-v2 v1.97's default RequestChecksumCalculation=WhenSupported, PutObject auto-selects CRC32 and takes the aws-chunked trailing-checksum path; for an unknown-length body it emits Transfer-Encoding: chunked without x-amz-decoded-content-length, which real S3 (and MinIO) require for aws-chunked uploads and will reject at the service.

Because media_ingest.go's uploadResource prefers the streaming path whenever the backend implements it (both local and S3 do), S3-backed deployments would hit this failing path — and since upload failure is non-fatal, media would silently never attach in production. The tests only exercise a fake storage (io.ReadAll), so S3Storage.UploadStream has no real-backend coverage and CI stays green.

The download side usually already knows the size (resp.ContentLength), but it isn't threaded into UploadStream. Fix options: set ContentLength on PutObjectInput (plumb the known size through), buffer into a bytes.Reader like the existing Upload, or use feature/s3/manager.Uploader. Please validate against a real S3/MinIO before merge.

Nits

• maxMessageResourceBytes is 100 MiB, but the ~2.5s synchronous window realistically only fits a few MiB of download+upload. Since the headline case is MP4 video, anything moderately large will hit the timeout and fall back to the [Video] placeholder. The 100 MiB cap only becomes meaningful once media moves off the ACK path (see #1).
• The PR description mentions a "20 MiB limit", but the code now uses 100 MiB — worth updating.
• looksLikeJSON in http_client.go is defined but unused (the actual detection is the inline strings.Contains(contentType, "json") check) — dead code, can be removed.