Skip to content

feat(client): expose streaming/multipart upload (io.Reader) for PutObject#104

Merged
cjimti merged 1 commit into
mainfrom
feat/streaming-upload-103
Jun 1, 2026
Merged

feat(client): expose streaming/multipart upload (io.Reader) for PutObject#104
cjimti merged 1 commit into
mainfrom
feat/streaming-upload-103

Conversation

@cjimti

@cjimti cjimti commented Jun 1, 2026

Copy link
Copy Markdown
Member

Closes #103

Summary

Adds PutObjectStream, a streaming upload path on client.Client that uploads an object from an io.Reader using the AWS SDK transfer manager, without buffering the full payload in memory. The existing PutObject accepts Body []byte and does bytes.NewReader(input.Body) (pkg/client/client.go), so peak memory scaled with object size and large/unbounded sources were impractical.

This unblocks downstream callers that need to stream large or unbounded data (e.g. truly-streaming a query/API export → S3) at the pkg/client composition layer.

What changed

File Change
pkg/client/stream.go (new) PutObjectStream, PutObjectStreamInput, ErrStreamTooLarge, and a counting limitReader
pkg/client/s3api.go New consumer-side ObjectUploader interface (satisfied by *transfermanager.Client) + compile-time check
pkg/client/client.go uploader ObjectUploader field on Client; wired in New() from the same *s3.Client
pkg/client/mock_test.go mockUploader for the streaming path
pkg/client/stream_test.go (new) Success, validation, error-wrapping, MaxBytes under/over, and limitReader tests
README.md "Streaming uploads" section
go.mod / go.sum Adds feature/s3/transfermanager

API

out, err := s3Client.PutObjectStream(ctx, &client.PutObjectStreamInput{
    Bucket:      "my-bucket",
    Key:         "exports/large.csv",
    Body:        reader,            // any io.Reader
    ContentType: "text/csv",
    MaxBytes:    500 * 1024 * 1024, // optional: abort past this many bytes
})
if errors.Is(err, client.ErrStreamTooLarge) { /* stream exceeded the bound */ }

Dependency note (important)

The issue named feature/s3/manager, but that module is now deprecated and trips this repo's staticcheck (checks: [all] in .golangci.yml), which would fail make lint. Per maintainer direction, this PR uses the AWS-recommended successor feature/s3/transfermanager.

⚠️ transfermanager is currently a v0.2.x developer preview — its API may change across minor versions. This is the accepted tradeoff vs. a deprecated-but-stable dependency. The ObjectUploader interface isolates the SDK behind a single seam, so swapping implementations later is localized to s3api.go + client.New().

Design decisions

  • No per-operation timeout. Unlike the buffered methods, PutObjectStream does not apply S3_TIMEOUT — streaming a large object can legitimately run far longer than a normal request. Callers control the deadline via ctx.
  • Size limit at the library level. The size-limit MCP extension measures len() of a buffered body and can't apply to a length-unknown reader, so MaxBytes is enforced by a counting limitReader that aborts with ErrStreamTooLarge (inclusive bound).
  • Read-only consistency. Like the existing client.PutObject, the streaming method is a direct library call; the read-only/size-limit extensions guard the MCP tool layer, not direct library calls. PutObjectStream is intentionally library-only (no MCP tool) — matching the immediate downstream need.
  • Consumer-defined interface. ObjectUploader is defined at the consumer (pkg/client) so the streaming path is mockable, consistent with the existing S3API/PresignAPI pattern.

Acceptance criteria (from #103)

  • ✅ Large io.Reader body uploads via the transfer manager without full buffering; ETag/VersionID populated.
  • ✅ Stream exceeding MaxBytes is aborted (ErrStreamTooLarge).
  • ✅ Read-only behavior consistent with PutObject (extensions guard the tool layer).
  • ✅ Unit tests cover the new path; 100% coverage on new code, package at 97.5%.

Adversarial review

Verified the riskiest assumptions against the SDK source (not just green CI):

  • No data race on limitReader. transfermanager reads the body only from a single coordinating goroutine (nextReader); worker goroutines consume pre-filled bytes.NewReader chunks off a channel and never touch the body. The unsynchronized counter is safe.
  • errors.Is(err, ErrStreamTooLarge) survives SDK wrapping. Traced the full wrap chain (%wmultipartUploadError.Unwrap() → our wrap); the sentinel is reachable end-to-end.
  • First-read failures abort cleanly before any object is created (no orphaned multipart upload).

Known limitation: the real-SDK path is exercised via mocks (no live S3/SeaweedFS in unit tests), consistent with how the rest of the client package is tested.

Verification

make verify passes: golangci-lint + vet, go test -race, coverage (≥80%), gosec (0 issues), govulncheck (0 affecting), gocyclo (≤15), deadcode, build.

Adds PutObjectStream, which uploads an object from an io.Reader using the
AWS SDK transfer manager without buffering the full payload in memory. This
unblocks downstream callers that need to stream large or unbounded sources
(e.g. query exports) into S3, which the existing []byte-based PutObject
cannot do.

- ObjectUploader interface (consumer-side, mockable) satisfied by
  *transfermanager.Client; wired in client.New()
- PutObjectStreamInput with optional MaxBytes bound enforced by a counting
  limitReader that aborts with ErrStreamTooLarge
- Library-only capability; no per-operation timeout (caller controls ctx)
- Uses feature/s3/transfermanager (feature/s3/manager is deprecated and
  trips staticcheck SA1019)
- Tests: 100% coverage on new code; README documents usage

Closes #103
@codecov

codecov Bot commented Jun 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.03%. Comparing base (baef060) to head (e4baa07).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #104      +/-   ##
==========================================
+ Coverage   86.83%   87.03%   +0.20%     
==========================================
  Files          38       39       +1     
  Lines        2134     2167      +33     
==========================================
+ Hits         1853     1886      +33     
  Misses        173      173              
  Partials      108      108              
Files with missing lines Coverage Δ
pkg/client/client.go 95.49% <100.00%> (+0.04%) ⬆️
pkg/client/stream.go 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cjimti cjimti merged commit 4c809da into main Jun 1, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(client): expose streaming/multipart upload (io.Reader) for PutObject

1 participant