feat(client): expose streaming/multipart upload (io.Reader) for PutObject#104
Merged
Conversation
Adds PutObjectStream, which uploads an object from an io.Reader using the AWS SDK transfer manager without buffering the full payload in memory. This unblocks downstream callers that need to stream large or unbounded sources (e.g. query exports) into S3, which the existing []byte-based PutObject cannot do. - ObjectUploader interface (consumer-side, mockable) satisfied by *transfermanager.Client; wired in client.New() - PutObjectStreamInput with optional MaxBytes bound enforced by a counting limitReader that aborts with ErrStreamTooLarge - Library-only capability; no per-operation timeout (caller controls ctx) - Uses feature/s3/transfermanager (feature/s3/manager is deprecated and trips staticcheck SA1019) - Tests: 100% coverage on new code; README documents usage Closes #103
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #104 +/- ##
==========================================
+ Coverage 86.83% 87.03% +0.20%
==========================================
Files 38 39 +1
Lines 2134 2167 +33
==========================================
+ Hits 1853 1886 +33
Misses 173 173
Partials 108 108
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #103
Summary
Adds
PutObjectStream, a streaming upload path onclient.Clientthat uploads an object from anio.Readerusing the AWS SDK transfer manager, without buffering the full payload in memory. The existingPutObjectacceptsBody []byteand doesbytes.NewReader(input.Body)(pkg/client/client.go), so peak memory scaled with object size and large/unbounded sources were impractical.This unblocks downstream callers that need to stream large or unbounded data (e.g. truly-streaming a query/API export → S3) at the
pkg/clientcomposition layer.What changed
pkg/client/stream.go(new)PutObjectStream,PutObjectStreamInput,ErrStreamTooLarge, and a countinglimitReaderpkg/client/s3api.goObjectUploaderinterface (satisfied by*transfermanager.Client) + compile-time checkpkg/client/client.gouploader ObjectUploaderfield onClient; wired inNew()from the same*s3.Clientpkg/client/mock_test.gomockUploaderfor the streaming pathpkg/client/stream_test.go(new)MaxBytesunder/over, andlimitReadertestsREADME.mdgo.mod/go.sumfeature/s3/transfermanagerAPI
Dependency note (important)
The issue named
feature/s3/manager, but that module is now deprecated and trips this repo'sstaticcheck(checks: [all]in.golangci.yml), which would failmake lint. Per maintainer direction, this PR uses the AWS-recommended successorfeature/s3/transfermanager.transfermanageris currently a v0.2.x developer preview — its API may change across minor versions. This is the accepted tradeoff vs. a deprecated-but-stable dependency. TheObjectUploaderinterface isolates the SDK behind a single seam, so swapping implementations later is localized tos3api.go+client.New().Design decisions
PutObjectStreamdoes not applyS3_TIMEOUT— streaming a large object can legitimately run far longer than a normal request. Callers control the deadline viactx.len()of a buffered body and can't apply to a length-unknown reader, soMaxBytesis enforced by a countinglimitReaderthat aborts withErrStreamTooLarge(inclusive bound).client.PutObject, the streaming method is a direct library call; the read-only/size-limit extensions guard the MCP tool layer, not direct library calls.PutObjectStreamis intentionally library-only (no MCP tool) — matching the immediate downstream need.ObjectUploaderis defined at the consumer (pkg/client) so the streaming path is mockable, consistent with the existingS3API/PresignAPIpattern.Acceptance criteria (from #103)
io.Readerbody uploads via the transfer manager without full buffering;ETag/VersionIDpopulated.MaxBytesis aborted (ErrStreamTooLarge).PutObject(extensions guard the tool layer).Adversarial review
Verified the riskiest assumptions against the SDK source (not just green CI):
limitReader.transfermanagerreads the body only from a single coordinating goroutine (nextReader); worker goroutines consume pre-filledbytes.NewReaderchunks off a channel and never touch the body. The unsynchronized counter is safe.errors.Is(err, ErrStreamTooLarge)survives SDK wrapping. Traced the full wrap chain (%w→multipartUploadError.Unwrap()→ our wrap); the sentinel is reachable end-to-end.Known limitation: the real-SDK path is exercised via mocks (no live S3/SeaweedFS in unit tests), consistent with how the rest of the
clientpackage is tested.Verification
make verifypasses: golangci-lint + vet,go test -race, coverage (≥80%), gosec (0 issues), govulncheck (0 affecting), gocyclo (≤15), deadcode, build.