Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce s3 client manager and s3 uploaders #458

Merged
merged 2 commits into from
Dec 4, 2024

Conversation

jfzunigac
Copy link
Contributor

This PR enhances the S3writer in a few ways:

  1. The uploading mechanism is now pluggable, now uploaders have to extend S3Uploader class. This change simplifies the implementation of uploads to S3 using alternative APIs, such as MultiPartUpload or TransferManager. This ensures backwards compatibility so newer implementations are easier to roll out. New config is added to reference the uploader class: optional string uploaderClass = "com.pinterest.singer.writer.s3.PutObjectUploader"
  2. Add support for uploading to buckets in different regions. The region needs to be specified when creating a new S3Client, this means one Singer instance can't upload to multiple buckets in different regions. To enable this, we introduce a new S3ClientManager which will create a new client per region. New S3Writer config for region: optional string region = "us-east-1". Additionally, we remove the S3Client.close() call in S3Writer since clients are shared resources.
  3. Use S3ObjectUpload to encapsulate an upload. Useful if metadata needs to be attached to each upload in the future
  4. Rename ObjectUploaderTask -> PutObjectUpolader to match pluggable uploader class. Functionality remains the same

@jfzunigac jfzunigac requested a review from a team as a code owner December 4, 2024 00:43
@jfzunigac jfzunigac changed the title introduce s3 client manager and s3 uploaders Introduce s3 client manager and s3 uploaders Dec 4, 2024
@jfzunigac jfzunigac force-pushed the s3writer/s3_uploader_manager branch from c4eaeac to 4f02509 Compare December 4, 2024 00:46
@jfzunigac jfzunigac force-pushed the s3writer/s3_uploader_manager branch from 4f02509 to 05999bf Compare December 4, 2024 00:53
@jfzunigac jfzunigac merged commit 8b6f32d into pinterest:master Dec 4, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants