S3 writer #434
Merged
S3 writer #434
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vahidhashemian
requested changes
Aug 2, 2024
singer/src/main/java/com/pinterest/singer/writer/s3/ObjectUploaderTask.java
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/S3Writer.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/S3Writer.java
Outdated
Show resolved
Hide resolved
jeffxiang
previously approved these changes
Aug 2, 2024
singer/src/main/java/com/pinterest/singer/utils/LogConfigUtils.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/ObjectUploaderTask.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/ObjectUploaderTask.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/ObjectUploaderTask.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/ObjectUploaderTask.java
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/ObjectUploaderTask.java
Outdated
Show resolved
Hide resolved
jfzunigac
reviewed
Aug 3, 2024
singer/src/main/java/com/pinterest/singer/writer/s3/S3Writer.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/S3Writer.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/S3Writer.java
Outdated
Show resolved
Hide resolved
singer/src/main/java/com/pinterest/singer/writer/s3/S3Writer.java
Outdated
Show resolved
Hide resolved
vahidhashemian
approved these changes
Aug 7, 2024
| config.setMinUploadTimeInSeconds(MIN_UPLOAD_TIME_IN_SECONDS); | ||
| } | ||
|
|
||
| // maxRetries = 5 if inputted value is less than 0, else inputted value |
There was a problem hiding this comment.
nit: inputted -> input
There was a problem hiding this comment.
agreed, thanks Vahid!
| The current design employs two thresholds, a timer-based (in seconds) and file-sized threshold (in megabytes). The writer will upload the log file to S3 when either of the thresholds is met first. | ||
| For example, if 50MB of log messages is reached before 5 seconds, the writer will upload the log file to S3 because the file size threshold has been exceeded. If 5 seconds is reached before 50MB, the writer will upload the log file to S3 because the timer threshold has expired. | ||
|
|
||
| ## Example writer configuration in Singer Log Configuration |
There was a problem hiding this comment.
nit: Is "Singer Log Configuration" more accurate or "Singer Logstream Configuration"?
There was a problem hiding this comment.
Ah good catch Vahid, singer Log Stream Configuration is more descriptive. Thank you!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feature for Singer ability to write to S3
Description
Giving Singer the ability to write log messages directly to S3 with the below configurations. Singer should expose a new set of configurations that provide users the ability to send their logs to S3 through Singer. There are two thresholds that trigger the upload to S3: time-based uploads (every X seconds) and file size limits (once we have seen X megabytes of log files) whichever occurs first.
Configuration for S3 writer
writer.type=s3
writer.s3.maxFileSizeMB=50 ← max(50, maxFileSizeMB), default 50
writer.s3.minUploadTimeInSeconds=5 ← max(30, minUploadTimeInSeconds), default 30
writer.s3.fileNameFormat=myFileNameFormat ← suffix right before a timestamp required
writer.s3.bucket=myBucket ← AWS bucket name required
writer.s3.keyPrefix=myKeyPrefix ← AWS keyPrefix name required
writer.s3.maxRetries=3 ← retry logic (if uploads unsuccessful), default 5
writer.s3.bufferDir="tmp/singer/s3" ← directory for buffered files that will be uploaded to s3, default"tmp/singer/s3"