out_s3: implement retry_limit parameter #10825
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
retry_limit parameter is not honored and is set to 5. This feature provides dynamic retry_limit based on configuration for out_s3 plugin. Signed-off-by: usharma <[email protected]>
WalkthroughReplaces the global MAX_UPLOAD_ERRORS with a per-instance Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor FLB as S3 Output Instance
participant CH as Chunk / Multipart Upload
participant S3 as AWS S3
participant INS as Instance (retry_limit)
FLB->>CH: initiate upload/flush/part
CH->>S3: send request
S3-->>CH: success / error
alt Success
CH-->>FLB: mark complete
else Error
CH->>INS: increment failure counter
INS->>INS: compare failures >= retry_limit
alt failures < retry_limit
INS-->>CH: schedule retry
else failures >= retry_limit
INS-->>CH: inactivate/discard upload
CH-->>FLB: log exceeded retry_limit
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 💡 Knowledge Base configuration:
You can enable these sources in your CodeRabbit configuration. 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
plugins/out_s3/s3.c (3)
1873-1881: Gate retry logic by FLB_OUT_RETRY_UNLIMITED
Wrap the existing retry‐limit check soFLB_OUT_RETRY_UNLIMITEDdoesn’t prematurely stop retries (else any non-negative counter ≥ –1 is always true):--- a/plugins/out_s3/s3.c +++ b/plugins/out_s3/s3.c @@ -1873,7 +1873,9 @@ upload_contents->retry_counter++; - if (upload_contents->retry_counter >= ctx->ins->retry_limit) { + if (ctx->ins->retry_limit != FLB_OUT_RETRY_UNLIMITED && + upload_contents->retry_counter >= ctx->ins->retry_limit) { flb_plg_warn(ctx->ins, "Chunk file failed to send %d times, will not " "retry", upload_contents->retry_counter); s3_store_file_inactive(ctx, upload_contents->upload_file);This change aligns with other plugins’ unlimited-retry gating and restores true unlimited behavior.
3291-3296: Gatecomplete_errorscheck byFLB_OUT_RETRY_UNLIMITED
Wrap the existing threshold test so it’s only applied whenretry_limitisn’t unlimited:- if (m_upload->complete_errors >= ctx->ins->retry_limit) { + if (ctx->ins->retry_limit != FLB_OUT_RETRY_UNLIMITED && + m_upload->complete_errors >= ctx->ins->retry_limit) {
1628-1633: Gate multipart upload error threshold whenretry_limitis finite
Modify inplugins/out_s3/s3.c:1628–1633as follows:- if (tmp_upload->upload_errors >= ctx->ins->retry_limit) { + if (ctx->ins->retry_limit != FLB_OUT_RETRY_UNLIMITED && + tmp_upload->upload_errors >= ctx->ins->retry_limit) {This skips the error threshold when
retry_limitisFLB_OUT_RETRY_UNLIMITED(-1), avoiding immediate completion for unlimited retries. (fossies.org)
🧹 Nitpick comments (1)
plugins/out_s3/s3.h (1)
59-61: Clarify that retry_limit comes from the output instance (Retry_Limit).To avoid confusion with plugin-local fields, consider wording this note to reference the generic output Retry_Limit (ins->retry_limit) that Fluent Bit core already provides, since that’s what the code uses.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
plugins/out_s3/s3.c(7 hunks)plugins/out_s3/s3.h(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/out_s3/s3.c (1)
src/flb_fstore.c (1)
flb_fstore_file_inactive(218-235)
🔇 Additional comments (1)
plugins/out_s3/s3.h (1)
100-103: Comment tweak looks fine.
Fixed the retry_limit as it was not needed and it part of default instance Signed-off-by: usharma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
retry_limit parameter is not honored and is set to 5. This feature provides dynamic retry_limit based on configuration for out_s3 plugin.
This feature fixes the infinite retry when buffers are cleared and Bucket name does not exist. It also implements retry_limit when users provide it on a out_s3 plugin level.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
flb.log
This run is for the scenario where we do not push the records.
valgrind_report.txt
This Report is justified as we dont want to delete the file and use
flb_fstore_file_inactive(ctx->fs, fsf)instead offlb_fstore_destroy(ctx->fs)It correlates with the current Master version :
valgrind_mem_check_4.0.8.txt
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Additional Tests Run with Logs :
[Buffers Cleared Everytime]
BUCKET DOES NOT EXIST
[Buffers Cleared Everytime]
BUCKET EXISTS
Summary by CodeRabbit
New Features
Documentation
fixes: #10819