feat(stable_config): add >100mb check for stable config files #1432
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksComparisonBenchmark execution time: 2026-01-07 13:28:18 Comparing candidate commit 23c5263 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 2 unstable metrics. CandidateCandidate benchmark detailsGroup 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Group 9
Group 10
Group 11
Group 12
Group 13
Group 14
Group 15
Group 16
Group 17
Group 18
Group 19
BaselineOmitted due to size. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1432 +/- ##
==========================================
+ Coverage 71.23% 71.25% +0.01%
==========================================
Files 411 411
Lines 65788 65863 +75
==========================================
+ Hits 46863 46929 +66
- Misses 18925 18934 +9
🚀 New features to boost your workflow:
|
vpellan
force-pushed
the
vpellan/stable-config-size-limit
branch
from
b226e19 to
23c5263
Compare
anna-git
reviewed
Jan 7, 2026
| if metadata.len() > 1024 * 1024 * 100 { | ||
| let anyhow_error = anyhow::anyhow!("Local file is too large (> 100mb)"); | ||
| let logged_result = LoggedResult::Err(anyhow_error); | ||
| return logged_result; |
Contributor
There was a problem hiding this comment.
is it by design that if one of them is too large, we bail out of stable config completely?
NIT: also was wondering if this check could be in a separate function which both local and fleet could reuse but not a big deal
There was a problem hiding this comment.
if one of them is too large, we bail out of stable config completely
If I remember correctly, we discussed this limitation a few months ago. For libraries that do not embed libdatadog for stable config -- like java and go -- we can attempt to open each file individually, and handle errors separately. It's my understanding that, in the libdatadog architecture, if either file is too big, we return a single error, and when the library consumes this error, it bails. Hence the behavior you described above. Correct?
And to my memory, we said there isn't a clean way to separate the error handling right now without breaking APIs, yes?
I really don't like this solution, since both inputs will have very different user experiences and thus should be logically separated, but I also acknowledge if it's the best we can do right now. In this case, can we add a comment in here that references a jira card that describes the issue behavior as well as how we might fix it in the future?
Contributor
There was a problem hiding this comment.
I think it wouldn't be a problem to return a config without the keys from the faulty file, and just add the validation error to the messages of LoggedResult. We could extend LoggedResult to contain error_messages and add the >100 mb error message in this vector. The tracers should read this vector always, and if they see it's filled up, log it as an error, so that it appears in error tracking as well. We just fill the error_messages, dont interrupt the function and just return an Ok() with whichever configs were read.
If no file could be read at all(very rare...), I guess we could reallyl return an Error..
| }, | ||
| ); | ||
| match result { | ||
| LoggedResult::Ok(..) => panic!("Expected error"), |
Contributor
There was a problem hiding this comment.
Suggested change
| LoggedResult::Ok(..) => panic!("Expected error"), | |
| LoggedResult::Ok(..) => panic!("Expected error for fleet file > 100mb, but got success instead"), |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds a check for stable config files >100mb and returns an error if it goes above that limit.
Motivation
Enforcing a limit at all provides value by protecting against extreme or clearly invalid file inputs
Additional Notes
The intentionally high limit is designed to minimize the risk of customers inadvertently hitting the cap and losing configuration silently. Under normal conditions, configuration files should remain well below this threshold
How to test the change?
Tests have been added