Repository navigation
Conversation
Recency::should_store caches each key's timeout in per_key_timeouts but never removes it when the metric expires. With labels that churn (e.g. pod_name from kubernetes_logs), the cache grows without bound even though the registry and recency map stay small.
|
Thank you for your contribution! Before we can merge this PR, please sign our Contributor License Agreement. To sign, copy and post the phrase below as a new comment on this PR.
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
More details
Timeout-cache entries are removed only after successful metric expiration, with the recency lock released before cache cleanup.
🤖 Bits Code Review · Commit ccb2de9
Summary
Fixes a memory leak in internal metrics when
expire_metrics_secsis set. Expired metrics were removed from the registry but stayed in a timeout cache forever, so memory grew with every new set of labels (for example, every new pod inkubernetes_logs). This removes the cache entry when the metric expires.References
Closes: #26906
Related: #17973, #17313
Vector configuration
Any config with
expire_metrics_secsset where labels keep changing, such askubernetes_logson nodes with many short-lived pods.How did you test this PR?
Added a unit test that creates and expires 10 rounds of 100 pod-labeled counters, and checks that the cache is empty after each round. Without the fix the cache grows by 100 entries every round (1000 after 10 rounds) and never shrinks; with it the cache stays empty. All
vector-coremetrics tests pass.I also ran Vector with a
filesource (include_file_tag: true,expire_metrics_secs: 5) while a script created and deleted 60,000 log files over 10 minutes:Does this PR include user facing changes?
no-changeloglabel to this PR.