Skip to content

fix(metrics): free expired metrics from the timeout cache - #26907

Open
vinzee wants to merge 1 commit into
vectordotdev:masterfrom
vinzee:fix/metrics-recency-per-key-timeouts-leak
Open

vinzee wants to merge 1 commit into
vectordotdev:masterfrom
vinzee:fix/metrics-recency-per-key-timeouts-leak

Conversation

@vinzee

@vinzee vinzee commented Oct 27, 2026 •

Copy link
Copy Markdown

Summary

Fixes a memory leak in internal metrics when expire_metrics_secs is set. Expired metrics were removed from the registry but stayed in a timeout cache forever, so memory grew with every new set of labels (for example, every new pod in kubernetes_logs). This removes the cache entry when the metric expires.

References

Closes: #26906
Related: #17973, #17313

Vector configuration

Any config with expire_metrics_secs set where labels keep changing, such as kubernetes_logs on nodes with many short-lived pods.

How did you test this PR?

Added a unit test that creates and expires 10 rounds of 100 pod-labeled counters, and checks that the cache is empty after each round. Without the fix the cache grows by 100 entries every round (1000 after 10 rounds) and never shrinks; with it the cache stays empty. All vector-core metrics tests pass.

I also ran Vector with a file source (include_file_tag: true, expire_metrics_secs: 5) while a script created and deleted 60,000 log files over 10 minutes:

Memory growth during label churn (over 10 min) Memory growth after label churn
Vector 0.58.0 (without fix) keeps growing, to ~230 MiB stays at ~200 MiB
With fix stays at ~100 MiB drops to 69 MiB

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

Recency::should_store caches each key's timeout in per_key_timeouts but
never removes it when the metric expires. With labels that churn (e.g.
pod_name from kubernetes_logs), the cache grows without bound even
though the registry and recency map stay small.
@vinzee
vinzee requested a review from a team as a code owner October 27, 2026 17:54
@github-actions github-actions Bot added the domain:core Anything related to core crates i.e. vector-core, core-common, etc label Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Before we can merge this PR, please sign our Contributor License Agreement.

To sign, copy and post the phrase below as a new comment on this PR.

Note: If the bot says your username was not found, the email used in your git commit may not be linked to your GitHub account. Fix this at github.com/settings/emails, then comment recheck to retry.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T17:57:39.831596Z ccb2de9 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadoghq-integration datadoghq-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Timeout-cache entries are removed only after successful metric expiration, with the recency lock released before cache cleanup.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit ccb2de9

@vinzee vinzee changed the title fix(metrics): remove expired keys from the per-key timeout cache fix(metrics): free expired metrics from the timeout cache Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:core Anything related to core crates i.e. vector-core, core-common, etc

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak in internal metrics when expire_metrics_secs is set

1 participant