Skip to content

fix(prometheus_exporter sink): close listener before the sink finishes - #26831

Open
thomasqueirozb wants to merge 3 commits into
masterfrom
prometheus-exporter-release-listener
Open

thomasqueirozb wants to merge 3 commits into
masterfrom
prometheus-exporter-release-listener

Conversation

@thomasqueirozb

@thomasqueirozb thomasqueirozb commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Summary

The prometheus_exporter sink runs its HTTP server in a detached task, and run() returned before that task closed the listener. During a reload, shutdown_diff waits for a sink with a conflicting resource to finish and then treats that resource as free. For this sink, only task scheduling made the address free in time. The sink now waits for its server to stop before it finishes.

No user-visible failure was reproduced (see below), so this PR has no changelog fragment. #26832, which removes the Windows reload sleep, depends on this change.

References

Related: #3035, #26832

Vector configuration

healthchecks:
  enabled: false
data_dir: /tmp/vector-reload-repro
sources:
  in:
    type: internal_metrics
    scrape_interval_secs: 1
sinks:
  out:
    type: prometheus_exporter
    inputs: [in]
    address: 127.0.0.1:19598
    flush_period_secs: 61 # changed on every reload

How did you test this PR?

  • Vector 0.60.0 (Homebrew, macOS): 100 SIGHUP reloads in each of 3 setups, with flush_period_secs changed before each reload. The setups were default threads with health checks on, --threads 1 with health checks on, and --threads 1 with the config above. All 300 reloads succeeded, with no AddrInUse or Failed to start Prometheus exporter errors. The race does not show up in practice, because the old server task runs before the new sink binds.
  • New unit test finished_sink_releases_address runs the sink to completion and then binds its address. On origin/master it fails every time with AddrInUse (4/4 tries); with this change it passes.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

@github-actions github-actions Bot added the domain:sinks Anything related to Vector's sinks label Oct 23, 2026
@thomasqueirozb
thomasqueirozb added this pull request to stack #26833 October 23, 2026 20:30
@thomasqueirozb thomasqueirozb added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Oct 23, 2026
@thomasqueirozb
thomasqueirozb marked this pull request as ready for review October 23, 2026 20:40
@thomasqueirozb
thomasqueirozb requested a review from a team as a code owner October 23, 2026 20:40
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T21:26:18.921502Z c1cd0ad New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c99a14d4f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/prometheus/exporter.rs Outdated
Comment thread src/sinks/prometheus/exporter.rs Outdated

@datadoghq-integration datadoghq-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The Prometheus exporter's graceful server shutdown await in exporter.rs has no timeout, meaning a stalled scrape client or an incomplete TLS handshake can block RunningTopology::shutdown_diff indefinitely during a configuration reload. The fix requires bounding the drain wait and aborting/joining the server on expiry, or decoupling listener-close completion from connection draining.

Open Bits AI session

🤖 Bits Code Review · Commit 0c99a14

Comment thread src/sinks/prometheus/exporter.rs Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:sinks Anything related to Vector's sinks no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants