Skip to content

fix(vdev): ensure integration test containers despawn - #26838

Open
Captain1653 wants to merge 3 commits into
vectordotdev:masterfrom
Captain1653:13017-clean-up-integration-test
Open

Captain1653 wants to merge 3 commits into
vectordotdev:masterfrom
Captain1653:13017-clean-up-integration-test

Conversation

@Captain1653

@Captain1653 Captain1653 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Removed conditional cleanup from the Makefile: previously, container cleanup logic relied on Makefile conditions, which didn't consistently trigger when integration tests failed.
  • Implemented an RAII guard: cleanup is now guaranteed at the code level via the Drop trait and ComposeTest.stop(), reliably tearing down test containers even on test failures or panics.
  • Preserved existing contract: retains full backward compatibility with AUTODESPAWN and AUTOSPAWN environment variables (defaults to true). This implementation reads was_running flag and honors it.
  • Maintained manual cleanup target: kept the explicit Makefile target test-integration-%-cleanup intact for debugging workflows when autodespawn is disabled.

Although the original issue was reported specifically for AWS integration tests, this change fixes the cleanup behavior at the core runner level (ComposeTest). Consequently, all integration test suites now benefit from reliable automatic teardown on failure.

References

Closes: #13017

Vector configuration

Not Applicable

How did you test this PR?

  1. Reproduction on master:

    • Intentionally introduced a test failure in amqp integration tests.
    • Ran make test-integration-amqp.
    • Result: Tests failed as expected, but the containers (rabbitmq, test runner) remained running in docker ps.
  2. Verification on this branch:

    • Cleaned up the leftover containers and ran the same failing test suite on this branch.
    • Result: The RAII guard executed on exit/unwind, tearing down the containers automatically. I can see in logs:
      [+] down 1/1
       ✔ Container vector-integration-amqp-3-8-rabbitmq-1 Removed
      
    • Verified container removal in docker ps - no containers.

Implementation Notes & Considerations

Bumped vdev patch version: Bumped the patch version of vdev directly in this PR, as this change fixes a defect in the integration test orchestration workflow.
AUTOSPAWN variable usage: Noticed that AUTOSPAWN was defined in the Makefile, but never actually referenced anywhere else in the codebase (only in MD-files) :

  git grep -rn "AUTOSPAWN" master

Is this expected legacy behavior or an intentional flag meant to be supported alongside AUTODESPAWN?

Direct call to domain ComposeTest::stop(): The RAII guard invokes self.test.stop() directly rather than re-exporting or routing through the CLI command handler (commands::compose_tests::stop). Calling the high-level CLI command would require changing visibility for internal CLI handlers, whereas ComposeTest::stop() cleanly handles container teardown at the domain level.

Preserved AUTODESPAWN in Makefile: Left the environment variable definitions in the Makefile intact to ensure zero impact on existing CI workflows, scripts, or external developer habits.

Retained manual cleanup Makefile target: Kept the explicit clean test-integration-%-cleanup target in the Makefile. If a developer explicitly runs with AUTODESPAWN=false to inspect containers after a failure, they still need a convenient way to tear down the environment afterwards.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

Contributor Guidelines

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Before pushing, follow our pre-push guidance.
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.

@github-actions github-actions Bot added the domain:vdev Anything related to the vdev tooling label Oct 24, 2026
pub fn new(test: &'a ComposeTest) -> Self {
let auto_despawn = std::env::var("AUTODESPAWN")
.or_else(|_| std::env::var("AUTOSPAWN"))
.map_or(true, |v| v.eq_ignore_ascii_case("true"));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the env parsing straightforward. If the team prefers stricter parsing (e.g., explicitly matching "1" / "0", or treating invalid strings as errors), I'm happy to adjust it to project conventions.

@Captain1653
Captain1653 marked this pull request as ready for review October 24, 2026 09:37
@Captain1653
Captain1653 requested a review from a team as a code owner October 24, 2026 09:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T10:51:42.477731Z 497e985 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadoghq-integration datadoghq-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The cleanup guard removes caller-owned environments, breaking repeated local testing and deleting containers before CI collects failure diagnostics. Restricting the guard to environments started by the test invocation preserves both workflows.

Open Bits AI session

🤖 Bits Code Review · Commit 2aa43a9

Comment thread vdev/src/testing/integration.rs Outdated
@Captain1653 Captain1653 changed the title fix(vdev): ensure integration test containers despawn on drop fix(vdev): ensure integration test containers despawn Oct 24, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a50168aa6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread vdev/src/testing/integration.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 497e985167

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

impl Drop for DespawnGuard<'_> {
fn drop(&mut self) {
if self.should_despawn {
let _ = self.test.stop();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clean up compose projects even after every service exits

When a failing integration loses all of its compose services, is_running() returns false because it queries only --status running; this Drop path then calls ComposeTest::stop(), which bails before both compose.stop() and runner.remove(), and the error is discarded here. Consequently, service-crash failures still leave the exited compose project and the live test-runner container behind, so the guard's cleanup path needs to handle an existing stopped project rather than requiring a running service.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch regarding crashed services and is_running(), but that is pre-existing behavior of ComposeTest::stop(). Modifying how stop() handles non-running environments affects other workflows and is outside the scope of this PR.

Let's wait for the core team's review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:vdev Anything related to the vdev tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration test containers not being removed after test failure.

1 participant