chore(dev): add multiservice opentelemetry trace e2e test by tessneau · Pull Request #26554 · vectordotdev/vector

tessneau · 2026-10-06T12:38:13Z

Summary

Adds a manual OpenTelemetry trace test using the opentelemetry-demo app to Vector's e2e harness.

The existing opentelemetry-traces e2e test uses telemetrygen, which we could extend to cover more trace cases. However this app produced traces by multiple language SDKs and instrumentation libraries, so we get more varied and realistic input.

The main goal is to use this test against PRs against during the trace data model migration (see Trace data model RFC) so that we get the added confidence. This and the heavier application setup (24 Compose services, ~22 min in the earlier CI run) is why I've kept it manually triggered for now, since I'm not sure you'd want to run this continuously in CI.

Here's the test's architecture:

See the README for details.

References

Uses the shared harness cleanup from #26736 and metrics readiness fixes from #26778, both merged into master. This PR now contains only the multiservice trace test changes.

Related: Trace data model RFC.

Vector configuration

data/vector.yaml.

How did you test this PR?

The last full Demo CI run before the capture-finalization refactor passed on 6cf64848c0. Both HTTP and gRPC passed the round-trip validation and both comparator checks.

The job took 21m57s and used the former opt-in build cache. The test now uses the shared cleanup fix merged in #26736 to preserve compiled tests between transport runs while clearing test output.

The current head moves capture finalization into Compose, leaving Rust to validate completed files. Checked the helper locally with isolated Docker containers: it waits for delayed output, records drain timeouts, and leaves containers on other networks untouched. A small Compose run confirmed that detached startup waits for finalization. The full Demo run after the capture-finalization and shared-harness changes is still pending.

Before that refactor, also ran locally with:

cargo vdev e2e run opentelemetry-traces-multiservice --retries 0 --always-show-logs

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

Contributor Guidelines

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Before pushing, follow our pre-push guidance.
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.

Reuse the existing vdev Compose harness and PR review trigger. Compare captured span semantics and multiplicities, require realistic trace coverage, and retain evidence for migration PRs. Generated with GPT-6 (Codex).

tessneau

/ci-run-e2e-opentelemetry-traces-multiservice

thomasqueirozb

/ci-run-e2e-opentelemetry-traces-multiservice

tessneau

/ci-run-e2e-opentelemetry-traces-multiservice

chatgpt-codex-connector · 2026-10-08T16:56:24Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T16:07:41.049001Z	`1a9cdf2`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadoghq-integration

The quote readiness probe reports success for a refused connection, allowing premature workload startup and false E2E failures. The inline replacement was verified to fail on a closed port and succeed on an open port.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 69aeb3a · @DataDog review to ask questions}

tessneau · 2026-10-08T17:06:08Z

/ci-run-e2e-opentelemetry-traces-multiservice

datadoghq-integration

More details

Compose configuration validation passed. The multiservice round trip remains unverified in this environment because no Docker socket was available.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit d67df6e · @DataDog review to ask questions}

tessneau

/ci-run-e2e-opentelemetry-traces-multiservice

… states

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ee47be2bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadoghq-integration

More details

Static review identified no reportable defect in capture finalization or span preservation. Compose configuration validation passed, but the full multiservice HTTP/gRPC run remains unverified.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 1ee47be}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72bfca434d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pront

Assigning this PR to @thomasqueirozb since he recently worked on #24118

datadoghq-integration

Preserving compiled artifacts exposes a startup race in native OTLP metric validation: warm runs can read captures before generation and export finish, causing false failures.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 30092e4}

datadoghq-integration · 2026-10-21T15:23:30Z

            "sh",
            "-c",
-            &format!("rm -rf /output/{test_name}/*"),
+            &format!("rm -rf /output/{test_name}"),


Wait for native OTLP captures before reusing compiled tests

On warm runs of opentelemetry-metrics-native, preserving compiled artifacts removes the compilation delay after Compose starts. Compose only waits for services to start, but vector_native_metrics_encode_to_otlp immediately reads and validates the capture while generators and collector exports may still be running. Missing or partial output can therefore fail a healthy pipeline. This suite never calls the newly polling internal-metrics helper. Add a bounded capture-readiness gate before its assertions.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means

👍 Useful: correct, and you're fixing it or will.

👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.

😕 Unclear: you can't tell what it means or why it matters.

👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

_{🤖 Bits Code Review} _{· Open Bits AI session}

feels viable but I haven't run into any failure in a few local test runs, and I'd rather follow up to avoid this PR getting any bigger 😅 and depending on compile time felt already shaky

short-term we could copy the same bounded wait as #26778 does for the opentelemetry-logs and datadog-metrics tests, maybe there's a better way

Add manual OTel Demo trace E2E validation

f3ad538

Reuse the existing vdev Compose harness and PR review trigger. Compare captured span semantics and multiplicities, require realistic trace coverage, and retain evidence for migration PRs. Generated with GPT-6 (Codex).

github-actions Bot added the domain: ci Anything related to Vector's CI environment label Oct 6, 2026

tessneau added 2 commits October 7, 2026 15:09

Clarify OTel trace test documentation and add architecture diagram

f002f26

Rename OTel trace E2E suite to multiservice traces

6cb87b3

tessneau changed the title ~~Add manual OTel Demo trace E2E validation~~ Add manual multiservice OpenTelemetry trace E2E validation Oct 7, 2026

Use cart failure flag for multiservice trace errors

f0aa5ba

tessneau commented Oct 7, 2026

View reviewed changes

tessneau added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Oct 7, 2026

tessneau changed the title ~~Add manual multiservice OpenTelemetry trace E2E validation~~ chore(dev): add multiservice opentelemetry e2e test Oct 7, 2026

thomasqueirozb reviewed Oct 7, 2026

View reviewed changes

Allow more time for multiservice trace workload in CI

d3d0e8b

tessneau mentioned this pull request Oct 8, 2026

chore(ci): allow OP team members to trigger integration & e2e tests #26589

Merged

1 of 2 tasks

tessneau changed the title ~~chore(dev): add multiservice opentelemetry e2e test~~ test(opentelemetry): add manual multiservice trace E2E validation Oct 8, 2026

chore: merge master into multiservice trace tests

69aeb3a

tessneau commented Oct 8, 2026

View reviewed changes

tessneau marked this pull request as ready for review October 8, 2026 16:56

tessneau requested a review from a team as a code owner October 8, 2026 16:56

tessneau requested a review from bruceg October 8, 2026 16:56

tessneau marked this pull request as draft October 8, 2026 17:01

datadoghq-integration Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread tests/e2e/opentelemetry-traces-multiservice/config/compose.yaml Outdated

tessneau added 2 commits October 8, 2026 19:02

test(opentelemetry): shorten multiservice trace workload runtime

bbecc34

fix(tests): fail quote readiness check on refused connections

d67df6e

tessneau marked this pull request as ready for review October 8, 2026 17:07

tessneau changed the title ~~test(opentelemetry): add manual multiservice trace E2E validation~~ chore(dev): add multiservice opentelemetry trace e2e test Oct 8, 2026

datadoghq-integration Bot reviewed Oct 8, 2026

View reviewed changes

tessneau commented Oct 9, 2026

View reviewed changes

fix(tests): exit multiservice workload promptly and capture container…

300e901

… states

github-actions Bot added the domain:vdev Anything related to the vdev tooling label Oct 9, 2026

tessneau added 2 commits October 19, 2026 15:35

test(opentelemetry): clarify workload completion dependency

ee670b2

fix(vdev): preserve build artifacts during test cleanup

417d7cc

tessneau mentioned this pull request Oct 19, 2026

fix(vdev): preserve build artifacts during test cleanup #26736

Merged

1 of 2 tasks

tessneau added 3 commits October 19, 2026 19:11

fix(ci): build vdev from source for harness changes

e9ed31a

docs(tests): replace trace architecture image with mermaid

d4dc6bf

test(opentelemetry): use typed multiservice trace coverage

c12b304

bruceg approved these changes Oct 19, 2026

View reviewed changes

Comment thread .config/nextest.toml Outdated

Comment thread tests/e2e/opentelemetry-traces-multiservice/README.md Outdated

Comment thread tests/e2e/opentelemetry/multiservice_traces/mod.rs Outdated

tessneau added 7 commits October 20, 2026 14:32

fix(tests): wait for metrics before e2e validation

e40ab50

refactor(tests): construct trace coverage through its type

6a09618

docs(tests): clarify the multiservice trace diagram

acaf182

refactor(tests): finalize trace captures before validation

d00c5d8

fix(ci): build candidate vdev in merge queue and manual runs

49759a1

chore(dev): merge shared compose harness fixes

8750b01

refactor(tests): use shared compose harness fixes

1ee47be

tessneau changed the base branch from master to tess.neau/fix-compose-test-cleanup October 20, 2026 14:51

github-actions Bot removed the domain:vdev Anything related to the vdev tooling label Oct 20, 2026

chatgpt-codex-connector Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread tests/e2e/opentelemetry-traces-multiservice/data/finalize.py

datadoghq-integration Bot reviewed Oct 20, 2026

View reviewed changes

fix(tests): drain vector before freezing trace captures

72bfca4

chatgpt-codex-connector Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread Cargo.toml Outdated

Comment thread tests/e2e/opentelemetry-traces-multiservice/data/finalize.py

fix(tests): keep finalized trace environment active

30092e4

tessneau mentioned this pull request Oct 20, 2026

chore(dev): reuse pinned opentelemetry demo compose files #26750

Draft

1 of 2 tasks

pront reviewed Oct 20, 2026

View reviewed changes

pront requested a review from thomasqueirozb October 20, 2026 18:08

pront assigned thomasqueirozb Oct 20, 2026

Base automatically changed from tess.neau/fix-compose-test-cleanup to master October 21, 2026 15:08

tessneau mentioned this pull request Oct 21, 2026

chore(dev): fail demo cart failures fast in multiservice trace e2e #26782

Open

1 of 2 tasks

datadoghq-integration Bot reviewed Oct 21, 2026

View reviewed changes

tessneau added 2 commits October 21, 2026 17:43

chore(dev): merge master after shared e2e fixes

295b512

chore(dev): remove temporary trace e2e harness setup

1a9cdf2

Conversation

tessneau commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

References

Vector configuration

How did you test this PR?

Does this PR include user facing changes?

Contributor Guidelines

Uh oh!

tessneau left a comment

Choose a reason for hiding this comment

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

tessneau left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadoghq-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tessneau commented Oct 8, 2026

Uh oh!

datadoghq-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

tessneau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

datadoghq-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

pront left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

datadoghq-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadoghq-integration Bot Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

tessneau Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tessneau commented Oct 6, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 8, 2026 •

edited

Loading

pront left a comment •

edited

Loading

tessneau Oct 21, 2026 •

edited

Loading