feat(validation): add repeatable runs with process telemetry - #123
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
morluto
force-pushed
the
feature/repeatable-validation-telemetry
branch
from
July 23, 2026 06:49
d7cd7cd to
07a483f
Compare
morluto
force-pushed
the
feature/repeatable-validation-telemetry
branch
from
July 23, 2026 06:51
07a483f to
0a7cbf7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A single validation run cannot distinguish a stable result from a flaky one, and the existing runner records only exit/output timing. It cannot answer whether repeated attempts agreed, where a timeout occurred, how much process-tree resource was used, or whether descendants survived shutdown.
This adds bounded validation run groups with explicit attempt count, concurrency, per-run and overall timeouts, output limits, sampler interval, and MCP readiness timeout. Every attempt remains an ordinary persisted validation run; the group adds aggregate classification, pass/fail/error/cancel counts, timing distribution, peak resource metrics, and ordered attempt references.
Process telemetry uses maintained
gopsutilAPIs and binds samples to PID plus creation time to avoid PID-reuse errors. Unsupported metrics are represented as unavailable with a reason, never fabricated as zero. Process-group cancellation and a bounded shutdown check report surviving descendants separately from semantic test classification.Declared
mcp_stdiovalidations use the official MCP Go SDK for initialize andtools/listmilestones. Protocol readiness is not inferred from arbitrary stdout, and readiness/shutdown timeouts abort the owned transport without an unbounded wait.CLI and MCP expose explicit repeat operations; defaults are applied at runtime so omitted schema fields behave consistently across clients.
Closes #115.
Suggested review order
internal/evidence/repeat.gofor bounded orchestration and aggregation.internal/evidence/telemetry.goand runner changes for process identity, metrics, and cleanup.internal/evidence/mcp_runner.gofor SDK-backed protocol milestones and shutdown.Testing
make verifymake test-raceCompatibility and scope
Existing single-run behavior remains available. Existing corpora require the explicit backed-up migration flow. This branch, #118, and #122 currently occupy the next Goose migration slot; whichever merges later must rebase and renumber its migration before merge.