Skip to content

feat(validation): add repeatable runs with process telemetry - #123

Merged
morluto merged 7 commits into
mainfrom
feature/repeatable-validation-telemetry
Jul 23, 2026
Merged

feat(validation): add repeatable runs with process telemetry#123
morluto merged 7 commits into
mainfrom
feature/repeatable-validation-telemetry

Conversation

@morluto

@morluto morluto commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Description

A single validation run cannot distinguish a stable result from a flaky one, and the existing runner records only exit/output timing. It cannot answer whether repeated attempts agreed, where a timeout occurred, how much process-tree resource was used, or whether descendants survived shutdown.

This adds bounded validation run groups with explicit attempt count, concurrency, per-run and overall timeouts, output limits, sampler interval, and MCP readiness timeout. Every attempt remains an ordinary persisted validation run; the group adds aggregate classification, pass/fail/error/cancel counts, timing distribution, peak resource metrics, and ordered attempt references.

Process telemetry uses maintained gopsutil APIs and binds samples to PID plus creation time to avoid PID-reuse errors. Unsupported metrics are represented as unavailable with a reason, never fabricated as zero. Process-group cancellation and a bounded shutdown check report surviving descendants separately from semantic test classification.

Declared mcp_stdio validations use the official MCP Go SDK for initialize and tools/list milestones. Protocol readiness is not inferred from arbitrary stdout, and readiness/shutdown timeouts abort the owned transport without an unbounded wait.

CLI and MCP expose explicit repeat operations; defaults are applied at runtime so omitted schema fields behave consistently across clients.

Closes #115.

Suggested review order

  1. internal/evidence/repeat.go for bounded orchestration and aggregation.
  2. internal/evidence/telemetry.go and runner changes for process identity, metrics, and cleanup.
  3. internal/evidence/mcp_runner.go for SDK-backed protocol milestones and shutdown.
  4. Persistence, application/CLI/MCP adapters, tests, and documentation.

Testing

  • make verify
  • make test-race
  • Windows cross-compilation for evidence, application, CLI, and MCP packages
  • Regression coverage for repeat ordering, cancellation, timeout phases, partial metric availability, descendant cleanup, runtime defaults, and repeated MCP readiness/shutdown behavior

Compatibility and scope

  • Storage and side-effect invariants remain intact
  • No unrelated cleanup or generated-output churn is included

Existing single-run behavior remains available. Existing corpora require the explicit backed-up migration flow. This branch, #118, and #122 currently occupy the next Goose migration slot; whichever merges later must rebase and renumber its migration before merge.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@morluto
morluto force-pushed the feature/repeatable-validation-telemetry branch from d7cd7cd to 07a483f Compare July 23, 2026 06:49
@morluto
morluto force-pushed the feature/repeatable-validation-telemetry branch from 07a483f to 0a7cbf7 Compare July 23, 2026 06:51
@morluto
morluto merged commit 8db0673 into main Jul 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add repeatable process validation with resource telemetry

1 participant