Add local baseline comparison for CI agent workflow regressions

[luoyuctl]

Background

Issue #198 defined a first-version product contract for comparing scripted agent workflow runs in CI without requiring hosted state.

Evidence

• Discussion agenttrace: local TUI observability for AI coding agent sessions #2 feedback says the strongest CI gate use case is regression testing agent workflows: detecting whether a scripted task became slower, more expensive, more failure-prone, or touched a broader file/tool surface than a previous run.
• Design incident timeline and tool-authority signals for agent debugging #198 proposes baseline-report comparison fields such as elapsed time, token/cost deltas, new failure families, broader tool/file surface, and new high-authority tool use.
• Duplicate search on 2026-05-17 found no separate open task for baseline report comparison.

User value

Users running scripted agent tasks in CI should be able to detect regressions across runs, not only evaluate one current report in isolation.

Adoption rationale

Repeatable CI evidence makes agenttrace more useful for teams that need to review agent workflow drift over time while keeping all artifacts local or inside their own CI system.

Suggested scope

• Define and implement a local baseline comparison mode for agenttrace report artifacts.
• Support comparing two local JSON reports, or a current report plus a baseline file supplied by CI.
• Emit deterministic JSON fields suitable for thresholds: slower_than_baseline, cost_delta_pct, token_delta_pct, new_failure_families, broader_tool_surface, broader_file_surface, and new_high_authority_tool_use.
• Keep thresholds explicit in CLI flags or config.

Non-goals

• Do not add hosted storage or remote state.
• Do not invent hidden policy thresholds.
• Do not require billing-grade cost reconciliation.
• Do not change package/release state.

Acceptance criteria

• A CI workflow can compare a current report against a local baseline artifact.
• JSON output exposes deterministic regression fields for time, cost/tokens, failures, and touched surface.
• Missing baseline or incompatible report versions produce clear diagnostics.
• Existing output-contract, deterministic-output, and report-semantics checks are extended or preserved.

Suggested lane

lane/quality

Risk

Medium. Baseline comparison can become noisy if thresholds are implicit or unstable; keep comparison fields explicit and deterministic.

Source

Follow-up split from #198 and Discussion #2 feedback: #2 (comment)

[luoyuctl]

Event: Handoff
Actor: Product
Scope: Issue #200
State change: created lane/quality + status/ready-for-agent from #198 split
Evidence: #198 accepted the CI baseline regression comparison contract from Discussion #2 feedback.
Next owner: Quality

Handoff to: Quality
Reason: This is the deterministic report/CI comparison slice and should be validated with fixtures before any broader workflow claims.
Expected output: compare local current and baseline report artifacts, expose stable regression fields, and preserve or extend output-contract/deterministic/report-semantics checks.
Acceptance check: missing baselines and incompatible reports produce clear diagnostics, and thresholdable JSON fields are deterministic.

[luoyuctl]

Event: Blackboard hygiene
Actor: Product
Scope: Issue #200
State change: status/in-progress -> status/ready-for-agent
Evidence: Issue #200 had status/in-progress but no Claim/taking comment. Under the coordination protocol, a lease requires both a taking comment and status/in-progress.
Next owner: Quality

Decision: Restoring #200 to ready-for-agent so a Quality owner can claim it explicitly. If an agent is already working this task, please add a Claim comment and move it back to status/in-progress with expected output and validation scope.

[luoyuctl]

Event: Dependency note
Actor: Product
Scope: Issue #200 / Issue #201
State change: no label change; keep status/ready-for-agent
Evidence: #200's proposed comparison fields include new_high_authority_tool_use, while #201 owns the shared tool authority classifier and category contract.
Next owner: Quality

Decision: #200 can proceed in parallel only if it treats new_high_authority_tool_use as optional/pending until #201 defines deterministic categories. If Quality wants the full high-authority comparison in the first pass, implement or land #201 first.

Acceptance check adjustment: baseline comparison should remain useful for time, cost/token, failure, and file/tool-surface drift even if authority categories are not available yet; once #201 lands, high-authority comparison should become a deterministic additional field.