Skip to content

Measurement specs write to a pre-archive change dir, recreating a phantom active change #549

Description

@robertcsakany

Symptom

After verify-subagent-pull-under-load was archived (#548), re-running its measurement specs recreated openspec/changes/verify-subagent-pull-under-load/ containing only a partial measurements.json. That directory looks like an active OpenSpec change to tooling that enumerates openspec/changes/*, but it is a leftover artifact of an archived one.

Cause

Both specs hardcode the pre-archive path:

  • tests/e2e/subagent-pull-measurements.spec.ts:46
  • tests/e2e/subagent-pull-under-load.spec.ts:68
const EVIDENCE_PATH = "openspec/changes/verify-subagent-pull-under-load/measurements.json";

record() calls mkdirSync(dirname(EVIDENCE_PATH), { recursive: true }), so the directory is created rather than failing loudly once the change moves to openspec/changes/archive/<date>-<name>/.

Consequences

  1. A phantom active-change directory reappears on every re-measure.
  2. Worse for evidence integrity: the write silently goes somewhere other than the archived heap-evidence.md's source of truth, so a re-run appears to leave the archived numbers "unchanged" while actually recording different ones elsewhere. That happened during test(verify-subagent-pull-under-load): verify the four scenarios the payload change shipped unverified #548 — the final post-fix run's numbers landed in the stray path and were never transcribed. The recorded verdict (~36×) is unaffected in direction, but the failure mode is exactly the staleness the evidence file exists to prevent.

Suggested fix

Resolve the change dir at runtime instead of hardcoding it — prefer openspec/changes/archive/*-verify-subagent-pull-under-load/ when the active dir is absent, and fail (rather than mkdirSync) when neither exists, so a misdirected write is loud instead of silent.

Low severity: both specs are opt-in (PI_SYNTH_AGENT_TICKS=1, and PI_E2E_MEASURE=1 for the measurement file), so neither runs in normal CI.

Found while shipping #548.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions