Skip to content

Coverage thresholds are a phantom gate — never run (no --coverage anywhere), provider undeclared, codecov upload uploads a file nothing generates #314

Description

@rpatricksmith

Summary

packages/cli/vitest.config.ts declares hard coverage thresholds (lines 80 / branches 75 / functions 80 / statements 80), and project-context.md states "Coverage thresholds enforced in vitest.config.ts." They are not enforced — they have never run. The gate is dead at three layers: no script or CI step passes --coverage, the coverage provider isn't a declared dependency, and the codecov upload step ships a file nothing generates. A quality gate everyone (including our own context docs and the pipeline planner) believes is enforcing is decorative.

Evidence (grounded)

  1. Thresholds declaredpackages/cli/vitest.config.ts:26: coverage: { provider: 'v8', thresholds: { lines: 80, branches: 75, functions: 80, statements: 80 } }. These evaluate only under vitest --coverage.
  2. Nothing passes --coverage:
    • packages/cli/package.json: "test": "vitest"; prepublishOnly: pnpm vitest run — neither has --coverage.
    • ana.json test command: pnpm run test -- --run — no --coverage.
    • CI .github/workflows/test.yml:63: cd packages/cli && pnpm vitest run — no --coverage. release.yml:38 likewise. → thresholds never evaluated; tests gate on results only.
  3. Codecov upload is a no-optest.yml:65-70 uploads ./packages/cli/coverage/coverage-final.json, but nothing generates it (no --coverage run precedes it; packages/cli/coverage/ does not exist), and fail_ci_if_error: false hides the missing file → CI stays green while uploading nothing.
  4. Provider undeclared + uninstalled@vitest/coverage-v8 appears in pnpm-lock.yaml only as an optional peer of vitest (peerDependenciesMeta), is not a devDependency in any package.json, and is not present in node_modules. So vitest run --coverage fails with MISSING DEPENDENCY today.

Root cause

The threshold config was added without wiring the three things that make it real: (a) a --coverage invocation, (b) the v8 provider as a declared dependency, (c) a CI step that actually runs it. The codecov upload was added pointing at the output path of a coverage run that never happens.

Why it matters

  • Verified over trusted, violated in our own infra. The product's whole thesis is mechanical gates that can't be skipped. This is a gate that silently skips itself.
  • It misled real planning. During remove-processcapturestrict, AnaPlan reframed the test-parity AC around "the real gate is the coverage thresholds in vitest.config.ts" and relaxed AC7 accordingly — a decision anchored to a gate that has never executed. Verify flagged it could not run the threshold check (provider absent) and correctly classified it environmental — but the deeper truth is no environment runs it.
  • project-context.md carries a false claim agents read as ground truth.

Proposed fix

Make it real on the Node-22 CI leg (already singled out for the codecov upload), so one wiring change fixes both the enforcement and the upload:

  1. Add @vitest/coverage-v8 (pinned to the vitest version, ^4.1.5) as a devDependency in packages/cli/package.json.
  2. Add a script, e.g. "test:coverage": "vitest run --coverage".
  3. In test.yml, run coverage on the Node-22 matrix leg (gate the run, not just the upload) so the thresholds enforce and coverage-final.json exists for the codecov step. Consider fail_ci_if_error: true once the file is real.
  4. Either keep the thresholds (now enforced) or, if we decide we don't want coverage gating, delete the thresholds block + the codecov step and correct project-context.md — don't leave dead config that misleads.

Acceptance

  • vitest run --coverage runs locally and in CI without MISSING DEPENDENCY.
  • A coverage run executes in CI (Node-22 leg), thresholds enforce (build fails if coverage drops below them), and coverage-final.json is generated for the codecov upload.
  • project-context.md's coverage claim is either true (gate wired) or removed (gate retired) — no phantom gate, no false claim.

Surfaced by

remove-processcapturestrict verify report (AC7 "PARTIAL" — coverage gate not runnable). Mechanically confirmed against main (commit 5fb5057).

Metadata

Metadata

Assignees

No one assigned

    Labels

    technical-debtKnown debt items from verify reportsverify-over-trustMechanical checks that prevent human error

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions