Skip to content

forensics.ts derive emits indefensible provenance (Codex cost ~7×, fan-out tokens dropped ~38%, commands_run=0) — committed into the proof chain #310

Description

@rpatricksmith

Summary

forensics.ts's deterministic derive (deriveClaude/deriveCodex/resolveTranscriptPath) emits provenance numbers that are wrong on real sessions, and those numbers are committed into the proof chain via cross-machine-provenance (.ana/plans/active/<slug>/provenance/<role>-<session_id>.json → git). The proof chain's entire value is being defensible; for Codex sessions and Claude fan-out sessions it currently carries indefensible numbers.

All findings below were reproduced against real transcripts on a dev machine (not theoretical). Filing for triage/scoping — do not auto-fix; the corrected derive should be designed once and coordinated with the planned anatrace-core port (cannibalize-then-follow), but the live bleeding is an Anatomia-side defect independent of that timeline.

Verified defects

  1. Codex cost double-counts cached input (~7× over). OpenAI's total_token_usage.input_tokens is gross (cache-inclusive). deriveCodex sets tokens.input = input_tokens (gross) and tokens.cache_read = cached_input_tokens (forensics.ts:558-560); computeCost then bills the cached tokens at both the input rate and the cache-read rate (pricing.ts:95-99). On a real rollout: input=7,131,005, cached=6,940,032 (97.3% of input) → $10.04 vs the defensible $1.37 (+635%) at gpt-5.5 rates. Fix direction: input_non_cached = input - cached (ccusage/codeburn both subtract).

  2. commands_run is always 0 on current Codex. deriveCodex increments only when name === 'shell' (forensics.ts:572), but current Codex names the shell tool exec_command (real rollout: exec_command:69, write_stdin:16, apply_patch:24, zero shell).

  3. Claude fan-out / subagent tokens are dropped (~38% undercount). resolveTranscriptPath globs ~/.claude/projects/**/<id>.jsonl (forensics.ts:638-665) and never discovers the separate projects/<proj>/<session>/subagents/agent-*.jsonl files that current Claude (2.1.x) writes for sub-agents (spawned via the Agent tool). On a real fan-out session: parent dedup output 172,424; its 32 subagent files add 107,638 (requestId overlap 0) → 38.4% output undercount. 873 subagent files across 85 fan-out sessions on one machine — this is the dominant mode, not an edge case.

  4. requestId-only dedup drops usage. deriveClaude skips token usage entirely when requestId is absent (forensics.ts:462). Should fall back to message.id (ccusage does). Currently low-impact on sampled corpus but algorithmically wrong and a drift risk.

Related / lower severity

  1. Codex files_touched hardcoded 0 (forensics.ts:593). A clean structured source exists: event_msg payload.type === 'patch_apply_end' carries changes keyed by file path.
  2. parseTestCounts regex-over-prose (forensics.ts:403-415) can over/under-count tests_executed/failures_encountered from arbitrary tool output.
  3. Codex "last total wins" is reset-unsafe (forensics.ts:556-562) — undercounts on compaction/resume (latent; not observed on sampled corpus). ccusage sums per-turn deltas instead.

Impact

  • The committed derived block in proof-chain provenance is wrong for every Codex session and every Claude fan-out session where processCapture is on (dogfood: always; customers: opt-in).
  • Directly undercuts "verified over trusted" / "never report a number you can't defend."

Notes for triage (not a fix mandate)

  • The correction is the real change; consider also a committed derive_version (sibling to price_table_version) so a corrected derive is a visible version bump, not a silent rewrite of committed history.
  • Decision already leaned (founder): correct + re-baseline + version, and narrow the frozen ProvenanceCounts contract to tokens/model/price_table_version (the best-effort counts should not be in the bit-frozen tier).
  • Coordinate the corrected logic with the future anatrace-core derive so the two don't diverge.

Evidence reproduced 2026-06-08 against real ~/.claude and ~/.codex transcripts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions