fix(codex): preserve wrapped sessions and recover state

[rudironsoni]

Description

Closes #2159.

Codex wrappers currently launch against a disposable CODEX_HOME, so session state created during a wrapped run can disappear when that temporary directory is removed. This change launches Codex against its durable home, keeps proxy routing process-local, and adds recovery for retained temporary homes and pinned recovery sources.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Performance improvement
• Code refactoring (no functional changes)

Changes Made

• Launch Codex against its durable CODEX_HOME and apply routing through process-local config overrides after the actual proxy port is resolved.
• Preserve custom provider identity and reject providers that cannot be redirected safely.
• Detect dangling temporary Codex homes before interactive wraps and offer recovery.
• Add headroom recover codex with automatic discovery, repeatable --source, preview, confirmation, retained backups, and rollback on failure.
• Search Python's temp root, $TMPDIR, /tmp, /private/tmp, and macOS /private/var/folders/*/*/T for retained headroom-codex-home-* directories.
• Reuse source-pinned/ copies left by interrupted or failed recovery attempts after the original temporary home has disappeared.
• Report deleted temporary homes still referenced by SQLite rollout paths without treating paths pasted into prompts or errors as filesystem evidence.
• Audit the durable thread index, rollout files, and history when no source remains, including indexed chat counts and history-only orphan records.
• Normalize legacy localhost headroom providers in both SQLite thread rows and rollout session_meta, including retries after an earlier broken recovery, while preserving user-defined remote providers named headroom.
• Merge compatible config, JSONL, rollout, SQLite, credential, and regular-file state without propagating deletions or runtime artifacts.
• Rewrite recovered thread rollout paths to the durable home and restore legacy Headroom thread providers to the active provider.
• Validate SQLite schemas, SQLx migration checksums, integrity, and foreign keys, and quarantine malformed JSONL.
• Preserve failed targets with an atomic rename before rollback, avoiding recursive-deletion races with live SQLite runtime files.
• Document discovery, migration, retained backups, rollback behavior, and the limits of deleted-source recovery.

Testing

• Unit tests pass (pytest)
• Linting passes (ruff check)
• Type checking passes (mypy)
• New tests added for new functionality
• Manual testing performed in isolated Docker containers

Test Output

$ uv run pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_recover_codex.py -q
120 passed

$ uv run ruff check headroom/cli/recover.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py
All checks passed!

$ uv run ruff format --check headroom/cli/recover.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py
3 files already formatted

$ uv run mypy headroom/cli/recover.py headroom/providers/codex/recovery.py
Success: no issues found in 2 source files

All validation ran in ghcr.io/astral-sh/uv:python3.12-bookworm against a writable disposable copy of a read-only source mount. Codex was not installed or launched, and no real user Codex state was read or modified.

The tests cover multi-root discovery, deleted-reference reporting, retained pinned-source recovery, durable SQLite path relocation, SQLite and rollout provider normalization, idempotent repair after an earlier broken recovery, remote provider preservation, unrelated dangling target rows, backup retention, atomic rollback, malformed-state quarantine, SQLite validation, and Windows-safe handle closure.

The repository shim E2E was not launched locally because this recovery work intentionally avoids launching Codex. Upstream CI exercises wrapper E2E in isolated environments.

Review Readiness

• I have performed a self-review
• This PR is ready for human review

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code where the behavior is hard to understand
• I have made corresponding documentation changes
• My changes generate no new warnings
• I have added tests that prove the fix is effective
• New and existing focused unit tests pass with my changes
• I have updated CHANGELOG.md if applicable

Additional Notes

The temporary-home behavior was introduced by #1507 in ad9d086f43a664c4c2a19060b847f2e03ce4f6ad. Related context: #730, #731, #961, #1034, #1050, #1349, #1853, #1889, #2103, and #2104.

A temporary home that macOS or TemporaryDirectory already deleted cannot be reconstructed unless a retained source-pinned/ copy exists. Recovery identifies genuine dangling SQLite paths, audits surviving durable history, and recovers any retained pinned source it can find. Prompt text without a rollout cannot reconstruct a full transcript.

The unchecked changelog item is not applicable because this repository does not require a changelog entry for this fix.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

PR governance

This PR does not yet satisfy the required template fields:

• Missing required section Real Behavior Proof.

Please update the PR body, or move the PR back to draft while it is still in progress.

[JerrettDavis]

Reviewed current head e6dc3af. This is a large Codex behavior change, but the durable-home launch path preserves provider identity through process-local config overrides, and the recovery flow now has transactional backups plus explicit SQLite handle closure before rollback restore. I fixed the Windows rollback/test portability issues and verified locally: uvx ruff@0.15.17 check/format for the Codex wrap/recovery files, and uv run --extra dev pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_recover_codex.py -q (107 passed, 1 skipped). CI is rerunning on this head.

[JerrettDavis]

Validated current head 16bce6e after merging latest headroomlabs/main and resolving the uv.lock Pillow floor conflict in favor of the upstream security floor.

Checks run:

• uv lock --check
• git diff --check
• uvx ruff@0.15.17 check headroom/cli/recover.py headroom/cli/wrap.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py tests/test_cli/test_wrap_codex.py
• uvx ruff@0.15.17 format --check headroom/cli/recover.py headroom/cli/wrap.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py tests/test_cli/test_wrap_codex.py
• uv run --extra dev pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_recover_codex.py -q (107 passed, 1 skipped)

[JerrettDavis]

Approved current head 59f4162 after adding the Windows-safe SQLite rollback handle cleanup.

Validated locally:

• uv run --extra dev python -m pytest tests/test_cli/test_recover_codex.py::test_recovery_rolls_back_when_source_sqlite_breaks_foreign_keys -q
• uv run --extra dev python -m pytest tests/test_cli/test_recover_codex.py tests/test_cli/test_wrap_codex.py -q
• uvx ruff@0.15.17 check headroom/cli/recover.py headroom/cli/wrap.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py tests/test_cli/test_wrap_codex.py
• uvx ruff@0.15.17 format --check headroom/cli/recover.py headroom/cli/wrap.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py tests/test_cli/test_wrap_codex.py
• uvx mypy@1.20.2 headroom/providers/codex/recovery.py headroom/cli/recover.py --ignore-missing-imports
• git diff --check

[JerrettDavis]

Reviewed the new provider-normalization commit 26540c2. The rollout/session_meta rewrite is scoped to legacy localhost headroom routing, preserves remote providers named headroom, and keeps newer target rollouts intact while repairing the stale provider marker.

Validated locally on the PR head:

• uv run --extra dev python -m pytest tests/test_cli/test_recover_codex.py::test_recovery_restores_legacy_headroom_threads_to_active_provider tests/test_cli/test_recover_codex.py::test_recovery_repairs_legacy_provider_after_a_previous_broken_recovery tests/test_cli/test_recover_codex.py::test_recovery_preserves_nonlocal_provider_named_headroom -q
• uvx ruff@0.15.17 check headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py

The visible red checks are still the shared dependency audit / release-smoke baseline being addressed separately by #2190.