fix(codex): preserve wrapped sessions and recover state

docs/content/docs/codex-recovery.mdx

@@ -0,0 +1,79 @@
+---
+title: Recover Codex State
+description: Merge sessions and configuration left in a temporary Headroom Codex home back into the durable Codex home.
+---
+
+Older versions of `headroom wrap codex` could run Codex with a temporary `CODEX_HOME` named `headroom-codex-home-*`. Chats and configuration created during that wrapped session stayed in the temporary home, so they disappeared from the normal Codex history after the wrapper exited. This recovery migrates the retained state back into the durable Codex home without deleting either source.
+
+This procedure can recover temporary homes that still exist and pinned sources retained by an interrupted recovery. A temporary home that was already deleted cannot be reconstructed unless one of those retained copies exists. Headroom reports deleted temporary homes still referenced by the Codex thread database. It does not treat paths pasted into prompts or error messages as recovery sources. See [issue #2159](https://github.com/headroomlabs-ai/headroom/issues/2159) for the regression details.
+
+## Before you recover
+
+Close Codex and any process using the temporary or durable Codex homes. Recovery detects changes made while creating each backup and aborts, but quiet homes are required for a consistent migration.
+
+The target defaults to `$CODEX_HOME` when it is set, otherwise `~/.codex`. Check that this is the durable home you normally use before confirming the migration.
+
+## Automatic recovery
+
+The first interactive `headroom wrap codex` with the fixed version searches Python's temporary directory, `$TMPDIR`, `/tmp`, `/private/tmp`, and the macOS `/private/var/folders/*/*/T` roots for non-empty `headroom-codex-home-*` directories. If it finds any, it lists them and offers to back up and recover them before Codex starts.
+
+Declining the prompt changes nothing. You can run the manual command later.
+
+## Manual recovery
+
+Let Headroom discover retained temporary homes and `source-pinned/` copies left by interrupted or failed recovery attempts, then preview the migration:
+
+```bash
+headroom recover codex
+```
+
+If no recoverable copy remains, the command audits the durable Codex thread database, rollout files, and `history.jsonl`. It reports indexed active and archived chats, surviving rollouts missing from the thread index, and history-only records whose rollouts no longer exist. History-only prompt text cannot reconstruct a full chat transcript.
+
+Codex filters the default resume picker by the current working directory. Run `codex resume --all` yourself to display indexed chats from every working directory. Headroom does not launch Codex during recovery.
+
+To select a known temporary home and an explicit durable target:
+
+```bash
+headroom recover codex \
+  --source /path/to/headroom-codex-home-12345 \
+  --target "${CODEX_HOME:-$HOME/.codex}"
+```
+
+Repeat `--source` to merge multiple homes. Headroom shows the target and every source before asking for confirmation. Use `--yes` only in automation where those paths have already been reviewed.
+
+## What gets migrated
+
+When a source used the localhost `headroom` model provider injected by the old wrapper, recovery rewrites that provider in both SQLite thread rows and rollout `session_meta` records to the active target provider. This also repairs newer target records left by an earlier broken recovery. A user-defined remote provider named `headroom` is preserved.
+
+- `history.jsonl` and other JSONL indexes are combined without duplicate records. Malformed input is excluded from the result and copied to the backup quarantine.
+- Session rollouts under `sessions/` and `archived_sessions/` keep the newer file when the same path exists in both homes.
+- SQLite databases are merged only when their tables, indexes, triggers, views, and migration checksums are compatible. Primary-key conflicts keep rows from the newer database. Recovered thread rows are rewritten to the durable rollout path, including rows restored from a pinned source after the original temporary home was deleted.
+- `config.toml` tables are merged recursively. Values from the newer config win, and localhost Headroom routing injected by the old wrapper is removed. A user-defined remote provider is preserved.
+- Other files, including credentials and user settings, keep the newer copy. The durable target wins equal modification-time ties, and missing source files never delete target files.
+
+Sockets, lock files, SQLite journals, FIFOs, and other runtime-only artifacts are recorded but not copied.
+
+## Backups and rollback
+
+Every source is pinned before migration. The existing durable home, including its current Codex configuration, is backed up before any merge begins. Backups are owner-only and retained next to the target:
+
+```text
+<target-parent>/.headroom-codex-recovery/<timestamp-pid>/
+├── source-pinned/
+├── target-before/
+├── target-failed/     # only when a merge is rolled back
+├── manifest.json
+└── quarantine/       # only when malformed input is found
+```
+
+`target-before/` is absent when the target did not exist before recovery. `manifest.json` records copied, merged, quarantined, and skipped paths.
+
+If configuration parsing, SQLite schema validation, integrity checks, foreign-key checks, or filesystem writes fail, Headroom atomically renames the failed target to `target-failed/` before restoring `target-before/`. This avoids recursive-deletion races with SQLite runtime files and preserves the failed merge for inspection. When the target did not exist before recovery, the partial target is retained only as `target-failed/`. The pinned source and recovery backup remain available for inspection.
+
+Recovery also refuses overlapping source and target paths, target symlink traversal, and a source that changes while it is being pinned.
+
+## After recovery
+
+The command prints the retained backup path after each successful source merge. Review its `manifest.json`, then start Codex normally and confirm that the recovered chats and settings appear. Keep the backup until you have verified the durable history.
+
+Fixed versions of `headroom wrap codex` run Codex against the durable home and apply proxy routing only to the launched process, so new wrapped sessions remain visible after Headroom exits.

docs/content/docs/configuration.mdx

@@ -19,6 +19,8 @@ headroom wrap codex --prepare-only
 
 Supported values are `rtk` and `lean-ctx`; unset defaults to `rtk`.
 
+If Codex history disappeared after using an older wrapper, see [Recover Codex State](/docs/codex-recovery) before wrapping Codex again.
+
 The proxy reads RTK lifetime savings with global scope by default so a shared
 daemon reports savings across the operator's projects. Set
 `HEADROOM_RTK_GAIN_SCOPE=project` to query `rtk gain --project` from the

docs/content/docs/meta.json

@@ -40,6 +40,7 @@
     "mcp",
     "---Configuration---",
     "configuration",
+    "codex-recovery",
     "pipeline-extensions",
     "filesystem-contract",
     "---Observability---",

e2e/wrap/run.py

@@ -581,8 +581,19 @@ def verify_codex_wrap(
 
     config_path = Path(base_env["HOME"]) / ".codex" / "config.toml"
     assert_true(
-        not config_path.exists(),
-        "Codex wrap should leave ~/.codex/config.toml untouched during normal launch",
+        config_path.exists(),
+        "Codex wrap should persist its MCP setup in the durable Codex home",
+    )
+    durable_config = config_path.read_text(encoding="utf-8")
+    assert_true(
+        "[mcp_servers.headroom]" in durable_config,
+        "Codex wrap should persist the Headroom MCP server",
+    )
+    assert_true(
+        'model_provider = "headroom"' not in durable_config
+        and "[model_providers.headroom]" not in durable_config
+        and "openai_base_url" not in durable_config,
+        "Codex wrap should keep proxy routing out of the durable config",
     )
 
     entries = read_jsonl(log_dir / "codex.jsonl")
@@ -591,49 +602,41 @@ def verify_codex_wrap(
     session_home = env_vars.get("CODEX_HOME")
     assert_true(
         isinstance(session_home, str) and session_home,
-        "Codex wrap should launch the child with a session-scoped CODEX_HOME",
+        "Codex wrap should launch the child with CODEX_HOME",
     )
     assert_true(
-        Path(session_home) != config_path.parent,
-        "Codex wrap should not point the child at the real ~/.codex home",
+        Path(session_home) == config_path.parent,
+        "Codex wrap should point the child at the durable ~/.codex home",
     )
     config = entries[-1].get("session_config")
     assert_true(
         isinstance(config, str) and config,
-        "Codex wrap should capture the session-scoped config during launch",
-    )
-    assert_true(
-        f'openai_base_url = "http://127.0.0.1:{port}/v1"' in config,
-        "Codex wrap should inject openai_base_url into the session config",
+        "Codex wrap should capture the durable config during launch",
     )
+    project_prefix = f"/p/{quote(project_dir.name, safe='')}"
+    expected_base_url = f"http://127.0.0.1:{port}{project_prefix}/v1"
+    argv = entries[-1].get("argv")
     assert_true(
-        f'base_url = "http://127.0.0.1:{port}/v1"' in config,
-        "Codex wrap should inject the headroom provider base_url into the session config",
+        isinstance(argv, list) and f'openai_base_url="{expected_base_url}"' in argv,
+        "Codex wrap should pass openai_base_url as a process-local override",
     )
     assert_true(
         'env_key = "OPENAI_API_KEY"' not in config,
         "Codex wrap should preserve OAuth and never inject env_key into the session config",
     )
-    # Bug 3 (#406): requires_openai_auth must be absent from headroom provider blocks.
     assert_true(
-        "requires_openai_auth" not in config,
-        "Codex wrap must NOT inject requires_openai_auth into the session headroom provider block",
-    )
-    assert_true(
-        "supports_websockets = true" in config,
-        "Codex wrap missing 'supports_websockets = true' in the session config",
-    )
-
-    assert_true(
-        env_vars.get("OPENAI_BASE_URL") == f"http://127.0.0.1:{port}/v1",
+        env_vars.get("OPENAI_BASE_URL") == expected_base_url,
         "Codex wrap should set OPENAI_BASE_URL",
     )
     assert_true(
         entries[-1]["probes"]
         == [
-            {"url": f"http://127.0.0.1:{port}/v1/models", "status": 200},
-            {"url": f"http://127.0.0.1:{port}/v1/chat/completions", "status": 200},
-            {"url": f"http://127.0.0.1:{port}/stats", "status": 200},
+            {"url": f"{expected_base_url}/models", "status": 200},
+            {"url": f"{expected_base_url}/chat/completions", "status": 200},
+            {
+                "url": f"http://127.0.0.1:{port}{project_prefix}/stats",
+                "status": 200,
+            },
         ],
         "Codex shim should prove OPENAI_BASE_URL points at a live proxy and that Headroom logged the wrapped message",
     )

headroom/cli/__init__.py

@@ -23,6 +23,7 @@
     mcp,
     perf,
     proxy,
+    recover,
     tools,
     update,
     wrap,

headroom/cli/main.py

@@ -61,6 +61,7 @@ def _register_commands() -> None:
         output_savings,  # noqa: F401
         perf,  # noqa: F401
         proxy,  # noqa: F401
+        recover,  # noqa: F401
         savings,  # noqa: F401
         tools,  # noqa: F401
         update,  # noqa: F401

headroom/cli/recover.py

@@ -0,0 +1,94 @@
+"""Recovery commands for state left behind by interrupted wrappers."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import click
+
+from headroom.cli.main import main
+from headroom.providers.codex.recovery import (
+    audit_codex_history,
+    discover_dangling_homes,
+    discover_referenced_temp_homes,
+    discover_retained_sources,
+    recover_codex_home,
+)
+
+
+@main.group()
+def recover() -> None:
+    """Recover agent state left in a temporary Headroom home."""
+
+
+@recover.command("codex")
+@click.option(
+    "sources",
+    "--source",
+    type=click.Path(path_type=Path, exists=True, file_okay=False, resolve_path=True),
+    multiple=True,
+    help="Temporary Codex home to merge. Repeat to merge more than one.",
+)
+@click.option(
+    "--target",
+    type=click.Path(path_type=Path, file_okay=False, resolve_path=True),
+    help="Active Codex home. Defaults to CODEX_HOME or ~/.codex.",
+)
+@click.option("--yes", is_flag=True, help="Apply the recovery without prompting.")
+def recover_codex(sources: tuple[Path, ...], target: Path | None, yes: bool) -> None:
+    """Merge sessions and configuration from dangling Codex homes."""
+    target = target or Path(os.environ.get("CODEX_HOME", Path.home() / ".codex"))
+    selected_sources = list(sources) or [
+        *discover_dangling_homes(),
+        *discover_retained_sources(target),
+    ]
+    if not selected_sources:
+        deleted_sources = [
+            source for source in discover_referenced_temp_homes(target) if not source.exists()
+        ]
+        if deleted_sources:
+            click.echo("Referenced temporary Codex homes were already deleted:")
+            for source in deleted_sources:
+                click.echo(f"  {source}")
+        audit = audit_codex_history(target)
+        if audit is not None:
+            click.echo(
+                f"Durable Codex history: {audit.indexed} indexed chats "
+                f"({audit.active} active, {audit.archived} archived)."
+            )
+            if audit.unindexed_rollouts:
+                click.echo(
+                    "Surviving rollout files missing from the thread database: "
+                    f"{len(audit.unindexed_rollouts)}"
+                )
+                for session_id in audit.unindexed_rollouts:
+                    click.echo(f"  {session_id}")
+            if audit.history_without_rollout:
+                click.echo(
+                    "History-only records without a surviving rollout: "
+                    f"{len(audit.history_without_rollout)}"
+                )
+                for session_id in audit.history_without_rollout:
+                    click.echo(f"  {session_id}")
+                click.echo("Their full transcripts cannot be restored without a retained rollout.")
+            if audit.indexed:
+                click.echo("Run `codex resume --all` to show chats from every working directory.")
+        click.echo("No recoverable Headroom Codex homes were found.")
+        return
+
+    click.echo(f"Target Codex home: {target}")
+    click.echo("Sources:")
+    for source in selected_sources:
+        click.echo(f"  {source}")
+    click.echo("Both the current target and each source will be backed up before merging.")
+    if not yes and not click.confirm("Recover these Codex homes?", default=True):
+        click.echo("Recovery cancelled. No Codex state was changed.")
+        return
+
+    for source in selected_sources:
+        try:
+            report = recover_codex_home(source=source, target=target)
+        except (OSError, RuntimeError, ValueError) as exc:
+            raise click.ClickException(f"Codex recovery failed: {exc}") from exc
+        click.echo(f"Recovery complete. Backup retained at {report.backup_dir}")