Skip to content

[codex] Fix runner trace matrix and Hermes detection#6

Merged
armand0e merged 1 commit into
mainfrom
codex/runner-matrix-hermes-claude-fixes
Jun 11, 2026
Merged

[codex] Fix runner trace matrix and Hermes detection#6
armand0e merged 1 commit into
mainfrom
codex/runner-matrix-hermes-claude-fixes

Conversation

@armand0e

Copy link
Copy Markdown
Collaborator

Summary

This patch release fixes the trace detection/conversion issues uncovered while testing the same two-turn prompt across every supported generation path.

  • Bumps Teich from 0.1.3 to 0.1.4 in pyproject.toml, src/teich/__init__.py, and uv.lock.
  • Recognizes Teich-exported Hermes external traces (external_session_meta with source: hermes-agent) as hermes, while still preserving the raw source metadata.
  • Normalizes Codex conversion metadata to trace_type: codex and keeps Codex's native source separately as source: exec.
  • Resumes Claude Code follow-up turns by native session id (--resume <sessionId>) instead of plain --continue, preventing Claude Code from injecting a synthetic "Continue from where you left off" turn and replaying the initial prompt.
  • Chmods both external runner home directories and /workspace before sandbox snapshotting, fixing permission-denied failures after Hermes/Claude follow-up turns create files.
  • Adds a generated Hermes example trace using the first examples/prompts.jsonl row, including its follow-up prompt.
  • Adds focused converter/runner regressions for Hermes detection, Codex metadata, Claude follow-up resume, and workspace chmod behavior.

Root Cause

The live runner matrix exposed three issues:

  1. Teich's Hermes runner exports traces as external_session_meta events with source: hermes-agent, but detection classified that shape as generic external_agent.
  2. Claude Code follow-up turns used --continue, which can add an implicit continuation prompt and replay the original prompt before the real follow-up. Using the native session id with --resume preserves the intended prompt sequence.
  3. Codex native session metadata uses source: exec to describe Codex CLI mode, not trace type. Conversion was surfacing that as metadata.trace_type, which made converted metadata disagree with detect_trace_type().

Validation

  • Live runner matrix completed for all supported providers: chat, hermes, claude-code, pi, and codex using google/gemini-3.1-flash-lite via OpenRouter.
  • Matrix validation confirmed each generated output converts to one row with turn_count=2 and includes both the initial prompt and follow-up prompt.
  • uv run pytest -> 345 passed, 21 skipped.
  • uv run ruff check .
  • uv run python -m compileall -q src tests
  • git diff --check
  • uv build
  • uv pip check

Notes

chat output is already structured training data, so detect_trace_type() correctly returns None for that file while conversion still accepts it as trace_type: chat.

Bump Teich to 0.1.4 for a patch release covering trace detection and runner conversion fixes.

Changes:

- recognize Teich-exported Hermes external traces as Hermes while preserving the raw source metadata

- keep Codex conversion metadata aligned with public detection by using trace_type=codex and source=exec

- resume Claude Code follow-up turns by native session id instead of plain --continue to avoid synthetic continuation prompts

- chmod mounted workspaces before external-runner sandbox snapshots

- add a generated Hermes example using the first prompts.jsonl row, including its follow-up turn

- add regression coverage for Hermes detection/conversion, Codex metadata, Claude follow-up resume, and workspace chmod

Validation:

- live runner matrix passed for chat, hermes, claude-code, pi, and codex using google/gemini-3.1-flash-lite via OpenRouter

- matrix validation confirmed detection/conversion output, one converted row, turn_count=2, and both prompt turns for every supported runner

- uv run pytest: 345 passed, 21 skipped

- uv run ruff check .

- uv run python -m compileall -q src tests

- git diff --check

- uv build

- uv pip check
@armand0e
armand0e marked this pull request as ready for review June 11, 2026 03:53
@armand0e
armand0e merged commit 6e7a2b8 into main Jun 11, 2026
3 checks passed
@armand0e
armand0e deleted the codex/runner-matrix-hermes-claude-fixes branch June 11, 2026 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant