Skip to content

CI: contract + canary + live e2e suites (channels, voice) - #9

Merged
dimavrem22 merged 2 commits into
mainfrom
ci-live-test-stack
Jul 2, 2026
Merged

CI: contract + canary + live e2e suites (channels, voice)#9
dimavrem22 merged 2 commits into
mainfrom
ci-live-test-stack

Conversation

@dimavrem22

Copy link
Copy Markdown
Contributor

Ports the proven plugin-fleet CI architecture (hermes-agent-plugin, openclaw-plugin) to the Codex bridge. External-events suite deliberately excluded, as in openclaw-plugin.

The three tiers

  1. tests.yml — offline unit matrix (Python 3.10/3.12, replaces ci.yml) on every push/PR, plus a per-PR contract gate against the real Codex host.
  2. canary.yml — the same contract suite on a 2×/day schedule (06:13 / 18:13 PT — one hour before the openclaw-plugin canary, two before hermes, so host breakage pages in waves, not all at once). Google Chat alert on scheduled failures only. live-channels chains off a passing canary via workflow_run.
  3. live-channels.yml / live-voice.yml — boot the real bridge (inkbox-codex run) driving a real codex app-server on the AUT identity (CODEX_INKBOX_* secrets); the shared driver identity (REMOTE_INKBOX_*) emails/texts/calls it and asserts delivery + content + transcripts. Mock-model leg (deterministic, free) + real gpt-5.5 leg; voice runs inbound_inkbox and outbound_realtime scenarios. Ready-PR/dispatch/canary-chain gating, the repo-wide inkbox-live-aut-tunnel concurrency lock, and failure-only log dumps (public repo) all match the fleet pattern.

"Freshest Codex main each time"

Codex is a Rust host — building main from source per run would cost 15–30 min of compile. OpenAI cuts the @openai/codex@alpha npm prerelease from main near-daily (today's alpha.33 was published ~2 h before this PR), so every contract/canary/live run installs that: a prebuilt binary of main within ~24 h, per run, no compile.

Codex-specific findings baked in

  • wire_api = "chat" was removed from Codex main (Deprecating `chat/completions` support in Codex openai/codex#7782): custom providers must speak the Responses API. The mock model server (tests/live/mock_openai.py) therefore implements POST /v1/responses (SSE streaming + non-streaming) alongside chat completions.
  • No account auth needed for the mock leg: the app-server handshake, thread/start, and turns on a custom provider all run unauthenticated — verified locally against alpha.33. The real leg authenticates with printenv OPENAI_API_KEY | codex login --with-api-key.
  • Contract suite is turn-deep: beyond raw-protocol probes (initialize, thread/start/resume, turn/interrupt, account/rateLimits/read, account/usage/read), it drives a full mock turn + cross-process thread resume through the bridge's own CodexAppServerClient — the exact notification stream (item/agentMessage/delta, item/completed, turn/completed) the gateway depends on, at zero token cost. 5/5 green locally vs alpha.33.
  • Unattended-runner policy: live legs set CODEX_APPROVAL_POLICY=never + CODEX_SANDBOX=read-only — nobody is on the other end to answer an approval text, and a stray command stays harmless.
  • thread/resume requires a persisted rollout, so a fresh no-turn thread can't resume — the contract asserts the method routes, and proves real resume turn-first (matching how the bridge actually resumes from sessions.json).

Live suite deltas from the fleet

  • Env key is CODEX_INKBOX_API_KEY; error-marker for a failed turn is the bridge's canned "hit an error" reply.
  • Tool names are scraped from inkbox_codex/tools.py string literals (no manifest here).
  • The contact-CRUD opt-in test does create + update through the agent and cleans up via the SDK — this bridge exposes no contact-delete tool.

dimavrem22 and others added 2 commits July 2, 2026 02:24
Three tiers, mirroring the fleet pattern:
- tests.yml: offline unit matrix (3.10/3.12) + per-PR contract gate vs the
  freshest Codex main prerelease (@openai/codex@alpha, cut near-daily).
- canary.yml: same contract suite 2x/day (06:13/18:13 PT), Chat alert on
  scheduled failure; live-channels chains off a PASSING canary.
- live-channels.yml / live-voice.yml: boot the real bridge + a real Codex
  app-server on the AUT identity; a remote identity drives email/SMS/
  cross-channel/voice scenarios end to end (mock model leg + real gpt-5.5
  leg; inbound Inkbox STT/TTS + outbound realtime call legs).

Contract suite runs a full turn through the bridge's own CodexAppServerClient
against a local deterministic Responses-API mock (wire_api "chat" is gone
from the host), plus raw-protocol probes for thread/turn/account methods —
no account auth, no token spend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bridge's channel prompt keeps replies short, so the model sometimes folds
near-duplicate contact tools together when enumerating. Ask harder (list ALL,
do not omit) and require a majority of the contact tools rather than every one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dimavrem22
dimavrem22 marked this pull request as ready for review July 2, 2026 02:39
@dimavrem22
dimavrem22 merged commit 1677d13 into main Jul 2, 2026
12 checks passed
@dimavrem22
dimavrem22 deleted the ci-live-test-stack branch July 2, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant