CI: contract + canary + live e2e suites (channels, voice) - #9
Merged
Conversation
Three tiers, mirroring the fleet pattern: - tests.yml: offline unit matrix (3.10/3.12) + per-PR contract gate vs the freshest Codex main prerelease (@openai/codex@alpha, cut near-daily). - canary.yml: same contract suite 2x/day (06:13/18:13 PT), Chat alert on scheduled failure; live-channels chains off a PASSING canary. - live-channels.yml / live-voice.yml: boot the real bridge + a real Codex app-server on the AUT identity; a remote identity drives email/SMS/ cross-channel/voice scenarios end to end (mock model leg + real gpt-5.5 leg; inbound Inkbox STT/TTS + outbound realtime call legs). Contract suite runs a full turn through the bridge's own CodexAppServerClient against a local deterministic Responses-API mock (wire_api "chat" is gone from the host), plus raw-protocol probes for thread/turn/account methods — no account auth, no token spend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bridge's channel prompt keeps replies short, so the model sometimes folds near-duplicate contact tools together when enumerating. Ask harder (list ALL, do not omit) and require a majority of the contact tools rather than every one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the proven plugin-fleet CI architecture (hermes-agent-plugin, openclaw-plugin) to the Codex bridge. External-events suite deliberately excluded, as in openclaw-plugin.
The three tiers
tests.yml— offline unit matrix (Python 3.10/3.12, replacesci.yml) on every push/PR, plus a per-PR contract gate against the real Codex host.canary.yml— the same contract suite on a 2×/day schedule (06:13 / 18:13 PT — one hour before the openclaw-plugin canary, two before hermes, so host breakage pages in waves, not all at once). Google Chat alert on scheduled failures only.live-channelschains off a passing canary viaworkflow_run.live-channels.yml/live-voice.yml— boot the real bridge (inkbox-codex run) driving a realcodex app-serveron the AUT identity (CODEX_INKBOX_*secrets); the shared driver identity (REMOTE_INKBOX_*) emails/texts/calls it and asserts delivery + content + transcripts. Mock-model leg (deterministic, free) + realgpt-5.5leg; voice runsinbound_inkboxandoutbound_realtimescenarios. Ready-PR/dispatch/canary-chain gating, the repo-wideinkbox-live-aut-tunnelconcurrency lock, and failure-only log dumps (public repo) all match the fleet pattern."Freshest Codex main each time"
Codex is a Rust host — building
mainfrom source per run would cost 15–30 min of compile. OpenAI cuts the@openai/codex@alphanpm prerelease from main near-daily (today's alpha.33 was published ~2 h before this PR), so every contract/canary/live run installs that: a prebuilt binary of main within ~24 h, per run, no compile.Codex-specific findings baked in
wire_api = "chat"was removed from Codex main (Deprecating `chat/completions` support in Codex openai/codex#7782): custom providers must speak the Responses API. The mock model server (tests/live/mock_openai.py) therefore implementsPOST /v1/responses(SSE streaming + non-streaming) alongside chat completions.thread/start, and turns on a custom provider all run unauthenticated — verified locally against alpha.33. The real leg authenticates withprintenv OPENAI_API_KEY | codex login --with-api-key.initialize,thread/start/resume,turn/interrupt,account/rateLimits/read,account/usage/read), it drives a full mock turn + cross-process thread resume through the bridge's ownCodexAppServerClient— the exact notification stream (item/agentMessage/delta,item/completed,turn/completed) the gateway depends on, at zero token cost. 5/5 green locally vs alpha.33.CODEX_APPROVAL_POLICY=never+CODEX_SANDBOX=read-only— nobody is on the other end to answer an approval text, and a stray command stays harmless.thread/resumerequires a persisted rollout, so a fresh no-turn thread can't resume — the contract asserts the method routes, and proves real resume turn-first (matching how the bridge actually resumes fromsessions.json).Live suite deltas from the fleet
CODEX_INKBOX_API_KEY; error-marker for a failed turn is the bridge's canned "hit an error" reply.inkbox_codex/tools.pystring literals (no manifest here).