Skip to content

fix(runners): scope schema-validation crashes to the failing case#60

Open
Saladerl wants to merge 3 commits into
tempoxyz:mainfrom
Saladerl:fix/flow-case-isolation
Open

fix(runners): scope schema-validation crashes to the failing case#60
Saladerl wants to merge 3 commits into
tempoxyz:mainfrom
Saladerl:fix/flow-case-isolation

Conversation

@Saladerl

Copy link
Copy Markdown

Contain per-case adapter failures in the flow and server-verify runners

AdapterClient.call() validates every adapter response against
adapter-response.schema.json and the operation's successRef — and it
raises (ValueError) on a violation rather than returning an error
response:

# harness.py
validate_value(response, "adapter-response.schema.json", ...)
if response.get("ok") is True:
    validate_value(response["value"], {"$ref": success_ref}, ...)  # raises

The vector runner wraps every call() per check
(run_operation_timed / run_adapter), so a schema-invalid response there
fails just that one check. The flow runner and the server-verify runner do
not: their only exception handler is the per-adapter one in main().

Consequence: if an adapter returns one schema-invalid value — say its
challenge.parse output for flow case 5 drops a required field — the raise
propagates out of the case loop and the whole adapter collapses into a single
adapter-run failure. The ~30 other flow cases that ran (or would have run)
report nothing, and the failure message doesn't say which case or operation
was at fault. The result is technically red, but useless for diagnosis and
wrong about scope.

Changes

  • flow_runner.py: new run_flow_case_safely() wrapper used by
    run_adapter_flows(). A case-level exception now yields a normal per-case
    result with error_type: "case_error: <message>", which the golden
    comparison reports as that case failing — all other cases still run and
    report individually.
  • server_verify_runner.py: the per-case client.call() is wrapped the same
    way, appending a failing RunResult for the case and continuing the loop.
  • flow_runner.py shutdown: server.kill() in the finally block is now
    followed by server.wait() so an unresponsive compliance server is reaped
    instead of lingering as a zombie for the rest of the CI job.

No behavior change for adapters that respond correctly; the golden comparison
path and result shapes are untouched.

Verification

test_flow_case_isolation.py (new; auto-discovered by ci.yml) checks both
directions: an exception from run_flow_case becomes a scoped per-case
failure carrying the original message, and a successful case result passes
through unchanged. Full script suite: 11 tests, OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant