fix(runners): scope schema-validation crashes to the failing case#60
Open
Saladerl wants to merge 3 commits into
Open
fix(runners): scope schema-validation crashes to the failing case#60Saladerl wants to merge 3 commits into
Saladerl wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contain per-case adapter failures in the flow and server-verify runners
AdapterClient.call()validates every adapter response againstadapter-response.schema.jsonand the operation'ssuccessRef— and itraises (
ValueError) on a violation rather than returning an errorresponse:
The vector runner wraps every
call()per check(
run_operation_timed/run_adapter), so a schema-invalid response therefails just that one check. The flow runner and the server-verify runner do
not: their only exception handler is the per-adapter one in
main().Consequence: if an adapter returns one schema-invalid value — say its
challenge.parseoutput for flow case 5 drops a required field — the raisepropagates out of the case loop and the whole adapter collapses into a single
adapter-runfailure. The ~30 other flow cases that ran (or would have run)report nothing, and the failure message doesn't say which case or operation
was at fault. The result is technically red, but useless for diagnosis and
wrong about scope.
Changes
flow_runner.py: newrun_flow_case_safely()wrapper used byrun_adapter_flows(). A case-level exception now yields a normal per-caseresult with
error_type: "case_error: <message>", which the goldencomparison reports as that case failing — all other cases still run and
report individually.
server_verify_runner.py: the per-caseclient.call()is wrapped the sameway, appending a failing
RunResultfor the case and continuing the loop.flow_runner.pyshutdown:server.kill()in thefinallyblock is nowfollowed by
server.wait()so an unresponsive compliance server is reapedinstead of lingering as a zombie for the rest of the CI job.
No behavior change for adapters that respond correctly; the golden comparison
path and result shapes are untouched.
Verification
test_flow_case_isolation.py(new; auto-discovered by ci.yml) checks bothdirections: an exception from
run_flow_casebecomes a scoped per-casefailure carrying the original message, and a successful case result passes
through unchanged. Full script suite: 11 tests, OK.