Skip to content

fix(runner): keep --output json parseable and contain vector-file crashes#57

Open
AlexBill01 wants to merge 2 commits into
tempoxyz:mainfrom
AlexBill01:fix/vector-runner-json-artifacts
Open

fix(runner): keep --output json parseable and contain vector-file crashes#57
AlexBill01 wants to merge 2 commits into
tempoxyz:mainfrom
AlexBill01:fix/vector-runner-json-artifacts

Conversation

@AlexBill01

Copy link
Copy Markdown

vector_runner: keep --output json machine-readable and survive malformed vector files

CI consumes this runner's stdout as a JSON artifact:

vector_runner.py --output json > results-<adapter>.json

Two failure modes can silently destroy that artifact today.

1. Stray print() calls bypass the json guard

VectorRunner.log() exists precisely to suppress human-readable output when
output_format == "json", but run_vector_file() and _record_result() still
call print() directly in a few places (missing vector file, vector file with
no commands, and the verbose per-test lines). Any of those lines lands in
front of the JSON document, so merge_vector_results.py fails to parse the
artifact and the adapter's entire pass/fail breakdown collapses into a single
"Could not read results" failure.

All of these call sites now go through self.log(), which is a no-op in json
mode and unchanged in text mode.

2. A malformed vector file aborts the whole run

run() invoked run_vector_file() unguarded. One vector file with invalid
JSON — or a scenario missing a required key like input or expected
raised straight out of main(), so in json mode nothing was written to
stdout and every check that had already run was lost.

Per-adapter and per-operation errors were already contained
(run_adapter / run_operation_timed catch exceptions and record failures);
file-level errors are now contained the same way: the runner records a failing
vector_file_error check for that file and keeps going, and the overall run
correctly reports fail.

Verification

$ python3 scripts/test_vector_runner.py
Ran 9 tests ... OK            # 6 existing + 3 new
$ python3 scripts/vector_runner.py --adapter typescript --vector base64url --output json | python3 -m json.tool
# parses cleanly, status "pass", 30 checks

The new tests pin down: no stdout leakage for command-less vectors, no stdout
leakage for missing vector files, and a malformed vector file producing a
failing check (not a crash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant