fix(runner): keep --output json parseable and contain vector-file crashes#57
Open
AlexBill01 wants to merge 2 commits into
Open
fix(runner): keep --output json parseable and contain vector-file crashes#57AlexBill01 wants to merge 2 commits into
AlexBill01 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vector_runner: keep
--output jsonmachine-readable and survive malformed vector filesCI consumes this runner's stdout as a JSON artifact:
Two failure modes can silently destroy that artifact today.
1. Stray
print()calls bypass the json guardVectorRunner.log()exists precisely to suppress human-readable output whenoutput_format == "json", butrun_vector_file()and_record_result()stillcall
print()directly in a few places (missing vector file, vector file withno
commands, and the verbose per-test lines). Any of those lines lands infront of the JSON document, so
merge_vector_results.pyfails to parse theartifact and the adapter's entire pass/fail breakdown collapses into a single
"Could not read results" failure.
All of these call sites now go through
self.log(), which is a no-op in jsonmode and unchanged in text mode.
2. A malformed vector file aborts the whole run
run()invokedrun_vector_file()unguarded. One vector file with invalidJSON — or a scenario missing a required key like
inputorexpected—raised straight out of
main(), so in json mode nothing was written tostdout and every check that had already run was lost.
Per-adapter and per-operation errors were already contained
(
run_adapter/run_operation_timedcatch exceptions and record failures);file-level errors are now contained the same way: the runner records a failing
vector_file_errorcheck for that file and keeps going, and the overall runcorrectly reports
fail.Verification
The new tests pin down: no stdout leakage for command-less vectors, no stdout
leakage for missing vector files, and a malformed vector file producing a
failing check (not a crash