Repository navigation
Conversation
…equest past the deadline for a pause A request whose turn came just before the deadline, while the run loop paused for its reports to catch up, went out after it: that pause ends at the deadline and the loop then sent. Against an API that had asked for the operation to be left alone for longer than the run had left, the run asked it again at the very end, and counted a second wait - which is how RunLoopTest.an_operation_told_to_wait_is_skipped failed once on CI. The loop now looks at the clock after its waits and does not send once the time is up, and what is heard once the time is up begins no wait. Two new tests show each, and fail every time without the change. EngineFaultsTest.a_redirection_nowhere_is_a_wait_for_a_connection failed because its stub's 302, the first request that stub answered, came later than the 300 ms read timeout on a slow runner; the engine rightly said it had been waiting for a reply. The first address now answers before it reads, and the request is larger than a connection holds back, so the engine waits for a reply only once it has one. FuzzingFindsWhatNominalMissesTest.the_awkward_values_are_what_find_it missed the empty search term with the seed a build machine drew; no size makes a draw reach it every time. Both runs now have a seed, which here alone decides what is sent, and the comment says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #442 and #444, whose CI each failed once in a test it did not touch; both said these would be put right in a pull request of their own after the release.
What you can do now that you could not before
A run no longer sends a request after its time is up because that request waited until then for the
run's reports to catch up: one whose turn came just before the deadline, during such a pause, used to
go out after it. When the API had asked for that operation to be left alone for longer than the run
had left, the run asked it again at the very end and counted a second wait. And the three tests that each failed once on CI during the
2.5.0 release, by timing or by chance, now pass every time.
Try it yourself
Without the change to the run loop, the two new tests fail every time:
How it works
A run sends nothing once its time is up. The run loop asks the scheduler what to send, and the
scheduler says "time is up" at the deadline. Between its answer and the request going out the loop
may wait: for room among the requests in flight, for the reports to catch up, for a quiet the API
asked for, for the engine's turn. Every one of those waits ends at the deadline, and all but the one
for the reports then stopped the run; that one went on to send. The loop now looks at the clock once
more after them, and does not send once the time is up. And what the API answers once the time is up
begins no wait, since the run sends nothing more: a wait begun then would last no time at all. That
is what one test caught. Asked by the API to leave an operation alone for an hour in a run of a second
and a half, the run left it alone until its deadline, then sent it one more request that was refused
again, and counted two waits where there was one.
A redirection test that no longer races the stub server. When one of its time limits runs out,
the engine says what it was still waiting for: a connection to the API, which never reached it, or a
reply, which the API sat on. The test sends the engine to an address that answers with a redirection
to one that takes connections and never says a word, and expects "a connection". The first address was
a stub server, and this test is the first its class runs, so its redirection was the first request
that server answered; on a slow runner that took longer than the 300 ms the engine waits, the wait
that ran out was the one for the first address's reply, and the engine rightly said so. The first
address now answers the moment it takes the connection, before reading anything, and the request
carries a body far larger than a computer holds back for one connection: the engine cannot finish
sending it, and start waiting for a reply, until that address has read it, which it does only after
answering.
A seed for the test that needs one value drawn by chance. That test compares the plan RESTest
carries with one that never pushes at the API, against an API that breaks on an empty search term.
The empty term is one of about forty awkward values drawn at random, so no number of requests makes a
run reach it every time, and the seed one build machine drew reached it in none of 1,200 requests.
Both runs are now given a seed of their own. Against this API the seed alone decides what is sent, and
this one reaches the empty term six times.
What changed in the code
restest-cli/src/main/java/io/restest/cli/RunLoop.java: after the waits that come between thescheduler's "send" and the send itself, and before the operation is asked whether it may be sent,
the loop gives the slot back and stops if the deadline has passed. The class comment says so.
restest-cli/src/main/java/io/restest/cli/ToldToWait.java:settledbegins and moves nothing oncethe run's time is up, as it already did once the run had stopped sending. The class comment says so.
RunLoopTest.a_request_that_waited_past_the_deadline_is_not_sent(new): the reports fall behind with the firstannouncement and catch up only after the deadline, so the loop is pausing for them, a request's turn
decided, when the deadline passes. The test records the run's deadline and when the stand-in engine
was last asked to send.
ToldToWaitTest.nothing_heard_once_the_time_is_up_begins_a_wait(new): the failure on CI, on a clockthe test moves: a wait capped at the deadline, then a refusal of a request sent at the deadline and
the first refusals of two other operations.
EngineFaultsTest.a_redirection_nowhere_is_a_wait_for_a_connection: a hand-written first addressthat answers before reading, keeps 64 KiB unread, and takes a 32 MiB
POSTbody, in place of thestub server and a
GET.FuzzingFindsWhatNominalMissesTest.the_awkward_values_are_what_find_it: both runs with--seed 20261110, and a comment that says what the seed does and why the size stays, in place ofone that said the size made the draw reach the empty term every time.
docs/known-limitations.mdalready says a wait longer than the run has left "isobeyed until the run's time is up", and
docs/report.mdthat a wait that began after the runstopped sending is not counted; both now hold to the moment.
Evidence
Tests: 2 added, 2 changed.
./mvnw verifyon macOS, Java 25: 4,983 tests, none failed. CI: run 42899328216.Architecture tests: unchanged — none weakened. No coverage or mutation threshold changed.
Mutation score: not applicable,
restest-oraclesis untouched.Per-request overhead: not measured. The loop reads the clock once more per request it sends.
Smoke run, two containerised APIs:
./mvnw verify -Psmokepassed,SmokeRunTest2 of 2.Operations covered and unique faults not compared: the change sends at most one request fewer per
run, at its very end.
Idle time: not applicable.
Artefact: each failure's cause, reproduced, and the repeated runs, on a Mac with 18 cores.
RunLoopTest.an_operation_told_to_wait_is_skippedtimes2, expected 1EngineFaultsTest.a_redirection_nowhere_is_a_wait_for_a_connectionOptional[REPLY], expectedCONNECTIONCONNECTIONFuzzingFindsWhatNominalMissesTest.the_awkward_values_are_what_find_itanswered 500The repeated runs used
@RepeatedTestin a copy of the test that is not part of this change, with-Dsurefire.rerunFailingTestsCount=0, and 18 busy loops beside them.Decisions taken
for at the start of the run to be counted once, and it should be: the run asked again, after its own
deadline, an operation the API had asked to be left alone for an hour.
between reading it and the request going out; what is heard once the time is up beginning no wait
alone would still send that request. Together, nothing goes out after the deadline but in that
moment, and whatever comes back to it counts for nothing, so the test cannot fail this way.
the run stops sending when its time is up, and these make the loop keep to it. The first sends at
most one request fewer per run; the second sends nothing differently at all.
second address before it gives up, which a test of the classification alone would not. 32 MiB is
many times what one connection within a machine holds back before a write has to wait, measured
with the receiving side keeping 64 KiB unread, as the test's does: 471 KiB on macOS and 2,644 KiB on
Linux (in a container), where the 32 MiB take 6 and 9 ms. A review of the first draft pointed out
that 8 KiB, the first choice, could slow the body down on Linux, where it took 28 ms; hence 64 KiB.
Windows was not measured; on a machine that held back more than 32 MiB, the test would be no worse
than before, since the first address still answers before it reads. What is still timed is sending
the body, given 10 s for each part and a minute in all.
report.jsonsays changes in one corner. A refusal heard between the deadline and themoment the loop noticed it is no longer counted in
toldToWait.operations[].timesandstated, andan operation whose only refusal came then is no longer listed: each was a wait of no length. A
review pointed out that a reader comparing runs could see the difference.
clock once its waits are over, before it builds the request. A request whose building takes it past
the deadline, which takes far less than a millisecond for nearly every request, still goes out; what
comes back to it counts for nothing. A review asked for the wording to say so, or for a second look
at the clock after building; the wording now says so, since that look would throw away a request
already built, a step of a series among them, to gain a fraction of a millisecond.
1,200 requests so that a change to how values are drawn, which gives the seed other values, is still
unlikely to miss the empty term; if one does, the test fails on that change, every time, rather than
once in a few hundred builds later.
Open questions
None.
masterAGENTS.mdhold🤖 Generated with Claude Code