Skip to content

Make three tests that failed by chance pass every time, and send no request past the deadline for a pause - #445

Open
ssegura wants to merge 1 commit into
masterfrom
fix/tests-that-failed-by-chance
Open

ssegura wants to merge 1 commit into
masterfrom
fix/tests-that-failed-by-chance

Conversation

@ssegura

@ssegura ssegura commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Related: #442 and #444, whose CI each failed once in a test it did not touch; both said these would be put right in a pull request of their own after the release.

What you can do now that you could not before

A run no longer sends a request after its time is up because that request waited until then for the
run's reports to catch up: one whose turn came just before the deadline, during such a pause, used to
go out after it. When the API had asked for that operation to be left alone for longer than the run
had left, the run asked it again at the very end and counted a second wait. And the three tests that each failed once on CI during the
2.5.0 release, by timing or by chance, now pass every time.

Try it yourself

git fetch && git switch fix/tests-that-failed-by-chance
./mvnw -q install -DskipTests
./mvnw -q test -pl restest-cli -Dsurefire.rerunFailingTestsCount=0 \
    -Dtest='RunLoopTest,ToldToWaitTest,FuzzingFindsWhatNominalMissesTest'
./mvnw -q test -pl restest-exec -Dsurefire.rerunFailingTestsCount=0 -Dtest=EngineFaultsTest

Expected: nothing from the first; from the second, the stub server's three warnings,
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
and exit code 0 from both.

Without the change to the run loop, the two new tests fail every time:

git checkout origin/master -- restest-cli/src/main
./mvnw -q test -pl restest-cli -Dsurefire.rerunFailingTestsCount=0 \
    -Dtest='RunLoopTest#a_request_that_waited*,ToldToWaitTest#nothing_heard*'
git checkout HEAD -- restest-cli/src/main

Expected, among the output (the wait counted twice is the failure CI saw):
  [nothing went once the time was up: Outcome[sent=3, answered=3, ...]]
  Expecting actual:
    2026-11-10T20:25:32.746503Z
  to be strictly before:
    2026-11-10T20:25:32.738194Z
  ...
  [one wait, an hour long, and nothing begun once the time was up]
  Expecting actual:
    [ForOperation[operation=GET /tans/{id}, times=2, stated=2, waited=PT1H, turnsSkipped=1],
      ForOperation[operation=POST /tans, times=1, stated=0, waited=PT0S, turnsSkipped=0],
      ForOperation[operation=GET /tans, times=1, stated=1, waited=PT0S, turnsSkipped=0]]

How it works

A run sends nothing once its time is up. The run loop asks the scheduler what to send, and the
scheduler says "time is up" at the deadline. Between its answer and the request going out the loop
may wait: for room among the requests in flight, for the reports to catch up, for a quiet the API
asked for, for the engine's turn. Every one of those waits ends at the deadline, and all but the one
for the reports then stopped the run; that one went on to send. The loop now looks at the clock once
more after them, and does not send once the time is up. And what the API answers once the time is up
begins no wait, since the run sends nothing more: a wait begun then would last no time at all. That
is what one test caught. Asked by the API to leave an operation alone for an hour in a run of a second
and a half, the run left it alone until its deadline, then sent it one more request that was refused
again, and counted two waits where there was one.

A redirection test that no longer races the stub server. When one of its time limits runs out,
the engine says what it was still waiting for: a connection to the API, which never reached it, or a
reply, which the API sat on. The test sends the engine to an address that answers with a redirection
to one that takes connections and never says a word, and expects "a connection". The first address was
a stub server, and this test is the first its class runs, so its redirection was the first request
that server answered; on a slow runner that took longer than the 300 ms the engine waits, the wait
that ran out was the one for the first address's reply, and the engine rightly said so. The first
address now answers the moment it takes the connection, before reading anything, and the request
carries a body far larger than a computer holds back for one connection: the engine cannot finish
sending it, and start waiting for a reply, until that address has read it, which it does only after
answering.

A seed for the test that needs one value drawn by chance. That test compares the plan RESTest
carries with one that never pushes at the API, against an API that breaks on an empty search term.
The empty term is one of about forty awkward values drawn at random, so no number of requests makes a
run reach it every time, and the seed one build machine drew reached it in none of 1,200 requests.
Both runs are now given a seed of their own. Against this API the seed alone decides what is sent, and
this one reaches the empty term six times.

What changed in the code

  • restest-cli/src/main/java/io/restest/cli/RunLoop.java: after the waits that come between the
    scheduler's "send" and the send itself, and before the operation is asked whether it may be sent,
    the loop gives the slot back and stops if the deadline has passed. The class comment says so.
  • restest-cli/src/main/java/io/restest/cli/ToldToWait.java: settled begins and moves nothing once
    the run's time is up, as it already did once the run had stopped sending. The class comment says so.
  • RunLoopTest.a_request_that_waited_past_the_deadline_is_not_sent (new): the reports fall behind with the first
    announcement and catch up only after the deadline, so the loop is pausing for them, a request's turn
    decided, when the deadline passes. The test records the run's deadline and when the stand-in engine
    was last asked to send.
  • ToldToWaitTest.nothing_heard_once_the_time_is_up_begins_a_wait (new): the failure on CI, on a clock
    the test moves: a wait capped at the deadline, then a refusal of a request sent at the deadline and
    the first refusals of two other operations.
  • EngineFaultsTest.a_redirection_nowhere_is_a_wait_for_a_connection: a hand-written first address
    that answers before reading, keeps 64 KiB unread, and takes a 32 MiB POST body, in place of the
    stub server and a GET.
  • FuzzingFindsWhatNominalMissesTest.the_awkward_values_are_what_find_it: both runs with
    --seed 20261110, and a comment that says what the seed does and why the size stays, in place of
    one that said the size made the draw reach the empty term every time.
  • No page changes. docs/known-limitations.md already says a wait longer than the run has left "is
    obeyed until the run's time is up", and docs/report.md that a wait that began after the run
    stopped sending is not counted; both now hold to the moment.

Evidence

  • Tests: 2 added, 2 changed. ./mvnw verify on macOS, Java 25: 4,983 tests, none failed. CI: run 42899328216.

  • Architecture tests: unchanged — none weakened. No coverage or mutation threshold changed.

  • Mutation score: not applicable, restest-oracles is untouched.

  • Per-request overhead: not measured. The loop reads the clock once more per request it sends.

  • Smoke run, two containerised APIs: ./mvnw verify -Psmoke passed, SmokeRunTest 2 of 2.
    Operations covered and unique faults not compared: the change sends at most one request fewer per
    run, at its very end.

  • Idle time: not applicable.

  • Artefact: each failure's cause, reproduced, and the repeated runs, on a Mac with 18 cores.

    Test Failed on CI Cause, reproduced After the change
    RunLoopTest.an_operation_told_to_wait_is_skipped Java 26, ubuntu, Release 2.5.0: jbang, a skill for coding agents, and the documentation at one address #442: times 2, expected 1 A request sent at the deadline, refused again, began a second wait. Too rare to reproduce by repetition: 300 runs of the test before the change, with every core busy, none failed. Shown instead by the two new tests, which fail on every run without the change (3 of 3) 300 runs with every core busy: none failed
    EngineFaultsTest.a_redirection_nowhere_is_a_wait_for_a_connection Java 26, ubuntu, Tell a build of master from the 2.5.0 release: master moves on to 2.6.0-SNAPSHOT #444: Optional[REPLY], expected CONNECTION The stub's 302 later than the 300 ms read timeout. Delaying it by 400 ms gives CI's message exactly. The test is the first of its class (surefire's report), and locally the 302 alone took about 175 ms of the 300 200 runs on macOS, every core busy, and 50 on Linux in a container: none failed. With the first address waiting 2 s before it answers: still CONNECTION
    FuzzingFindsWhatNominalMissesTest.the_awkward_values_are_what_find_it Java 25, macOS, Tell a build of master from the 2.5.0 release: master moves on to 2.6.0-SNAPSHOT #444: no answered 500 With CI's seed, 1749106978450864122, the run sends no empty term in 1,200 requests, the same 1,200 values on each of 3 runs. Twenty other seeds reached it 3 to 13 times each Seed 20261110: 6 empty terms, the first within the first 200 requests, the same values on each of 3 runs. 25 runs of the test, every core busy: none failed

    The repeated runs used @RepeatedTest in a copy of the test that is not part of this change, with
    -Dsurefire.rerunFailingTestsCount=0, and 18 busy loops beside them.

Decisions taken

  • The code was wrong, not the expectation, for the run loop. The test expects a wait the API asked
    for at the start of the run to be counted once, and it should be: the run asked again, after its own
    deadline, an operation the API had asked to be left alone for an hour.
  • Two changes rather than one. The look at the clock alone leaves a moment of a few microseconds
    between reading it and the request going out; what is heard once the time is up beginning no wait
    alone would still send that request. Together, nothing goes out after the deadline but in that
    moment, and whatever comes back to it counts for nothing, so the test cannot fail this way.
  • No switch for either. They are not a lever an experiment would turn off: the budget already says
    the run stops sending when its time is up, and these make the loop keep to it. The first sends at
    most one request fewer per run; the second sends nothing differently at all.
  • The redirection test stays end to end. It checks that the engine really does not hear from the
    second address before it gives up, which a test of the classification alone would not. 32 MiB is
    many times what one connection within a machine holds back before a write has to wait, measured
    with the receiving side keeping 64 KiB unread, as the test's does: 471 KiB on macOS and 2,644 KiB on
    Linux (in a container), where the 32 MiB take 6 and 9 ms. A review of the first draft pointed out
    that 8 KiB, the first choice, could slow the body down on Linux, where it took 28 ms; hence 64 KiB.
    Windows was not measured; on a machine that held back more than 32 MiB, the test would be no worse
    than before, since the first address still answers before it reads. What is still timed is sending
    the body, given 10 s for each part and a minute in all.
  • What report.json says changes in one corner. A refusal heard between the deadline and the
    moment the loop noticed it is no longer counted in toldToWait.operations[].times and stated, and
    an operation whose only refusal came then is no longer listed: each was a wait of no length. A
    review pointed out that a reader comparing runs could see the difference.
  • What the run loop promises is narrower than "nothing after the deadline". The loop looks at the
    clock once its waits are over, before it builds the request. A request whose building takes it past
    the deadline, which takes far less than a millisecond for nearly every request, still goes out; what
    comes back to it counts for nothing. A review asked for the wording to say so, or for a second look
    at the clock after building; the wording now says so, since that look would throw away a request
    already built, a step of a series among them, to gain a fraction of a millisecond.
  • A seed, and the size kept. The seed makes the test the same on every machine. The size stays at
    1,200 requests so that a change to how values are drawn, which gives the seed other values, is still
    unlikely to miss the empty term; if one does, the test fails on that change, every time, rather than
    once in a few hundred builds later.

Open questions

None.


  • Targets master
  • English throughout, including comments and test names
  • The hard rules in AGENTS.md hold

🤖 Generated with Claude Code

…equest past the deadline for a pause

A request whose turn came just before the deadline, while the run loop paused for its reports to
catch up, went out after it: that pause ends at the deadline and the loop then sent. Against an API
that had asked for the operation to be left alone for longer than the run had left, the run asked it
again at the very end, and counted a second wait - which is how
RunLoopTest.an_operation_told_to_wait_is_skipped failed once on CI. The loop now looks at the clock
after its waits and does not send once the time is up, and what is heard once the time is up begins
no wait. Two new tests show each, and fail every time without the change.

EngineFaultsTest.a_redirection_nowhere_is_a_wait_for_a_connection failed because its stub's 302,
the first request that stub answered, came later than the 300 ms read timeout on a slow runner; the
engine rightly said it had been waiting for a reply. The first address now answers before it reads,
and the request is larger than a connection holds back, so the engine waits for a reply only once it
has one.

FuzzingFindsWhatNominalMissesTest.the_awkward_values_are_what_find_it missed the empty search term
with the seed a build machine drew; no size makes a draw reach it every time. Both runs now have a
seed, which here alone decides what is sent, and the comment says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant