Skip to content

Keep two timing tests of the command line green on a busy Windows runner - #433

Merged
ssegura merged 1 commit into
masterfrom
fix/timing-tests-on-a-busy-runner
Oct 30, 2026
Merged

ssegura merged 1 commit into
masterfrom
fix/timing-tests-on-a-busy-runner

Conversation

@ssegura

@ssegura ssegura commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Related: None.

What you can do now that you could not before

Nothing changes for someone running RESTest: only tests change, and nothing under src/main.
Continuous integration stops going red by chance on a busy Windows runner. On 30 October 2026 two
command-line tests failed on windows-latest and passed when re-run, with no code changed: the
refused-lap timing test on Java 25 (run 41073186888, attempt 1),
and the rate test on Java 26 (run 41080563445, attempt 1).
Neither now counts a slow start against the run, and neither checks less than it did.

Try it yourself

From a clean checkout, with only Java and git. The three rate tests and the refused-lap test pass,
and Maven prints nothing:

git fetch && git switch fix/timing-tests-on-a-busy-runner
./mvnw -q test -pl restest-cli -am -Dsurefire.failIfNoSpecifiedTests=false \
    -Dtest='KeepingToARateCommandTest,RunLoopTest#a_slow_401_to_the_lap_stops_the_run_when_it_arrives'

To see that the refused-lap test still catches what it is there for, break that behaviour: make the
run keep the slot it took when the 401 stops it. The test then fails, five seconds after the 401
rather than within a fraction of one:

sed -i.bak '328s|slots.release();|// slots.release();|' restest-cli/src/main/java/io/restest/cli/RunLoop.java
./mvnw -q test -pl restest-cli -am -Dsurefire.failIfNoSpecifiedTests=false \
    -Dtest='RunLoopTest#a_slow_401_to_the_lap_stops_the_run_when_it_arrives'
mv restest-cli/src/main/java/io/restest/cli/RunLoop.java.bak restest-cli/src/main/java/io/restest/cli/RunLoop.java

Expected, among Maven's output:
  [it stopped once the answer was in, and waited for nothing it was not owed]
  Expecting actual:
    5.015510417S
  to be less than:
    4S

How it works

The refused-lap test checks that a run whose opening lap the API refused with 401 stops as soon
as the last 401 arrives, and does not then wait five seconds (the wait for late answers) for an
answer nobody owes it. By design that 401 takes two seconds to arrive. The test timed the whole run
from its start and wanted less than four seconds, so a slow start and a slow machine had to fit in
the other two. It now times from the moment the 401 arrived. A correct run ends a fraction of a
second later; a broken one ends five seconds later; the four-second line sits between them with
almost four seconds to spare, whatever happened before the 401.

The rate test checks that a run kept to ten requests a second spaces them a turn apart, sends
nothing once its time is up, and says how much quiet the rate made. The "nothing once its time is
up" part needs the run's time to end it, and RESTest takes the time it spends reading the document
and starting up out of that same time, on purpose. On the failing runner, starting up took 4.5
seconds of a 2-second budget, so the run sent nothing. Now, if a run had too little time after
starting up (fewer than three turns between its first request and the end of its time), the test
runs it again with twice the time, up to 8 seconds. On an ordinary machine it runs once, in two
seconds, as before. On a busy one it takes longer instead of failing. Every check is made on a run
that had room, and none is looser.

The other two tests in that class end their runs after 200 requests, not when time runs out, but
they still had only two seconds, starting up included. The same slow start would have failed them.
They now get the 30-second ceiling that every other command-line test ended by its number of
requests already uses, and they still check that their 200 requests took less than two seconds.

What changed in the code

  • RunLoopTest: the stand-in engine ApiEngine now records every answer, by operation and by the
    monotonic clock (everyAnswer). firstAnsweredSince(operation, since) finds the first answer of
    one run when the engine serves several, as it does in this test's loop over two patiences.
    a_slow_401_to_the_lap_stops_the_run_when_it_arrives measures from the first listShelters answer
    of each run to the run's end. The bound itself (PATIENT.minusSeconds(1), four seconds) is unchanged.
  • KeepingToARateCommandTest: run takes the budget. keepingToTheRate(budget) clears the
    stand-in's request log and the captured output, then runs. turnsLeft(budget) reads the first
    stored request and the report's timeline.startedAt. LONGEST (four times BUDGET) caps the
    retries. CEILING (30 s) is for the two tests ended by their number of requests. assertFasterThanTheRate
    now compares with BETWEEN.multipliedBy(20) (two seconds, as before) instead of BUDGET,
    which those tests no longer use.
  • Nothing under src/main, no settings, no documentation.

Evidence

  • Tests: none added, three changed. ./mvnw -q verify -pl restest-cli -am is green locally on
    macOS — CI: run 41086627777,
    green on its first attempt, Java 21, 25 and 26 on Linux, macOS and Windows
  • Architecture tests: unchanged — none weakened
  • Mutation score (if oracles or constraints were touched): not applicable, none touched. I checked
    the changed assertion against the fault it guards by hand: with the slot release removed from
    RunLoop (the sed above), the test fails at 5.01 s after the 401 against the 4 s bound.
  • Per-request overhead: not applicable, test-only change
  • Smoke run, two containerised APIs: not applicable, test-only change
  • Idle time: not applicable, test-only change
  • Artefact: numbers from temporary probes, which are not in the change.
    • Refused-lap test: the run ended 0.23 s after the 401 with 200 ms of patience, and 0.015 s
      after it with none. The whole run took 2.30 s and 2.02 s.
    • Rate test, with its first budget forced down to 150 ms so that the run had no room: it ran at
      150 ms, 300 ms and 600 ms. The last run had five turns of room, sent six requests, and passed
      every check.

Decisions taken

  • Refused-lap test: measure from the 401 instead of raising a number. History (git log -S):
    the test came in with "Stop a run after a certain number of 401" (Stop a run after a certain number of 401 #422). There, the slow answer
    took 0.8 s and the four-second bound left 3.2 s of room. Then "Keep a run to a rate given in one
    line" (Keep a run to a rate given in one line: at most so many requests a minute, evenly spaced #425) raised the slow answer to 2 s, because a Windows runner had finished the lap after the
    401 came, and did not touch the bound. That halved the room to 2 s, and the failure here took 4.155 s.
    Raising the bound again would bring it closer to the five seconds a broken run takes. Measuring
    from the 401 takes the two-second answer and the start out of the measurement. The moment is the
    stand-in engine's own record of when it answered, not the moment a listener heard of it, since
    listeners hear later and on their own schedule.
  • Rate test: run again instead of a bigger budget. The budget has to pay for starting up:
    RunCommand sets the deadline at the moment the command started plus the budget, on purpose.
    The stall was 4.5 s (the run's own summary says "0 requests to 0 operations in 4.5s"), so a fixed
    budget that would have survived it would be 8 to 10 s, paid on every build on every platform.
    Running again costs nothing on an ordinary machine. The trigger is the run's own record (nothing
    sent, or fewer than three turns between its first request and the end of its time), never a
    failed check. A run that had room and sent too few requests, sent them too close together or sent
    one too late still fails. The cap of 8 s makes sure that a run that never gets going still ends.
  • The upper bound on arrivals stays budget / turn + 2, now using the budget the run actually
    had. I considered measuring it from the first request, which is tighter. But that bound is
    within a request of what a busy machine produces, and this change is about removing chance
    failures, not adding a stricter check.
  • The two no-rate tests of the same class are included even though neither failed. They have
    the same cause, the same 4.5 s stall would have failed them, and the fix is the one ARun
    already documents for runs ended by their number of requests.

Open questions

None.


  • Targets master
  • English throughout, including comments and test names
  • The hard rules in AGENTS.md hold
  • Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code

Two restest-cli tests failed on windows-latest on 30 October 2026 and passed when re-run, with
nothing under src/main changed.

The refused-lap test timed the whole run from its start against four seconds, two of which the
slow 401 itself takes since it was raised from 0.8 s; a Windows runner took 4.155 s. It now times
from the 401 to the run's end: a correct run ends a fraction of a second after it, one that keeps
the slot it took ends five seconds after it.

The rate test's run spent 4.5 s of its 2 s budget starting up and sent nothing. RESTest pays for
starting up out of the budget on purpose, so a run that had no room for three turns between its
first request and the end of its time is now run again with twice the time, up to 8 s; every check
is made on a run that had room. The two other tests of the class, ended by their number of
requests, get the 30 s ceiling the other such command-line tests use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura
ssegura merged commit 7b63b22 into master Oct 30, 2026
11 checks passed
@ssegura
ssegura deleted the fix/timing-tests-on-a-busy-runner branch October 30, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant