Skip to content

Keep a run to a rate given in one line: at most so many requests a minute, evenly spaced - #425

Merged
ssegura merged 3 commits into
masterfrom
feat/m13-3-rate-ceiling
Oct 29, 2026
Merged

ssegura merged 3 commits into
masterfrom
feat/m13-3-rate-ceiling

Conversation

@ssegura

@ssegura ssegura commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Related: None.

What you can do now that you could not before

You can promise an API's owner a rate in one line. --set engine.mostRequestsPerMinute=300 keeps a
run to five requests a second, evenly spaced: never two closer than a fifth of a second, and never a
burst to make up for a quiet spell. The summary then says how much of the run's idle time the rate
made, so that a slow pace you asked for is not read as a slow tool. Until now nothing bounded a run
but its requests in flight, and against a fast API that is tens of thousands of requests a second.

Try it yourself

From a clean clone, with only Java and git. A one-file Java server stands in for an API whose owner
was promised five requests a second. When it is stopped, it says how many requests arrived, how
close the closest two came, and the most that arrived within any one second:

git fetch && git switch feat/m13-3-rate-ceiling
./mvnw -q install -DskipTests
cat > Owner.java <<'EOF'
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;

/**
 * An API whose owner was promised five requests a second. It answers everything, and when it is
 * stopped it says how many requests arrived, how close the closest two came, and the most that
 * arrived within any one second.
 */
public class Owner {
    private static final List<Long> arrivals = new ArrayList<>();

    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress(8430), 0);
        server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
        server.createContext("/", exchange -> {
            synchronized (arrivals) {
                arrivals.add(System.nanoTime());
            }
            byte[] body = "[]".getBytes();
            exchange.getResponseHeaders().add("Content-Type", "application/json");
            exchange.sendResponseHeaders(200, body.length);
            exchange.getResponseBody().write(body);
            exchange.close();
        });
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            List<Long> at;
            synchronized (arrivals) {
                at = new ArrayList<>(arrivals);
            }
            at.sort(null);
            long closest = Long.MAX_VALUE;
            int most = 0;
            for (int i = 0, j = 0; i < at.size(); i++) {
                if (i > 0) {
                    closest = Math.min(closest, at.get(i) - at.get(i - 1));
                }
                while (at.get(i) - at.get(j) >= 1_000_000_000L) {
                    j++;
                }
                most = Math.max(most, i - j + 1);
            }
            System.out.printf(java.util.Locale.ROOT, "%d requests arrived; the closest two %.3f s apart; at most %d in any"
                    + " one second%n", at.size(), closest / 1e9, most);
        }));
        server.start();
        System.out.println("an API promised five requests a second, at http://localhost:8430");
    }
}
EOF
cat > shelter.yaml <<'EOF'
openapi: 3.0.3
info: {title: Shelter whose owner was promised a rate, version: '1'}
paths:
  /pets:
    get: {operationId: listPets, responses: {'200': {description: every pet}}}
  /shelters:
    get: {operationId: listShelters, responses: {'200': {description: every shelter}}}
  /owners:
    get: {operationId: listOwners, responses: {'200': {description: every owner}}}
EOF
java Owner.java &
./restest run shelter.yaml --url http://localhost:8430 --budget 10s --set engine.mostRequestsPerMinute=300
kill %1

Expected, as it came out of a clean clone of this branch:

an API promised five requests a second, at http://localhost:8430
RESTest testing Shelter whose owner was promised a rate at http://localhost:8430

3 of 3 operations can be tested, seed -4367604517000614864, budget 10s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent
  2 setting(s) are not what RESTest does by default; --print-settings lists them with where each came from

48 requests to 3 operations in 10.0s, 99% of it idle (94% keeping to the rate it was given)
  opening lap: 3 requests in 428ms, 3 of 3 operations answered 2xx
  48 2xx
no faults found
...
48 requests arrived; the closest two 0.175 s apart; at most 6 in any one second
  • The bracket after the idle share is new. 94% of the run was quiet because of the rate, which
    is not the tool being slow.
  • The two settings are the rate and its switch. The switch, engine.limitTheRate, was worked
    out on because a rate was named.
  • The closest two were 0.175 s apart where they arrived, not 0.2 s. The rate is kept where
    RESTest sends. The first request of a run also opens the connection and arrives a little late,
    so the one after it arrives early by comparison. That once let 6 fall within one second at the
    stand-in.

The same, without the --set:

598252 requests to 3 operations in 10.0s, 3% of it idle
598252 requests arrived; the closest two 0.000 s apart; at most 65846 in any one second

How it works

A rate of N requests a minute is a gap: a minute divided by N, 200 ms at 300. The part of RESTest
that sends requests gives no request its turn sooner than that gap after the one before, however
much room it has for requests in flight. It counts from the moment the last request actually went,
so a quiet spell is never made up for with a burst. The part that decides what to send asks first
how long the next request would wait, and builds it only when its turn comes. Requests therefore do
not pile up waiting, built from values that were fresh seconds before. When the run's time is up,
whatever still waits for a turn is not sent.

Naming the rate is enough. engine.limitTheRate is the switch, off by default, and it is worked out
as on once engine.mostRequestsPerMinute is named anywhere: in a settings file, the environment or
--set. A rate typed and then ignored because the switch was forgotten would let the run go as fast
as it can while its owner believed a promise had been made. A printed settings file writes the
switch out as false. A run whose switch is off while its rate was changed, or given over such a
file, says so under its first lines.

Idle time is the share of a run with nothing in flight. It exists to catch a slow tool, so the
quiet the rate makes is counted apart: the time after a request went out and before the next was
allowed to. It is reported in brackets beside the quiet an API asked for, and in report.json as
idleKeepingToTheRate.

What changed in the code

  • restest-core.
    • EngineSettings gains limitTheRate and mostRequestsPerMinute (600), with
      timeBetweenRequests(), which rounds up so a minute never holds one request too many.
    • Settings works the switch out from whether the rate was named.
    • HttpEngine gains untilItsTurn() (zero for an engine with no rate); CredentialedEngine
      passes it on, and dropWhatIsHeld() now also lets go what waits for a turn.
    • EngineStatistics gains idleKeepingToTheRate, with a constructor that keeps the existing call
      sites.
  • restest-exec.
    • ConcurrencyLimiter keeps the one schedule, under the lock that already keeps the hold of
      ADR-0042, which comes first. A request waiting for its turn waits on a condition of its own,
      holds no slot and earns nothing.
    • It counts the requests handed over and not yet given a slot, so that untilItsTurn() can add a
      turn for each. With no rate it counts nothing and takes no lock.
    • EngineActivity reads the limiter's next turn to count the idle time the rate made.
  • restest-cli.
    • RunLoop sits out the turn right before it builds anything, beside the whole API's quiet, with a
      plain sleepUntil split from sitOut.
    • RunCommand prints the line about a rate that is not kept to.
  • restest-report. The field in report.json's engine, and the console's bracket, which now
    leaves out a part that rounds to 0%.
  • Docs.
    • ADR-0043 is new; ADR-0009, ADR-0025 and ADR-0042 are amended.
    • The settings and switches pages, the report page, the known limitations, DESIGN.md's
      glossary, the manual's chapters 7 and 10, and the README.
  • Left for later.
    • The HTML page saying how much of the idle share the rate made, which needs a mock-up first.
    • A rate per operation, or one taken from an API's RateLimit fields.
    • The one request that fetches a document over the network, which the rate does not count.
    • Redirections followed with engine.followRedirects on, which take no turn of their own.

Evidence

  • Tests: ./mvnw verify is green at the tip of this branch: 4,942 tests, 0 failures, BUILD
    SUCCESS. 28 new test methods:

    • KeepingToARateTest: a plain socket stand-in that holds four answers and releases them
      together. The requests that waited behind the stall go out a turn apart. With the engine's turn
      taken out they went 12 to 157 microseconds apart, and the test fails.
    • ConcurrencyLimiterTest:
      • the first request at once and the next a turn later;
      • no burst after a quiet spell, on a clock the test sets;
      • waiting for a turn earns nothing;
      • a hold that begins during a turn still holds;
      • closing, and dropping what is held, let a waiting request go;
      • the count of requests waiting;
      • clocks starting near the ends of their range.
    • EngineActivityTest: the idle time the rate made, exact on a clock the test sets.
    • RunLoopTest: the loop builds a request only when its turn comes, and does not count that
      quiet as the API's.
    • KeepingToARateCommandTest: three whole runs against WireMock.
      • At 600 a minute, the stored run's sends are never closer than half a turn, and nothing is sent
        after the time is up.
      • With no rate, it runs as before.
      • A rate given over a printed file that turns the switch off is said.
    • EngineSettingsTest, SettingsTest, SettingsFromEverywhereTest and CredentialedEngineTest:
      • naming a rate turns it on, from any layer;
      • a switch named off wins;
      • 0, a fraction and too large a number are refused;
      • the gap is rounded up;
      • the decorator passes the wait on.
    • JsonReportTest and ConsoleReportTest cover the field and the bracket.

    CI: on this pull request.

  • Architecture tests: DocumentedSettingsTest, DocumentedSwitchesTest and
    DocumentedReportTest hold the two settings, the switch and the field to their pages. None
    weakened.

  • Mutation score: no oracle or constraint touched.

  • Per-request overhead: none to be seen. With no rate the engine takes no lock it did not
    take before. From the dataset of local APIs, 20 s runs in pairs with the same seed:

    master This branch
    petclinic-rest, seeds 1, 2, 3: requests 115,061; 107,686; 104,623 104,566; 105,596; 110,442
    blogapi, seeds 1, 2, 3: requests 70,008; 75,433; 68,661 59,958; 74,322; 68,663
  • Smoke run: CI runs it on this pull request.

  • Idle time: with no rate, the same as before:

    master This branch
    petclinic-rest, seeds 1, 2, 3 2.3%, 2.3%, 2.1% 2.3%, 2.2%, 2.3%
    blogapi, seeds 1, 2, 3 1.9%, 1.9%, 2.1% 1.6%, 1.7%, 2.2%
  • Artefact: the rate kept, from stored runs of 20 s. The gaps are between the moments
    RESTest recorded sending one request and the next:

    Run Requests Shortest gap Most in any one second Idle (the rate's part) Last request, against the time being up
    blogapi, 30 a minute 10 2,000.6 ms 1 98% (96%) 1.448 s before
    blogapi, 300 a minute 95 200.7 ms 5 92% (87%) 0.205 s before
    blogapi, 3,000 a minute 810 20.3 ms 43 75% (56%) 0.003 s before
    petclinic-rest, 300 a minute 95 201.2 ms 5 92% (86%) 0.174 s before

    None of these runs had a request left unsent at the end. pet-clinic for 30 s at 300 a minute, the
    manual's own command, printed
    145 requests to 43 operations in 30.0s, 85% of it idle (81% keeping to the rate it was given).

Decisions taken

  • Per minute, a whole number, as EvoMaster writes it (maintainer). It is exact for any limit of
    so many a minute. Schemathesis's 5/s, 100/m would have needed a new kind of setting, and
    requests a second with decimals writes a hundred a minute as 1.6667.
  • Evenly spaced, never a burst (maintainer, who asked whether 600 a minute could mean 600 in a
    few seconds and then nothing). A sliding window, as Schemathesis's, allows exactly that.
  • A switch and a number, with the number turning the switch on (maintainer). This keeps the rule
    that a switch is a yes-or-no setting, never a number set to zero, and still makes it one line.
    • The switch is not printed as a comment: that rule follows other settings' values, and this one
      follows whether another was named.
    • A run says so when a printed file's switch would leave a rate unkept. That covers a rate other
      than the default, and any rate given in the environment or with --set over such a file, so
      that --set engine.mostRequestsPerMinute=600 over a printed file does not go unsaid.
  • What waits for its turn when the time is up is dropped (maintainer), as ADR-0042 drops what a
    hold holds. There is hardly ever more than one; in the runs above, none.
  • One schedule, in the engine. A first design kept a turn in the loop and another in the engine,
    each with its own clock; after a stall both would run at the rate, and a queue that formed in the
    engine would never drain. The loop asks the engine instead.
  • The idle time the rate made is reported now (maintainer), in report.json and the summary,
    as the quiet an API asks for has been since ADR-0009's amendment of 29 October. The page is left
    for a mock-up.
  • SettingsTest.every_setting_survives_the_round_trip reads one setting at a time and expects
    the defaults back. For engine.mostRequestsPerMinute it now expects the defaults with the rate
    limited, since naming the rate is how it is turned on. Every other setting is checked as before.
  • The console leaves out a part of the bracket that rounds to 0%, for the API's quiet too, which
    printed (0% waiting as the API asked) before.
  • Simplifications considered and not made:
    • Working the switch out from the rate's value rather than from its being named: that would undo
      the maintainer's choice, since --set engine.mostRequestsPerMinute=600 would then do nothing.
    • Always reading the clock in the limiter: a run with no hold and no rate pays nothing for either,
      as before.
    • Merging the hold's wait and the turn's: they wait on different conditions on purpose.
    • The six-argument EngineStatistics constructor stays, as RunFinished's shorter ones did.

Open questions

None.


  • Targets master
  • English throughout, including comments and test names
  • The hard rules in AGENTS.md hold
  • Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code

ssegura and others added 3 commits October 29, 2026 14:25
--set engine.mostRequestsPerMinute=300 keeps a run to five requests a
second: never two closer than a minute divided by the rate, and never a
burst after a quiet spell. Naming the rate turns engine.limitTheRate on;
off by default. The engine keeps the turns, the run loop builds each
request when its turn comes, what waits for a turn when the time is up
is dropped, and the summary and report.json say how much of the idle
time the rate made. ADR-0043; amendments to ADR-0009, ADR-0025 and
ADR-0042.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te's runs from loading the machine

On a Windows runner with Java 26, the opening lap of RunLoopTest's slow
401 finished after the 401 it waits for, at 0.8 s, so the run sent only
the lap. That answer now takes 2 s. The two runs with no rate in
KeepingToARateCommandTest end after 200 requests rather than spending
their whole budget at full speed beside the other tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…within a turn after it

A Windows runner with Java 25 sent the last request of a run kept to 600
a minute 55 ms after its time was up, past the 25 ms the test allowed.
The loop hands a request over only once its turn has come and stops at
the deadline, so such a request is only as late as the machine is busy;
one still waiting for its turn is dropped, and would go a turn or more
late if it were not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura
ssegura merged commit 0579e91 into master Oct 29, 2026
11 checks passed
@ssegura
ssegura deleted the feat/m13-3-rate-ceiling branch October 29, 2026 13:55
ssegura added a commit that referenced this pull request Oct 30, 2026
…asks, keep to a rate, and Maven Central (#431)

**Related:** None. This is the release of 2.4.0, with what `master`
holds on 30 October: #422 to #429.

## What you can do now that you could not before

- **Get 2.4.0.** Once this is merged and tagged, it installs four ways,
as 2.3.0 does:
  - `brew install isa-group/tap/restest`;
  - `docker run ghcr.io/isa-group/restest:2.4.0`;
  - the release's archives, with Java alone;
  - or built from the source.
- **What 2.4.0 adds:**
- a run whose opening lap the API answers with `401` throughout stops
there and ends with `5` (#422);
- an operation answered `429`, or `503` with `Retry-After`, is left
alone for as long as the API asks (#424);
- `--set engine.mostRequestsPerMinute=<n>` keeps a run to a rate, evenly
spaced (#425);
- the eight modules go to Maven Central as
`es.us.isa.restest:restest-*`, from this release on (#429);
- fixes: both reports kept when a quoted value is cut inside an emoji
(#427), `F920` and `F921` reported on operations RESTest read whole
(#428), and every Jackson library on a version with no known flaw
(#423).
- **What stays the same as in 2.3.0:** the command line, plans and
dictionaries. A run can end with one exit code more, `5`, and
`report.json` only gains members.

## Try it yourself

From a clean clone, with only Java and git:

```bash
git clone --quiet --branch release/2.4.0 https://github.com/isa-group/RESTest.git restest-2.4.0 && cd restest-2.4.0
./mvnw -q install -DskipTests
./restest version
./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://127.0.0.1:1 --budget 3s; echo "exit $?"
ls restest-out
grep -o '"verdict" *: *"[A-Za-z]*"' restest-out/report.json
dist/assemble.sh ../release --archives > /dev/null && ls ../release
tar -xzf ../release/restest-2.4.0.tgz -C ../release && ../release/restest-2.4.0/bin/restest version
```

Below is the real output, from a fresh clone of this branch on 30
October; the Java line will be your own. Nothing listens on port 1, so
the run ends with `3`, and it still writes both files:

```
RESTest 2.4.0
Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64
RESTest testing Pet Shelter at http://127.0.0.1:1

4 of 4 operations can be tested, seed -4339058821541680037, budget 3s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent

64 requests to 4 operations in 363ms, 90% of it idle
  opening lap: 4 requests in 46ms, 0 of 4 operations answered 2xx
  64 no reply
  1 of them were pushing at the API with values nobody sensible would send
  3 creation(s) meant to begin a series about a thing the run created were not accepted, so no series began
no faults found
report written to restest-out/report.json (9.6 KiB)
page written to restest-out/report.html (90.7 KiB)
the run itself was not kept; pass --store to keep every request and reply
restest: nothing at http://127.0.0.1:1 answered any of the 64 requests, so the API was never actually tested. Check the address, and that it is running.
exit 3
report.html
report.json
"verdict":"nothingAnswered"
restest-2.4.0
restest-2.4.0.tgz
restest-2.4.0.zip
RESTest 2.4.0
Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64
```

With Docker as well, the image is built from the same clone, as the
release builds it:

```bash
docker build --quiet --tag restest:2.4.0 . > /dev/null && docker run --rm restest:2.4.0 version
```

```
RESTest 2.4.0
Java 21.0.13 (Eclipse Adoptium), Linux 7.0.14-linuxkit aarch64
```

What the tag will leave on the Central Portal, laid out from the same
clone, now with no `-SNAPSHOT` in any name:

```bash
./mvnw -q package -Pcentral -DskipTests -pl restest-cli -am && dist/central.sh target/central
ls target/central/maven-central/es/us/isa/restest
ls target/central/maven-central/es/us/isa/restest/restest-core/2.4.0
grep -c "<dependency>" target/central/maven-central/es/us/isa/restest/restest-cli/2.4.0/restest-cli-2.4.0.pom
ls restest-cli/target/lib | wc -l
```

```
/…/restest-2.4.0/target/central/maven-central
restest-cli
restest-core
restest-exec
restest-gen
restest-oracles
restest-report
restest-spec
restest-store
restest-core-2.4.0-javadoc.jar
restest-core-2.4.0-javadoc.jar.md5
restest-core-2.4.0-javadoc.jar.sha1
restest-core-2.4.0-sources.jar
restest-core-2.4.0-sources.jar.md5
restest-core-2.4.0-sources.jar.sha1
restest-core-2.4.0.jar
restest-core-2.4.0.jar.md5
restest-core-2.4.0.jar.sha1
restest-core-2.4.0.pom
restest-core-2.4.0.pom.md5
restest-core-2.4.0.pom.sha1
70
      70
```

## How it works

**The version.** `-SNAPSHOT` is how Maven writes "on the way to this
version", and this pull request removes it from every `pom.xml`. After
the merge, the tag `v2.4.0` on the merge commit starts the release
workflow ([`docs/ci.md`](docs/ci.md#releases)):
- it builds the image and the archives, and checks them on three
operating systems and two processors;
- it installs the Homebrew formula for the archive on macOS and Linux;
- it signs the eight modules and uploads them to the Central Portal,
once a maintainer approves the `central` job; the Portal validates them
and publishes nothing;
- it pushes the image, and leaves a draft release for a maintainer to
write the notes in and publish.

A maintainer publishes the deployment on the Portal before publishing
the release, and publishing the release starts the Homebrew workflow,
which writes 2.4.0 into the tap.

**The base images.** Step 1 of every release moves the `Dockerfile`'s
two base images to the newest patch published today. This time the Java
in the image moves too, from 21.0.12.1 to 21.0.13, as the output above
shows, and the pages that print it say so.

**The pages.** The README and chapters 2 and 9 of the manual name the
version to install, and now name 2.4.0. Every other page was written for
2.4.0 as each change landed, from #422 to #429.

## What changed in the code

No Java code changed, and nothing under `src/`.

- **Version:** every `pom.xml`, from `2.4.0-SNAPSHOT` to `2.4.0`, ten
files. The JSON Schemas' `$id` already names `v2.4.0` since #421.
- **`Dockerfile`:** the digests of `eclipse-temurin:25-jdk` and
`eclipse-temurin:21-jre` as they are today.
- **`README.md`:** the installation section fetches, clones and runs
`2.4.0`. Its `restest version` output is the image's above.
- **`docs/manual/02-installing.md`:** the same for Docker, the archive,
the checksums, the source and Updating. The Docker and archive outputs
are those above. The two lines under Homebrew's `restest version` are
what this pull request's run of `release.yml` printed in
`check-homebrew` on macOS, as `docs/ci.md` asks:

  ```
  RESTest 2.4.0
  Java 21.0.12.1 (Homebrew), Mac OS X 26.6.2 aarch64
  ```

They are the lines chapter 2 already had once `2.3.0` became `2.4.0`,
since the formula still installs Homebrew's `openjdk@21` 21.0.12.1 on
macOS; only the version changed.
- **`docs/manual/09-exit-codes-and-ci.md`:** the workflow fetches and
runs the 2.4.0 archive. Its sentence on releases before 2.3.0, which
write no page, stays as it is.
- **`docs/DESIGN.md`:** the stack table named `jackson-core` 2.22.1,
which #423 moved to 2.22.3 in the root POM; it now says 2.22.3.

## Evidence

- [x] **Tests:** `./mvnw verify` is green at `d19edf0f`, with version
2.4.0: 4,945 tests, 0 failures, BUILD SUCCESS in 2 min 33 s. 141 of them
are the architecture and documentation rules, `PublishedSchemasTest`
among them, which on a release requires the schemas' `$id` to name it.
CI on `5baafe06`:
[CI](https://github.com/isa-group/RESTest/actions/runs/41073186888) and
[Release](https://github.com/isa-group/RESTest/actions/runs/41073186769),
19 checks green, and `central` and `publish` skipped, as on every pull
request. In that Release run, `check-homebrew` installed the formula for
this branch's archive on macOS and on Ubuntu: 625 MB and 823 MB, as the
pages say. *Java 25 on windows-latest* failed on its first attempt in
`RunLoopTest.a_slow_401_to_the_lap_stops_the_run_when_it_arrives`, which
took 4.16 s where it allows less than 4 s; the same row passed when
re-run, and that test passed on every row of `master`'s last runs.
Nothing under `src/` changed here.
- [x] **Architecture tests:** unchanged, none weakened.
- [x] **Mutation score:** no oracle or constraint touched.
- [x] **Per-request overhead:** unchanged; nothing under `src/` changed.
- [x] **Smoke run:** unchanged, for the same reason; CI runs it on this
pull request.
- [x] **Idle time:** unchanged.
- [x] **Artefact:** `check-restest.sh 2.4.0` passes on both the archive
and the image built here, every check `ok`.
  - **The archive**, `--launcher` on `restest-2.4.0/bin/restest`:
    - version;
    - `JAVA_TOOL_OPTIONS`;
    - an unknown option answering `2`;
    - a run with no answer ending with `3` and its files;
    - the schemas and `THIRD-PARTY.txt` in the folder;
    - `127` with no Java and with a `JAVA_HOME` that holds none;
    - a symbolic link.
- **The image**, `--image restest:2.4.0`: the same, and `docker stop`
ending a run with `143` and a report that says it was cut short.
- **What Maven Central gets:** `dist/central.sh` passes on the build
above, with every check it makes, fetching each module offline with
Maven among them; `restest-cli`'s POM names the 70 libraries
`restest-cli/target/lib` holds, which the archive's `lib/` carries
beside `restest-cli` itself.

## Decisions taken

- **No release evaluation on the Web Fuzzing Dataset, as for 2.3.0.**
The three safeguards change a run only against an API that answers `401`
to the whole opening lap (#422), that asks to wait (#424), or when a
rate is named (#425, off otherwise); #428 was measured across the
dataset's 52 documents in its own pull request; #427 and #423 change no
request.
- **The base images are today's patch,** as step 1 of a release asks
while Dependabot is off. The image's Java moves from 21.0.12.1 to
21.0.13, so the `restest version` lines of the README and chapter 2 that
come from the image move with it.
- **Chapter 2's Homebrew lines come from this pull request's CI,** not
from a Mac here. 2.4.0 cannot be installed from the tap before it is
published, and `docs/ci.md` names `check-homebrew`'s output as their
source. They needed no commit of their own: what the job printed is what
the page already said.
- **`docs/DESIGN.md`'s `jackson-core` version is corrected here,** being
one line in the release's own stack table; the rest of the table agrees
with the root POM, the wrapper and the `Dockerfile`.
- **The release notes are not in this pull request.** They go into the
draft release after the tag, as `docs/ci.md` says.

## Open questions

- **A new Dependabot alert,
[48](https://github.com/isa-group/RESTest/security/dependabot/48), high,
opened today:** Guava up to 33.7.1 allocates memory without bound when
it deserialises some of its collections, fixed in 33.7.2. The release
carries `guava-33.5.0-jre.jar`, which the OpenAPI parser and the HTTP
client bring and the root POM pins. RESTest's own code uses neither
Guava nor Java serialisation. Whether 2.4.0 waits for a pull request
that moves Guava, as #423 did for Jackson, is the maintainer's call;
this one does not touch it.
- **After the merge, each step taken by a maintainer:**
- the tag `v2.4.0` on the merge commit, and the approval of the
`central` job when the run asks for it;
  - the notes in the draft;
- the deployment `RESTest 2.4.0 run <run>` published on the Central
Portal, and then the release, which also starts the Homebrew workflow
for 2.4.0;
- the README's installation commands tried from outside, `brew install
isa-group/tap/restest` among them, and
`https://repo1.maven.org/maven2/es/us/isa/restest/restest-cli/2.4.0/`
answering;
- the pull request that moves `master` to `2.5.0-SNAPSHOT`, and the
schemas' `$id` to `v2.5.0`.

---

- [x] Targets `master`
- [x] English throughout, including comments and test names
- [x] The hard rules in `AGENTS.md` hold
- [x] Nothing from "Out of scope" in `docs/DESIGN.md` without explicit
approval

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ssegura added a commit that referenced this pull request Oct 30, 2026
…ner (#433)

**Related:** None.

## What you can do now that you could not before

Nothing changes for someone running RESTest: only tests change, and
nothing under `src/main`.
Continuous integration stops going red by chance on a busy Windows
runner. On 30 October 2026 two
command-line tests failed on `windows-latest` and passed when re-run,
with no code changed: the
refused-lap timing test on Java 25 ([run 41073186888, attempt
1](https://github.com/isa-group/RESTest/actions/runs/41073186888/attempts/1)),
and the rate test on Java 26 ([run 41080563445, attempt
1](https://github.com/isa-group/RESTest/actions/runs/41080563445/attempts/1)).
Neither now counts a slow start against the run, and neither checks less
than it did.

## Try it yourself

From a clean checkout, with only Java and git. The three rate tests and
the refused-lap test pass,
and Maven prints nothing:

    git fetch && git switch fix/timing-tests-on-a-busy-runner
./mvnw -q test -pl restest-cli -am
-Dsurefire.failIfNoSpecifiedTests=false \

-Dtest='KeepingToARateCommandTest,RunLoopTest#a_slow_401_to_the_lap_stops_the_run_when_it_arrives'

To see that the refused-lap test still catches what it is there for,
break that behaviour: make the
run keep the slot it took when the 401 stops it. The test then fails,
five seconds after the 401
rather than within a fraction of one:

sed -i.bak '328s|slots.release();|// slots.release();|'
restest-cli/src/main/java/io/restest/cli/RunLoop.java
./mvnw -q test -pl restest-cli -am
-Dsurefire.failIfNoSpecifiedTests=false \

-Dtest='RunLoopTest#a_slow_401_to_the_lap_stops_the_run_when_it_arrives'
mv restest-cli/src/main/java/io/restest/cli/RunLoop.java.bak
restest-cli/src/main/java/io/restest/cli/RunLoop.java

    Expected, among Maven's output:
[it stopped once the answer was in, and waited for nothing it was not
owed]
      Expecting actual:
        5.015510417S
      to be less than:
        4S

## How it works

**The refused-lap test** checks that a run whose opening lap the API
refused with 401 stops as soon
as the last 401 arrives, and does not then wait five seconds (the wait
for late answers) for an
answer nobody owes it. By design that 401 takes two seconds to arrive.
The test timed the whole run
from its start and wanted less than four seconds, so a slow start and a
slow machine had to fit in
the other two. It now times from the moment the 401 arrived. A correct
run ends a fraction of a
second later; a broken one ends five seconds later; the four-second line
sits between them with
almost four seconds to spare, whatever happened before the 401.

**The rate test** checks that a run kept to ten requests a second spaces
them a turn apart, sends
nothing once its time is up, and says how much quiet the rate made. The
"nothing once its time is
up" part needs the run's time to end it, and RESTest takes the time it
spends reading the document
and starting up out of that same time, on purpose. On the failing
runner, starting up took 4.5
seconds of a 2-second budget, so the run sent nothing. Now, if a run had
too little time after
starting up (fewer than three turns between its first request and the
end of its time), the test
runs it again with twice the time, up to 8 seconds. On an ordinary
machine it runs once, in two
seconds, as before. On a busy one it takes longer instead of failing.
Every check is made on a run
that had room, and none is looser.

**The other two tests in that class** end their runs after 200 requests,
not when time runs out, but
they still had only two seconds, starting up included. The same slow
start would have failed them.
They now get the 30-second ceiling that every other command-line test
ended by its number of
requests already uses, and they still check that their 200 requests took
less than two seconds.

## What changed in the code

- `RunLoopTest`: the stand-in engine `ApiEngine` now records every
answer, by operation and by the
monotonic clock (`everyAnswer`). `firstAnsweredSince(operation, since)`
finds the first answer of
one run when the engine serves several, as it does in this test's loop
over two patiences.
`a_slow_401_to_the_lap_stops_the_run_when_it_arrives` measures from the
first `listShelters` answer
of each run to the run's end. The bound itself
(`PATIENT.minusSeconds(1)`, four seconds) is unchanged.
- `KeepingToARateCommandTest`: `run` takes the budget.
`keepingToTheRate(budget)` clears the
stand-in's request log and the captured output, then runs.
`turnsLeft(budget)` reads the first
stored request and the report's `timeline.startedAt`. `LONGEST` (four
times `BUDGET`) caps the
retries. `CEILING` (30 s) is for the two tests ended by their number of
requests. `assertFasterThanTheRate`
now compares with `BETWEEN.multipliedBy(20)` (two seconds, as before)
instead of `BUDGET`,
  which those tests no longer use.
- Nothing under `src/main`, no settings, no documentation.

## Evidence

- [x] Tests: none added, three changed. `./mvnw -q verify -pl
restest-cli -am` is green locally on
macOS — CI: [run
41086627777](https://github.com/isa-group/RESTest/actions/runs/41086627777),
green on its first attempt, Java 21, 25 and 26 on Linux, macOS and
Windows
- [x] Architecture tests: unchanged — none weakened
- [x] Mutation score (if oracles or constraints were touched): not
applicable, none touched. I checked
the changed assertion against the fault it guards by hand: with the slot
release removed from
`RunLoop` (the `sed` above), the test fails at 5.01 s after the 401
against the 4 s bound.
- [x] Per-request overhead: not applicable, test-only change
- [x] Smoke run, two containerised APIs: not applicable, test-only
change
- [x] Idle time: not applicable, test-only change
- [x] Artefact: numbers from temporary probes, which are not in the
change.
- Refused-lap test: the run ended 0.23 s after the 401 with 200 ms of
patience, and 0.015 s
    after it with none. The whole run took 2.30 s and 2.02 s.
- Rate test, with its first budget forced down to 150 ms so that the run
had no room: it ran at
150 ms, 300 ms and 600 ms. The last run had five turns of room, sent six
requests, and passed
    every check.

## Decisions taken

- **Refused-lap test: measure from the 401 instead of raising a
number.** History (`git log -S`):
the test came in with "Stop a run after a certain number of 401" (#422).
There, the slow answer
took 0.8 s and the four-second bound left 3.2 s of room. Then "Keep a
run to a rate given in one
line" (#425) raised the slow answer to 2 s, because a Windows runner had
finished the lap after the
401 came, and did not touch the bound. That halved the room to 2 s, and
the failure here took 4.155 s.
Raising the bound again would bring it closer to the five seconds a
broken run takes. Measuring
from the 401 takes the two-second answer and the start out of the
measurement. The moment is the
stand-in engine's own record of when it answered, not the moment a
listener heard of it, since
  listeners hear later and on their own schedule.
- **Rate test: run again instead of a bigger budget.** The budget has to
pay for starting up:
`RunCommand` sets the deadline at the moment the command started plus
the budget, on purpose.
The stall was 4.5 s (the run's own summary says "0 requests to 0
operations in 4.5s"), so a fixed
budget that would have survived it would be 8 to 10 s, paid on every
build on every platform.
Running again costs nothing on an ordinary machine. The trigger is the
run's own record (nothing
sent, or fewer than three turns between its first request and the end of
its time), never a
failed check. A run that had room and sent too few requests, sent them
too close together or sent
one too late still fails. The cap of 8 s makes sure that a run that
never gets going still ends.
- **The upper bound on arrivals stays `budget / turn + 2`**, now using
the budget the run actually
had. I considered measuring it from the first request, which is tighter.
But that bound is
within a request of what a busy machine produces, and this change is
about removing chance
  failures, not adding a stricter check.
- **The two no-rate tests of the same class are included** even though
neither failed. They have
the same cause, the same 4.5 s stall would have failed them, and the fix
is the one `ARun`
  already documents for runs ended by their number of requests.

## Open questions

None.

---

- [x] Targets `master`
- [x] English throughout, including comments and test names
- [x] The hard rules in `AGENTS.md` hold
- [x] Nothing from "Out of scope" in `docs/DESIGN.md` without explicit
approval

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant