Repository navigation
Leave an operation or API alone when the API asks it to wait - #424
Merged
Merged
Conversation
…API once two do A reply of 429, or of 503 with Retry-After, makes the operation that drew it wait for as long as the API says, or for a wait that doubles from a second to a minute when it does not say; every other operation goes on. When a second operation is told to wait while another waits, nothing at all is sent until the whole API's wait is over, the requests the engine already holds included. report.json says who was told to wait under toldToWait, and the summary how much of the idle time the API asked for. One switch, schedule.waitWhenToldTo, on by default, and two settings for the guessed wait. ADR-0042. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…indows too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 tasks done
ssegura
added a commit
that referenced
this pull request
Oct 30, 2026
…asks, keep to a rate, and Maven Central (#431) **Related:** None. This is the release of 2.4.0, with what `master` holds on 30 October: #422 to #429. ## What you can do now that you could not before - **Get 2.4.0.** Once this is merged and tagged, it installs four ways, as 2.3.0 does: - `brew install isa-group/tap/restest`; - `docker run ghcr.io/isa-group/restest:2.4.0`; - the release's archives, with Java alone; - or built from the source. - **What 2.4.0 adds:** - a run whose opening lap the API answers with `401` throughout stops there and ends with `5` (#422); - an operation answered `429`, or `503` with `Retry-After`, is left alone for as long as the API asks (#424); - `--set engine.mostRequestsPerMinute=<n>` keeps a run to a rate, evenly spaced (#425); - the eight modules go to Maven Central as `es.us.isa.restest:restest-*`, from this release on (#429); - fixes: both reports kept when a quoted value is cut inside an emoji (#427), `F920` and `F921` reported on operations RESTest read whole (#428), and every Jackson library on a version with no known flaw (#423). - **What stays the same as in 2.3.0:** the command line, plans and dictionaries. A run can end with one exit code more, `5`, and `report.json` only gains members. ## Try it yourself From a clean clone, with only Java and git: ```bash git clone --quiet --branch release/2.4.0 https://github.com/isa-group/RESTest.git restest-2.4.0 && cd restest-2.4.0 ./mvnw -q install -DskipTests ./restest version ./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://127.0.0.1:1 --budget 3s; echo "exit $?" ls restest-out grep -o '"verdict" *: *"[A-Za-z]*"' restest-out/report.json dist/assemble.sh ../release --archives > /dev/null && ls ../release tar -xzf ../release/restest-2.4.0.tgz -C ../release && ../release/restest-2.4.0/bin/restest version ``` Below is the real output, from a fresh clone of this branch on 30 October; the Java line will be your own. Nothing listens on port 1, so the run ends with `3`, and it still writes both files: ``` RESTest 2.4.0 Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64 RESTest testing Pet Shelter at http://127.0.0.1:1 4 of 4 operations can be tested, seed -4339058821541680037, budget 3s what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent 64 requests to 4 operations in 363ms, 90% of it idle opening lap: 4 requests in 46ms, 0 of 4 operations answered 2xx 64 no reply 1 of them were pushing at the API with values nobody sensible would send 3 creation(s) meant to begin a series about a thing the run created were not accepted, so no series began no faults found report written to restest-out/report.json (9.6 KiB) page written to restest-out/report.html (90.7 KiB) the run itself was not kept; pass --store to keep every request and reply restest: nothing at http://127.0.0.1:1 answered any of the 64 requests, so the API was never actually tested. Check the address, and that it is running. exit 3 report.html report.json "verdict":"nothingAnswered" restest-2.4.0 restest-2.4.0.tgz restest-2.4.0.zip RESTest 2.4.0 Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64 ``` With Docker as well, the image is built from the same clone, as the release builds it: ```bash docker build --quiet --tag restest:2.4.0 . > /dev/null && docker run --rm restest:2.4.0 version ``` ``` RESTest 2.4.0 Java 21.0.13 (Eclipse Adoptium), Linux 7.0.14-linuxkit aarch64 ``` What the tag will leave on the Central Portal, laid out from the same clone, now with no `-SNAPSHOT` in any name: ```bash ./mvnw -q package -Pcentral -DskipTests -pl restest-cli -am && dist/central.sh target/central ls target/central/maven-central/es/us/isa/restest ls target/central/maven-central/es/us/isa/restest/restest-core/2.4.0 grep -c "<dependency>" target/central/maven-central/es/us/isa/restest/restest-cli/2.4.0/restest-cli-2.4.0.pom ls restest-cli/target/lib | wc -l ``` ``` /…/restest-2.4.0/target/central/maven-central restest-cli restest-core restest-exec restest-gen restest-oracles restest-report restest-spec restest-store restest-core-2.4.0-javadoc.jar restest-core-2.4.0-javadoc.jar.md5 restest-core-2.4.0-javadoc.jar.sha1 restest-core-2.4.0-sources.jar restest-core-2.4.0-sources.jar.md5 restest-core-2.4.0-sources.jar.sha1 restest-core-2.4.0.jar restest-core-2.4.0.jar.md5 restest-core-2.4.0.jar.sha1 restest-core-2.4.0.pom restest-core-2.4.0.pom.md5 restest-core-2.4.0.pom.sha1 70 70 ``` ## How it works **The version.** `-SNAPSHOT` is how Maven writes "on the way to this version", and this pull request removes it from every `pom.xml`. After the merge, the tag `v2.4.0` on the merge commit starts the release workflow ([`docs/ci.md`](docs/ci.md#releases)): - it builds the image and the archives, and checks them on three operating systems and two processors; - it installs the Homebrew formula for the archive on macOS and Linux; - it signs the eight modules and uploads them to the Central Portal, once a maintainer approves the `central` job; the Portal validates them and publishes nothing; - it pushes the image, and leaves a draft release for a maintainer to write the notes in and publish. A maintainer publishes the deployment on the Portal before publishing the release, and publishing the release starts the Homebrew workflow, which writes 2.4.0 into the tap. **The base images.** Step 1 of every release moves the `Dockerfile`'s two base images to the newest patch published today. This time the Java in the image moves too, from 21.0.12.1 to 21.0.13, as the output above shows, and the pages that print it say so. **The pages.** The README and chapters 2 and 9 of the manual name the version to install, and now name 2.4.0. Every other page was written for 2.4.0 as each change landed, from #422 to #429. ## What changed in the code No Java code changed, and nothing under `src/`. - **Version:** every `pom.xml`, from `2.4.0-SNAPSHOT` to `2.4.0`, ten files. The JSON Schemas' `$id` already names `v2.4.0` since #421. - **`Dockerfile`:** the digests of `eclipse-temurin:25-jdk` and `eclipse-temurin:21-jre` as they are today. - **`README.md`:** the installation section fetches, clones and runs `2.4.0`. Its `restest version` output is the image's above. - **`docs/manual/02-installing.md`:** the same for Docker, the archive, the checksums, the source and Updating. The Docker and archive outputs are those above. The two lines under Homebrew's `restest version` are what this pull request's run of `release.yml` printed in `check-homebrew` on macOS, as `docs/ci.md` asks: ``` RESTest 2.4.0 Java 21.0.12.1 (Homebrew), Mac OS X 26.6.2 aarch64 ``` They are the lines chapter 2 already had once `2.3.0` became `2.4.0`, since the formula still installs Homebrew's `openjdk@21` 21.0.12.1 on macOS; only the version changed. - **`docs/manual/09-exit-codes-and-ci.md`:** the workflow fetches and runs the 2.4.0 archive. Its sentence on releases before 2.3.0, which write no page, stays as it is. - **`docs/DESIGN.md`:** the stack table named `jackson-core` 2.22.1, which #423 moved to 2.22.3 in the root POM; it now says 2.22.3. ## Evidence - [x] **Tests:** `./mvnw verify` is green at `d19edf0f`, with version 2.4.0: 4,945 tests, 0 failures, BUILD SUCCESS in 2 min 33 s. 141 of them are the architecture and documentation rules, `PublishedSchemasTest` among them, which on a release requires the schemas' `$id` to name it. CI on `5baafe06`: [CI](https://github.com/isa-group/RESTest/actions/runs/41073186888) and [Release](https://github.com/isa-group/RESTest/actions/runs/41073186769), 19 checks green, and `central` and `publish` skipped, as on every pull request. In that Release run, `check-homebrew` installed the formula for this branch's archive on macOS and on Ubuntu: 625 MB and 823 MB, as the pages say. *Java 25 on windows-latest* failed on its first attempt in `RunLoopTest.a_slow_401_to_the_lap_stops_the_run_when_it_arrives`, which took 4.16 s where it allows less than 4 s; the same row passed when re-run, and that test passed on every row of `master`'s last runs. Nothing under `src/` changed here. - [x] **Architecture tests:** unchanged, none weakened. - [x] **Mutation score:** no oracle or constraint touched. - [x] **Per-request overhead:** unchanged; nothing under `src/` changed. - [x] **Smoke run:** unchanged, for the same reason; CI runs it on this pull request. - [x] **Idle time:** unchanged. - [x] **Artefact:** `check-restest.sh 2.4.0` passes on both the archive and the image built here, every check `ok`. - **The archive**, `--launcher` on `restest-2.4.0/bin/restest`: - version; - `JAVA_TOOL_OPTIONS`; - an unknown option answering `2`; - a run with no answer ending with `3` and its files; - the schemas and `THIRD-PARTY.txt` in the folder; - `127` with no Java and with a `JAVA_HOME` that holds none; - a symbolic link. - **The image**, `--image restest:2.4.0`: the same, and `docker stop` ending a run with `143` and a report that says it was cut short. - **What Maven Central gets:** `dist/central.sh` passes on the build above, with every check it makes, fetching each module offline with Maven among them; `restest-cli`'s POM names the 70 libraries `restest-cli/target/lib` holds, which the archive's `lib/` carries beside `restest-cli` itself. ## Decisions taken - **No release evaluation on the Web Fuzzing Dataset, as for 2.3.0.** The three safeguards change a run only against an API that answers `401` to the whole opening lap (#422), that asks to wait (#424), or when a rate is named (#425, off otherwise); #428 was measured across the dataset's 52 documents in its own pull request; #427 and #423 change no request. - **The base images are today's patch,** as step 1 of a release asks while Dependabot is off. The image's Java moves from 21.0.12.1 to 21.0.13, so the `restest version` lines of the README and chapter 2 that come from the image move with it. - **Chapter 2's Homebrew lines come from this pull request's CI,** not from a Mac here. 2.4.0 cannot be installed from the tap before it is published, and `docs/ci.md` names `check-homebrew`'s output as their source. They needed no commit of their own: what the job printed is what the page already said. - **`docs/DESIGN.md`'s `jackson-core` version is corrected here,** being one line in the release's own stack table; the rest of the table agrees with the root POM, the wrapper and the `Dockerfile`. - **The release notes are not in this pull request.** They go into the draft release after the tag, as `docs/ci.md` says. ## Open questions - **A new Dependabot alert, [48](https://github.com/isa-group/RESTest/security/dependabot/48), high, opened today:** Guava up to 33.7.1 allocates memory without bound when it deserialises some of its collections, fixed in 33.7.2. The release carries `guava-33.5.0-jre.jar`, which the OpenAPI parser and the HTTP client bring and the root POM pins. RESTest's own code uses neither Guava nor Java serialisation. Whether 2.4.0 waits for a pull request that moves Guava, as #423 did for Jackson, is the maintainer's call; this one does not touch it. - **After the merge, each step taken by a maintainer:** - the tag `v2.4.0` on the merge commit, and the approval of the `central` job when the run asks for it; - the notes in the draft; - the deployment `RESTest 2.4.0 run <run>` published on the Central Portal, and then the release, which also starts the Homebrew workflow for 2.4.0; - the README's installation commands tried from outside, `brew install isa-group/tap/restest` among them, and `https://repo1.maven.org/maven2/es/us/isa/restest/restest-cli/2.4.0/` answering; - the pull request that moves `master` to `2.5.0-SNAPSHOT`, and the schemas' `$id` to `v2.5.0`. --- - [x] Targets `master` - [x] English throughout, including comments and test names - [x] The hard rules in `AGENTS.md` hold - [x] Nothing from "Out of scope" in `docs/DESIGN.md` without explicit approval 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: None. The second of the safeguards planned for 2.4.0.
What you can do now that you could not before
An API that asks RESTest to slow down is now obeyed. When an operation answers
429 Too Many Requests, or503with aRetry-Afterheader, RESTest sends that operation nothing for as long asthe API asks, while it goes on testing the others. When a second operation is told to wait while
another still waits, the limit is plainly the whole API's, and nothing at all is sent until it is
over.
Before, a
429was an ordinary answer, and a quick one, so the engine took a stream of them as roomto send more. Against an API limited to 20 requests a second, RESTest 2.3.0 sent 1,193,708 requests
inside the quiet the API had asked for, in 20 seconds. This branch sends 277.
Try it yourself
From a clean clone, with only Java and git. A one-file Java server stands in for an API that lets 20
requests a second through, answers
429withRetry-After: 1to the rest, and counts the requeststhat arrive while the quiet it asked for lasts:
Below is the real output, from a fresh clone of this branch on 29 October 2026. The run prints its
findings first; only its summary is shown here. Two things are new. The first line ends with
(96% waiting as the API asked). The line startingtold to wait:is new too.The last line is the stand-in's. Of the 289 requests it refused, 269 arrived while its quiet lasted.
They are the requests already on their way when each of its 20 quiet seconds began.
The exit code is
1because pet-shelter accepts data its document forbids (F206), as it does onmaster; waiting changes no exit code. The3 no replyare requests the engine still held, forquiet, when the run's time ran out. They were never sent.
With the switch off, the run sends as it did before, so this run is only 5 seconds long:
(That run prints its findings too; only these lines are shown. Its
report.jsonhas"toldToWait":null.)How it works
An API that is asked too much answers
429, "too many requests", or503, "not now". It may say ina
Retry-Afterheader when to ask again: a number of seconds, or a date. What it does not say iswhether the limit covers one operation or the whole API. HTTP allows both, and the APIs of the
dataset RESTest is measured on show both kinds:
createTeleTanspends its quota of about 1,000 creations an hour within 3seconds. Its other four operations keep answering.
fetchGeneXrefsGETpasses on a third party's limit of 15 requests a second.So RESTest leaves alone the operation that was told to wait, and skips its turns. It waits for as long
as the header says, or, when the header says nothing, for 1 second, then 2, 4 and so on up to a
minute. The wait goes back to 1 second as soon as the operation gives any answer that does not ask it
to wait.
When a second operation is told to wait while another still waits, the limit is plainly the whole
API's. Then nothing at all is sent until the wait is over: neither the requests the run had not yet
handed to the engine (the part that sends) nor those already handed over. The engine then starts
again from one request at a time and climbs as it always does.
The request that drew the
429is not sent again: every turn builds a new request anyway. A503without
Retry-Afteris a server error like any other, and makes nothing wait.report.jsonsays,under
toldToWait, which operations were told to wait, how often, for how long, and how many oftheir turns were skipped. The summary says how much of the time the run sent nothing because the
API asked, so that this is never read as RESTest being slow. One switch,
schedule.waitWhenToldTo,on by default, turns it all off. The decision, the measurements and the alternatives are in
ADR-0042.
What changed in the code
restest-cliRetryAfter(new) reads the header: seconds, or a date in the three forms HTTP says a readermust accept. A date is measured from the reply's own
Dateheader when it has one.ToldToWait(new) keeps the waits, shaped like 13.1'sRefusedOpeningLap.RunLooptells it the moment each request settles, from the engine's threads, inside theregion that gives the request's slot back.
the whole API waits.
stated
Retry-Afteronly moves the end of a wait still going later.volatile"latest end" means that once every wait is over, nothing takes the lock.RunLoop:outside;
when a request of that round could not be built;
Outcomegainswaited.RunCommandbuildsToldToWait, puts its summary onRunFinished, and prints the line underthe seed while waiting is on.
restest-execConcurrencyLimiter.holdForgives no slot until the hold ends and drops the limit to the floor;waiting out a hold earns no growth.
dropWhatIsHeldlets held requests go away unsent.close()lets every request still waiting for a slot go away unsent.OkHttpEnginepasses these on. A request accepted just before the engine closes now comes backas never sent, instead of throwing on its own thread.
restest-coreHttpEnginegainsholdFor(Duration)anddropWhatIsHeld(), abstract so that no wrapper canforget them;
CredentialedEnginepasses them on.Interaction.responseHeaderValues(name)reads a reply's headers even when its body broke off.Waited(new) rides onRunEvent.RunFinishedas 13.1'sUnauthorizeddoes.ScheduleSettingsgainswaitWhenToldTo,firstGuessedWait(1s) andlongestGuessedWait(
1m).restest-gen:Scheduler.notSenthands a skipped step of a series back as one that could notbe built, so the series goes on without it or ends.
restest-reportJsonReportwritestoldToWait.ConsoleReportadds the share in brackets and thetold to wait:line.first.
fewer) and ADR-0025 (the settings).
docs/known-limitations.md,faults.md,report.md,settings.mdandswitches.md, the lastincluding "Getting the seed back", now four things. Manual chapters 5 and 10; chapter 10 shows a
real run against the stand-in.
5when the whole API keeps asking to wait, or asks past the budget;RateLimitandRateLimit-Policyfields;Evidence
Tests:
./mvnw verifyis green at the tip of this branch: 4,887 tests, 0 failures, BUILDSUCCESS. The new ones:
RetryAfterTest: each form of the header, a two-digit year, a wrong day name, the reply'sDate,values too large for any clock, and what cannot be read.
ToldToWaitTest: guessing and doubling, what brings the guess back, stale refusals, the wholeAPI's own wait, hostile headers, the switch, the summary, and a race between the thread that
answers and the thread that asks.
RunLoopTest, 9 new cases:ToldToWaitCommandTest: 5 whole runs against WireMock, read from its record of every request:ConcurrencyLimiterTestandOkHttpEngineTest: hold, extend, drop and close.SchedulerTest,InteractionTest,CredentialedEngineTest,SettingsTest,JsonReportTestandConsoleReportTest.CI: on this pull request.
Architecture tests:
DocumentedSwitchesTestnow also holds the file for getting the seedback to the waiting switch;
DocumentedReportTestandDocumentedSettingsTesthold the new keys andsettings to their pages. None weakened.
Mutation score: no oracle or constraint touched.
Per-request overhead: none to be seen. blogapi without its key, 20 s, sends 146,147
requests on
masterand 149,242 on this branch.Smoke run: CI runs it on this pull request.
Idle time: 1.9% on
master, 1.5% on this branch, for the same blogapi run. Nothing waitedthere.
Artefact. Every stored report of the Web Fuzzing Dataset's 52 APIs, run locally, was
scanned first:
429, each for one operation only, with noRetry-After;503, none withRetry-After.Replayed apart from RESTest's code over runs stored with 2.3.0, 60 s each, with a first wait of
1 s doubling to 60 s:
Then built, and run against
master, same seed:mastercreateTeleTan; 57,728 of its turns skippedfetchGeneXrefsGETtold to wait 4 timesgenome-nexus calls a service of its own over the internet, and its runs vary that much with nothing
changed. jasper's 293 replies of
503and commafeed's 421, from runs of 20 s with every requestkept, carry no
Retry-After, and nothing waited there.Decisions taken
429. The data above showed that this would leave cwa-verification waiting 57 of its 60 seconds,for a quota of one operation. The maintainer chose the operation, with the whole API waiting once a
second operation is told while another waits.
5when the whole API keeps asking to wait, andthe
RateLimitfields, are left as possible extensions (maintainer). None of the dataset's APIsmakes the whole API wait or sends those fields.
503waits only withRetry-After. This was asked and confirmed by the maintainer. If every503waited, jasper's four broken operations, which answer503together, would make the whole APIwait, and its 4,807 requests and 1,674 replies of 2xx would fall to 159 and 52. Those
503s arefaults to find, not requests to slow down.
only on a 2xx was built and measured: cwa-verification's
createTeleTananswers400to inventedvalues before it counts its quota, so with the rule chosen it gets 12 short waits and 62 refusals a
minute, against 6 and 35. A 2xx depends on the invented values too, though, and an operation for
which a valid request is rarely built would otherwise wait the longest guess after every
429.nothing a new turn does not. EvoMaster resends because it sends one request at a time.
that otherwise up to sixteen queued requests would reach the API inside its quiet.
They are now dropped when the run stops sending, and counted under
noReply, which the docs say.limitations say so.
Before, such a request went out after the run was over, or failed on its own thread and counted as
lost.
schedule.*, beside 13.1's stop, because the loop decides the wait.turns it off, and the line under the seed speaks while it is on.
Interaction.responseHeaderValues(name)rather thanresponseHeaders(), reusing the package'slookup without regard to case.
toldToWait.idleat the top of the block, since a round where every operation waits is theAPI's asking too.
step) is not counted as the API's. It is bounded by the lap's patience.
built: the generator always ends up building a request it can send, so no document produces that
round on demand. The fix is two lines and is in the code.
Open questions
None.
masterAGENTS.mdholddocs/DESIGN.mdwithout explicit approval🤖 Generated with Claude Code