diff --git a/docs/adr/0009-non-blocking-engine.md b/docs/adr/0009-non-blocking-engine.md index 791a66b38..b8b625670 100644 --- a/docs/adr/0009-non-blocking-engine.md +++ b/docs/adr/0009-non-blocking-engine.md @@ -1,6 +1,6 @@ # ADR-0009: The engine never blocks on computation; idle time replaces a throughput floor -**Status:** Accepted +**Status:** Accepted, amended on 29 October 2026 **Date:** 2026-09-11 ## Context @@ -55,3 +55,12 @@ says nothing about real APIs and does not pretend to. - **A fixed requests-per-minute floor.** Rejected: assumes knowledge of the API's latency. - **Measuring wall-clock throughput only.** Confounds our speed with the API's. - **Doing all expensive work up front, before testing.** Exactly what failed in 2026. + +## Amendment (29 October 2026): idle time the API asked for + +An API can ask a run to send nothing for a while — a 429, or a 503 with `Retry-After` — and a run now +obeys it ([ADR-0042](0042-wait-when-told-to-wait.md)). Time spent that way is idle time: nothing is +in flight. It is not the tool being slow, which is what idle time exists to catch, so a run says how +much of its idle time came while it waited as the API asked, in `report.json` +(`toldToWait.idle`) and in brackets after the idle share on the first line of its summary. The +tool's own idleness is the rest. diff --git a/docs/adr/0016-how-a-fault-is-presented.md b/docs/adr/0016-how-a-fault-is-presented.md index 17fc5764d..3743dc811 100644 --- a/docs/adr/0016-how-a-fault-is-presented.md +++ b/docs/adr/0016-how-a-fault-is-presented.md @@ -1,6 +1,6 @@ # ADR-0016: A fault is identified by its catalogue number, described in our own words, and classified twice -**Status:** Accepted, amended at M1.10 and on 10, 12, 15, 17 and 22 October 2026 +**Status:** Accepted, amended at M1.10 and on 10, 12, 15, 17, 22 and 29 October 2026 **Date:** 2026-09-15 ## Context @@ -467,3 +467,14 @@ It is what decides how a run ends, and every row of `report.json` that counts fa ([ADR-0037](0037-errors-and-warnings.md)). A reply that is both an error and a warning — a request broken on purpose, accepted with a status code the document does not declare — is one of each, as it was two faults before. + +## Amendment (29 October 2026): a reply that asks the run to wait, now that it waits + +The amendment of 15 October said a 503 that carries `Retry-After` is reported like any other, and +that until RESTest waited when told to, a 503 its own pace provoked would be reported too. It now +waits ([ADR-0042](0042-wait-when-told-to-wait.md)): the operation that drew a 429, or a 503 with +`Retry-After`, is left alone for as long as the API asks. Nothing changes in how such a reply is +reported. A 503 is still `F121`, and a 429 the document does not declare still `F920`: the API did +answer it, and the document did leave it out. There are few such replies once the run waits — one +for each wait, and those already on their way when it began — rather than one for every request the +run's pace provoked. diff --git a/docs/adr/0025-settings.md b/docs/adr/0025-settings.md index ab63d4781..4da7558b0 100644 --- a/docs/adr/0025-settings.md +++ b/docs/adr/0025-settings.md @@ -1,6 +1,6 @@ # ADR-0025: The numbers somebody decided live in one settings object, layered from four sources, and never in the plan -**Status:** Accepted, amended at M11.1, M9.1, M10.1, M10.3, M8.5, M11.2, M12.1b, M9.5, M9.6, M9.7, M10.4-M10.7 and M9.8-M9.9, at 2.1, and on 1, 12, 13, 14, 15, 17, 22, 25 and 28 October 2026 +**Status:** Accepted, amended at M11.1, M9.1, M10.1, M10.3, M8.5, M11.2, M12.1b, M9.5, M9.6, M9.7, M10.4-M10.7 and M9.8-M9.9, at 2.1, and on 1, 12, 13, 14, 15, 17, 22, 25, 28 and 29 October 2026 **Date:** 2026-09-22 ## Context @@ -797,3 +797,19 @@ also decides when a run ends rather than what it sends switches that are not levers, and the page of switches has a file that turns it off, since it is on by default. The rule has no number to put beside it: the stretch it judges is the opening lap, and 401 is a fact of HTTP rather than a choice. One hundred and two settings in nine groups. + +## Amendment (29 October 2026): a switch and two numbers for an API that asks the run to wait + +`schedule.waitWhenToldTo`, on by default, with `schedule.firstGuessedWait` (`1s`) and +`schedule.longestGuessedWait` (`1m`) beside it ([ADR-0042](0042-wait-when-told-to-wait.md)). An +operation the API answers 429, or 503 with `Retry-After`, is left alone for as long as the API says, +or, when it does not say, for the first of the two numbers, doubled each time it is told again up to +the second; the whole API once a second operation is told while another waits. They are in +`schedule.*`, beside the stop of 28 October, because the loop that decides what to send decides the +wait; `engine.*` holds what the HTTP engine itself reads. The switch is listed among the switches that +are not levers: it is a courtesy to the API rather than a way of finding more, so it plays no part in +an ablation, and it changes what is sent only against an API that asks to wait. Since which turns it +skips depends on when the API asked, the file for getting the seed back turns it off, and a run with +it on says under its seed that the seed alone does not repeat it. A first wait of no time is refused, +since an API that does not say how long would be asked again at once, and so is a ceiling below the +first wait. One hundred and five settings in nine groups. diff --git a/docs/adr/0042-wait-when-told-to-wait.md b/docs/adr/0042-wait-when-told-to-wait.md new file mode 100644 index 000000000..663024dc2 --- /dev/null +++ b/docs/adr/0042-wait-when-told-to-wait.md @@ -0,0 +1,263 @@ +# ADR-0042: An operation the API asks to be left alone is left alone, and the whole API once a second one is + +**Status:** Accepted +**Date:** 2026-10-29 + +## Context + +An API that is being asked too much can say so. It answers 429, "too many requests", or 503, "not +now", and it may say in a `Retry-After` header when to ask again: either as a number of seconds, +`Retry-After: 30`, or as a moment, `Retry-After: Thu, 29 Oct 2026 10:00:00 GMT`. + +Until now RESTest heard such a reply as an ordinary answer. It did not read the header and nothing +paused. Worse, a 429 is usually a quick answer, and the engine's limiter judges only how long answers +take, so a stream of 429s read to it as room to send *more*. A testing tool pointed at somebody +else's API should obey an API that asks it to slow down. + +What the dataset of local APIs shows (the Web Fuzzing Dataset's APIs, run on one machine). Every +stored report of its 52 APIs was read, from two campaigns: + +- **Two APIs ever answer 429, and each for one operation only, with no `Retry-After`.** + - cwa-verification's `createTeleTan` has a quota of its own, about 1,000 creations an hour. A + 60-second run spends it at 2.6 s, and the operation answers 429 to 36,440 requests after that, + with an empty body. The API's other four operations go on answering all the while: 4,171 times + 201 and 4,102 times 200 after that moment. + - genome-nexus's `fetchGeneXrefsGET` passes on the limit of a service it calls, 27 times in 60 s: + `You have exceeded the limit of 15 requests per second; please reduce your concurrent + connections`. Its other 22 operations answer 200 and 404. +- **Three APIs answer 503** — commafeed, jasper and spring-ecommerce — and none of the 503s their + reports kept carries `Retry-After`. They are server errors, not requests to wait. + +HTTP lets a server count requests per resource or for the whole server, and a 429 does not say +which ([RFC 6585](https://www.rfc-editor.org/rfc/rfc6585#section-4)). + +Among the related tools, waiting after a 429 is common, and slowing down afterwards is not. EvoMaster +waits for `Retry-After`, or ten seconds without it, and sends the request again. RestTestGen and +AutoRestTest do the same a bounded number of times, and Schemathesis does only when asked. RESTler +sends it again every five seconds without reading the header, and WuppieFuzz only logs it. Each of +them that waits makes the whole API wait, whichever operation was refused; none returns to a slower +pace afterwards, none waits on a 503 by default, and none reads the IETF draft's `RateLimit` fields. + +## Decision + +### 1. What asks the run to wait, and for how long + +- **The replies.** A reply of 429, with or without `Retry-After`. A reply of 503 that carries a + `Retry-After` the run can read; a 503 without one is a server error like any other. A reply whose + body broke off counts the same, since its headers came with its status line. +- **Stated.** `Retry-After` as seconds, or as a moment in any of the three forms + [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110#section-5.6.7) says a reader must accept: + - the current one, `Sun, 06 Nov 1994 08:49:37 GMT`; + - the RFC 850 one, with a two-digit year read as RFC 9110 says, so that `94` is 1994 and not 2094; + - C's `asctime`, `Sun Nov 6 08:49:37 1994`. + + A moment is measured from the reply's own `Date` header when it has one that can be read, so that + a clock out of step on either side changes nothing; otherwise from the run's clock. A moment + already past, or `0`, is a wait of no time at all. Of several `Retry-After` headers, the first that + can be read counts. +- **Guessed.** When the reply gives no length that can be read, the operation waits + `schedule.firstGuessedWait`, one second. Each time the same operation is given a guessed wait again + it doubles, up to `schedule.longestGuessedWait`, a minute. + - A second is the shortest wait that clears a limit counted per second, as genome-nexus's is. + - A minute clears a limit counted per minute in one wait, and an operation whose quota is spent + for the hour, as cwa-verification's is, is still tried once a minute. + - The guess goes back to one second as soon as the operation gives any answer that does not ask + it to wait — a 2xx, a 400, a 404, a 500, a 503 without `Retry-After` — to a request sent after + its last wait ended. Decided by the maintainer on 29 October 2026, after going back only on a + success had been built and measured (see the alternatives). + - A request that got no reply resets nothing, and a stated wait leaves the guess where it was. +- **Nothing an API writes can break a run.** Every end is cut at the run's deadline before anything + is worked out on the clock. A wait longer than the run has left is the whole of what it has left; + a number of seconds too large for any clock is too. + +### 2. Who waits: the operation, and the whole API once a second one is + +The operation whose request drew the reply has its turns skipped until its wait ends: nothing is +built for it and nothing is sent. Every other operation goes on. That is what both APIs above need: +one operation is limited, the rest are not. Decided by the maintainer on 29 October 2026, from the +replay in the evidence below. + +When an operation is told to wait while another operation still waits, the limit is plainly the whole +API's. Then every request waits, both kinds: + +- those the run had not yet handed to the part that sends; +- those already handed over and not yet sent, which the engine holds back. + +The engine then starts again from its fewest requests in flight (`engine.minConcurrency`) and climbs +as it already does. Requests already in flight come back as usual. The thread that hears the second +refusal tells the engine at once, `HttpEngine.holdFor`, rather than leaving it to the loop that +decides what to send, which might be waiting for room to send and notice only later. + +The whole API has a guessed wait of its own, which doubles in the same way, so that an API limited as +a whole and saying nothing is asked a little more slowly each time, rather than once a second for the +whole run. It goes back to the first guess as an operation's does: after any answer that does not ask +to wait, to a request sent once its last wait was over. + +When the run stops sending while the whole API still waits — its time is up, or it was stopped — what +the engine holds is never sent: each such request comes back as one that never went out +(`HttpEngine.dropWhatIsHeld`), and is counted with the requests that got no reply. A wait that lasts +to the end of the run holds the engine beyond it, so that nothing reaches the API after the run's time +is up. + +### 3. A refusal of a request sent before the wait began + +Requests go out several at a time and come back in any order, so a refusal is weighed by when its +request went out. One sent before the wait it belongs to was over had not heard of that wait. Its +refusal begins no wait and doubles nothing; a moment it states only moves the end of a wait still +going, and only later. Any other answer of such a request does not bring the guess back to the start +either. + +### 4. The refused request is not sent again + +It is recorded as it came back, and judged by the rules like any reply: + +- a 429 the document does not declare is a warning, `F920`; +- a 503 is a fault, `F121`. + +The operation's next turn after the wait builds a new request, as every turn does. Resending would buy +nothing a new turn does not: every request a run sends is new. EvoMaster resends because it sends one +request at a time, and the refused one is all it had. A step of a series about a thing the run +created, whose operation is waiting, is handed back to its series as one that could not be built. +The series then goes on without it, or ends, and never repeats a step that never went out. + +### 5. A round with nothing to send + +A round over the operations can send nothing because each of them was waiting or could not be built. +It is then not gone round again at once, which would only skip them all again as fast as the loop can +turn. The loop sits it out until the soonest wait is over, the deadline, or a stop from outside. When +a request of that round could not be built, it looks again after a twentieth of a second instead, +since a request that could not be built once may be built the next time. Such a round is also not +one where "nothing in the document could be sent", which ends a run. An API of one operation that is +told to wait is the commonest case. + +### 6. What a run says + +- **`report.json`**, `toldToWait`, beside `unauthorized`: + - `idle`: how much of the run's idle time came while it waited as the API asked — the whole API + waiting, or every operation it could send waiting; + - `wholeApi`: how often and how long nothing at all was sent; + - `operations`: each operation told to wait, with how many waits, how many of them the API stated, + how long they lasted and how many of its turns were skipped. + + Lengths are counted up to the moment the run stopped sending, and waits that began after it are not + counted. A run that waits when told to and never was writes zeros and no operation. With the switch + off, it is `null`. +- **The summary.** + - The idle share on its first line is followed by how much of it the API asked for, `40% of it idle + (38% waiting as the API asked)`, so that quiet the API asked for is not read as the tool's own + slowness ([ADR-0009](0009-non-blocking-engine.md)'s amendment). + - A line under the replies, `told to wait: …`, names the whole API and up to three operations. +- **The exit code** does not change. A run told to wait ends with the number its findings give. + +### 7. One switch, two numbers, and the seed + +`schedule.waitWhenToldTo`, on by default, with `schedule.firstGuessedWait` and +`schedule.longestGuessedWait` beside it ([ADR-0025](0025-settings.md)'s amendment). They are in +`schedule.*` because the loop that decides what to send decides the wait; `engine.*` holds what the +HTTP engine itself reads. + +The switch is listed among those that are not levers. It is a courtesy to the API rather than a way of +finding more, so it plays no part in an ablation. It changes what is sent only against an API that +asks to wait. + +It makes what a run sends depend on the API's replies, though: which turns are skipped depends on +when the API asked. So, while it is on, the line under the seed says the seed alone does not repeat +the run, and the switches page's file for getting the seed back turns it off. + +## The evidence + +Replayed before it was built, apart from RESTest's code, over runs of the two APIs stored with +RESTest 2.3.0, 60 s each. The replay keeps each request's recorded reply. That suits a quota that +stays spent, and overstates the waits for a limit per second. Each rule was replayed with a first +wait of 1 s doubling to 60 s. + +| | Every request to the API waits | Only the operation told to wait waits | +|---|---|---| +| cwa-verification | 57 s of 60 waiting; 2xx replies from 9,275 to 1,002 | no 2xx lost; 54,260 requests that would have drawn a 429 not sent | +| genome-nexus | 27 s waiting | no 2xx lost; 76 requests not sent | + +Then built, and run against `master` (which carries +[ADR-0041](0041-stop-when-the-api-refuses-the-whole-opening-lap.md)), with the same seed: + +| Run | `master` | This change | +|---|---|---| +| cwa-verification, 60 s: replies of 429 | 34,699 | 62, in 12 waits of `createTeleTan`, its 400s to the values RESTest made up bringing the guess back to a second; 57,728 of its turns skipped | +| cwa-verification, 60 s: 2xx replies, operations answering 2xx | 8,756, 3 of 5 | 9,503, 3 of 5 | +| cwa-verification, 60 s, with `schedule.waitWhenToldTo=false` | — | 34,931 replies of 429, 8,778 2xx, `toldToWait` `null` | +| genome-nexus, 60 s, two runs of `master` and one of this change: replies of 429 | 5 and 1 | 6; `fetchGeneXrefsGET` told to wait 4 times, 32 of its turns skipped | +| genome-nexus, the same runs: 2xx replies, operations answering 2xx | 2,735 and 1,766, 17 and 16 of 23 | 2,214, 17 of 23 | +| blogapi without its key, 20 s: requests, idle | 146,147, 1.9% | 149,242, 1.5% | +| A stand-in limiting the whole API to 20 requests a second, 20 s: requests that reached it inside the quiet it asked for | 1,193,708 | 277, over 20 waits of the whole API | + +genome-nexus calls a service of its own over the internet, and its runs vary that much with nothing +changed: its two runs of `master` alone differ by 969 replies of 2xx, and three runs of earlier +versions of this change gave 2,817, 2,355 and 3,128. An operation that answered 2xx on `master` and +not on one of those runs, `fetchPdbHeaderGET`, was never told to wait. jasper's 293 replies of 503 +and commafeed's 421, from runs of 20 s with every request kept, carry no `Retry-After`, and no +operation of either waited. + +## Alternatives considered + +- **Every request to the API waits, at the first 429.** This is what the plan's row first said, and + what every related tool that waits does. Against both APIs that answer + 429 it gives the table's first column. +- **Every 503 waits, as a 429 does, with or without `Retry-After`.** A 503 says the API cannot answer + now, most often because something behind it broke, while a 429 says the caller is going too fast. + Replayed over the runs of jasper and commafeed with every request kept, 20 s each: jasper's four + broken operations answer 503 together, so the whole API would wait, and its 4,807 requests and + 1,674 replies of 2xx would fall to 159 and 52; commafeed would send 459 fewer requests to the one + operation that answers 503 when it cannot read the feed it was given. Those 503s are faults to + find, not requests to slow down. Kept as the plan's row said, and confirmed by the maintainer on + 29 October 2026: a 503 waits only when it says for how long. +- **The guess goes back to the start only on a success, 2xx.** A 400 or a 404 is about the values + RESTest made up, and an API may check those before it counts its quota, as cwa-verification's + `createTeleTan` does: each of its 400s starts the guess again, so that operation is asked again + every second or two. Built and measured, going back only on a success gave it 6 waits that double + and 35 refusals in a minute, against 12 waits and 62 refusals. The maintainer chose to go back on + any answer that does not ask to wait: a success depends on the values RESTest made up too, and an + operation for which a valid request is rarely built would otherwise wait the longest guess after + every 429. +- **The scheduler skips the operations that wait**, rather than the loop skipping what the scheduler + proposes. It would keep "the scheduler decides what comes next" true to the letter. But it costs a + new kind of step and a change to every place that reads one, for what the loop does in one check. +- **Generating the request of a skipped turn and throwing it away**, so that every later request + keeps the values its seed gives. It would cost the generation of every skipped request, 54,260 in + cwa-verification's minute. It would also not make a run that skips turns repeatable, since which + turns are skipped depends on when the API asked. +- **A fixed wait without the header**, as EvoMaster's ten seconds. It is too long for a limit per + second and too short for an hourly quota, which a doubling wait finds either way. +- **Releasing what the engine already holds during a wait of the whole API, and saying so.** It would + be simpler, but the requests already handed over and not in flight — up to + `schedule.workAheadFactor` × `engine.maxConcurrency` less those in flight, sixteen by default — + would reach an API inside the quiet it asked for, and the limiter would climb back while they went. + +## Consequences + +- **An operation answered 429, or 503 with `Retry-After`, is sent nothing for as long as the API asks, + and every other operation goes on.** On cwa-verification, the 34,699 429s of a minute become 62. +- **The whole API waits once a second operation is told to.** A limit over the whole API therefore + costs a refusal from each of the first two operations before it is known. +- **The requests for an operation already handed to the engine when it is told to wait still go out.** + There are at most `schedule.workAheadFactor` × `engine.maxConcurrency` of them, and in practice that + operation's share of them. The known limitations say so. +- **`report.json` gains `toldToWait`,** and the summary a line. +- **`HttpEngine` gains `holdFor` and `dropWhatIsHeld`.** On closing, the engine no longer sends a + request it was handed and had not yet sent: such a request comes back as one that never went out. + A run that stops sending while the whole API waits has what is held dropped the same way, and + those requests are counted with the ones that got no reply. +- **A run with the switch on depends on the API's replies,** and says so under its seed. + +## Still open + +- **13.2b, stopping a run that the whole API keeps asking to wait,** or that it asks to wait past the + end of its budget, with `5` and a verdict word of its own. None of the dataset's APIs asks its whole + API to wait. +- **13.2c, the IETF draft's `RateLimit` and `RateLimit-Policy` fields** (draft-ietf-httpapi- + ratelimit-headers-11, May 2026), which would let a run stay inside an API's quota rather than find + it by going over. None of the dataset's APIs sends them. +- **The page.** It shows the idle share as before. A section for the waits is a change of layout, + which needs a mock-up first. +- **An API that answers 429 to too many failed sign-ins for one account.** That is a refusal that + depends on the values RESTest invented — a wrong password — and a rule that reacts to an answer + should not count such a refusal. Nothing here tells it apart: the sign-in operation is left alone + for a while, as any other. diff --git a/docs/adr/README.md b/docs/adr/README.md index 0479d5ad5..b09481aa6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -61,14 +61,14 @@ so records written from now on cite none. | [0006](0006-event-stream-and-store.md) | One event stream; every interaction persisted | Accepted, amended at M1.4, M1.6, M1.7, M11.1, M9.1 and M11.3, and in #325, #327 and #334, and on 10, 19 and 25 October 2026 | | [0007](0007-specification-parser-boundary.md) | The parser sits behind our own interface; OAS 2.0, 3.0.x and 3.1.x | Accepted, amended at M0.2, in #303 and in #328, reversed at M1.2 | | [0008](0008-extension-points-no-ai-abstractions.md) | Extension points, and no AI-specific abstractions | Accepted | -| [0009](0009-non-blocking-engine.md) | The engine never blocks; idle time replaces a throughput floor | Accepted | +| [0009](0009-non-blocking-engine.md) | The engine never blocks; idle time replaces a throughput floor | Accepted, amended on 29 October 2026 | | [0010](0010-idl-strategy.md) | IDL: relicensed assets, ANTLR4 parser, solver behind an interface | Accepted, not yet built (14 October 2026) | | [0011](0011-evaluation-harness.md) | Evaluation harness in `evaluation/`, outside the Maven build — amended at M1.9: in a repository of its own, so `evaluation/` was never created | Accepted, amended at M1.9 and on 9 October 2026 | | [0012](0012-canonical-model.md) | The canonical model: immutable records, sealed types, recorded gaps | Accepted, amended at M1.8, M2.1a and M2.1b, and on 17 October 2026 | | [0013](0013-input-generation.md) | Where input values come from, and how a campaign is put together | Accepted, amended at M1.11, M2.7a, M2.4, M2.10a, M9.1, M9.3, M10.1 and M10.3, and on 10, 12, 13 and 25 October 2026 | | [0014](0014-response-conformance.md) | A reply is judged against the specification document itself | Accepted, amended at M1.8 and on 10, 17, 19 and 22 October 2026 | | [0015](0015-command-line-contract.md) | One command, a budget spent in full, and what an exit code means | Accepted, amended at M1.7, M1.8, M2.7a, M2.10a, M11.1, M9.1, M11.3, M12.1a, M12.1c and M12.1b, in #314 and #344, and on 1, 11, 12, 19, 22, 25 and 28 October 2026 | -| [0016](0016-how-a-fault-is-presented.md) | A fault: catalogue number as identity, our words as description, classified twice | Accepted, amended at M1.10 and on 10, 12, 15, 17 and 22 October 2026 | +| [0016](0016-how-a-fault-is-presented.md) | A fault: catalogue number as identity, our words as description, classified twice | Accepted, amended at M1.10 and on 10, 12, 15, 17, 22 and 29 October 2026 | | [0017](0017-what-we-take-from-autoresttest.md) | What we take from the tool that won the 2026 competition, and what we refuse | Proposed, amended at M9.1, M9.4 and M10.1 | | [0018](0018-a-choice-between-shapes.md) | A choice between shapes is one more shape | Accepted, amended on 10 October 2026 | | [0019](0019-the-samples-a-document-writes-down.md) | A document's own sample values are read, sent as written, and named in the record | Accepted | @@ -77,7 +77,7 @@ so records written from now on cite none. | [0022](0022-the-characters-a-value-is-made-of.md) | The characters a value is made of are read where its length already is, and the regular expressions are somebody else's | Accepted, amended at M9.6, at 2.1, and on 12 October 2026 | | [0023](0023-the-campaign-file.md) | A plan is an ordered list of named sources, and the order is the preference | Accepted, amended at M9.1, M10.1 and M10.3, and on 1, 12 and 13 October 2026; its §5 superseded by ADR-0034 | | [0024](0024-the-competition-version.md) | v2.0 is the version submitted to the 2027 competition, and what that leaves for 2.1 | Accepted, amended on 3, 4, 5, 11 and 12 October 2026 | -| [0025](0025-settings.md) | The numbers somebody decided live in one settings object, layered from four sources, and never in the plan | Accepted, amended at M11.1, M9.1, M10.1, M10.3, M8.5, M11.2, M12.1b, M9.5, M9.6, M9.7, M10.4-M10.7 and M9.8-M9.9, at 2.1, and on 1, 12, 13, 14, 15, 17, 22, 25 and 28 October 2026 | +| [0025](0025-settings.md) | The numbers somebody decided live in one settings object, layered from four sources, and never in the plan | Accepted, amended at M11.1, M9.1, M10.1, M10.3, M8.5, M11.2, M12.1b, M9.5, M9.6, M9.7, M10.4-M10.7 and M9.8-M9.9, at 2.1, and on 1, 12, 13, 14, 15, 17, 22, 25, 28 and 29 October 2026 | | [0026](0026-what-a-run-sends-first.md) | A run opens with every operation once, in five steps, and one part of the tool owns the clock | Accepted, amended at M10.3 and on 28 October 2026 | | [0027](0027-changing-one-thing-in-an-accepted-request.md) | A request the API accepted is sent again with one thing changed, and says what it broke | Accepted, amended at M10.2, M8.5 and M10.4-M10.7, and on 12, 13, 15 and 17 October 2026 | | [0028](0028-sequences-over-things-a-run-creates.md) | A creation may start a short series about the thing it made, and each series asks one question | Accepted, amended at M11.2 and on 12 and 13 October 2026 | @@ -94,3 +94,4 @@ so records written from now on cite none. | [0039](0039-the-html-report.md) | The HTML report: one page beside report.json, drawn in the browser from the whole of it, fetching nothing | Accepted | | [0040](0040-homebrew.md) | RESTest is installed with Homebrew from a tap of its own, by a formula that brings its Java and is written into the tap once a release is published | Accepted | | [0041](0041-stop-when-the-api-refuses-the-whole-opening-lap.md) | A run stops sending when the API answers 401 to every request of its opening lap, and ends with `5` | Accepted | +| [0042](0042-wait-when-told-to-wait.md) | An operation the API asks to be left alone is left alone, and the whole API once a second one is | Accepted | diff --git a/docs/faults.md b/docs/faults.md index 5911acd92..016b03a78 100644 --- a/docs/faults.md +++ b/docs/faults.md @@ -127,8 +127,9 @@ one that fell over. this check such replies are only counted, under `serverErrors`. Where the surroundings are known to be unwell, switch the check off in one line (`checks.otherServerError`). - **A 503 that asks the run to wait is reported too.** An API that cannot keep up may answer 503 - with a `Retry-After` header; RESTest does not wait yet ([known - limitations](known-limitations.md#pointed-at-somebody-elses-api)), so each such reply is a fault. + with a `Retry-After` header. RESTest then leaves the operation alone for that long + ([`toldToWait`](report.md#toldtowait)), so there are few such replies, and each is a fault like + any other 503. - **501 is `F205`'s**, as for `F100`, and a code of 600 or more is left alone, since it is no server error whatever its first digit. @@ -331,9 +332,10 @@ and `default` included. - A reply to a request whose body RESTest sent, on purpose, under a media type the operation does not accept: a document is not expected to describe how an API refuses a body it never promised to take. -- **A 429 that asks the run to slow down is reported too**, unless the document declares it: RESTest - does not wait yet ([known limitations](known-limitations.md#pointed-at-somebody-elses-api)), so each - such reply is a fault. +- **A 429 that asks the run to slow down is reported too**, unless the document declares it. RESTest + then leaves the operation alone for as long as the API asks ([`toldToWait`](report.md#toldtowait)), + so there are few such replies; each one the document leaves out is still a reply it does not + declare. - **Too many to read?** Declare the replies the API really gives, or switch the check off in one line (`checks.undeclaredStatusCode`). diff --git a/docs/known-limitations.md b/docs/known-limitations.md index cf164fa08..cee8ff47e 100644 --- a/docs/known-limitations.md +++ b/docs/known-limitations.md @@ -6,13 +6,27 @@ current release: an entry leaves it in the release that lifts it, and that relea ## Pointed at somebody else's API -- **A run does not wait when the API asks it to.** A reply of 429, or of 503 with `Retry-After`, is - an ordinary answer: the header is not read and nothing pauses. A 429 is usually a quick answer, and - the engine's limiter judges only how long answers take, so a stream of 429s reads to it as room to - send more. Against an API that limits its callers, lower `engine.maxConcurrency` - ([the settings](settings.md)) before you start. A 503 the run's own pace provoked is reported as a - fault, `F121`, like any other; `checks.otherServerError=false` stops that - ([the faults](faults.md#f121-other-server-error)). +- **A run waits when the API asks it to, but never stops for it.** An operation answered 429, or 503 + with `Retry-After`, is sent nothing for as long as the API asks, and the whole API when a second + operation is told to wait while another waits ([`toldToWait`](report.md#toldtowait)). An API that + asks for a wait longer than the run has left is obeyed until the run's time is up, and the run then + ends with the number its findings give, not with one that says the API kept it waiting. What is + not done yet: + - the `RateLimit` and `RateLimit-Policy` fields an API may send to say how much of its quota is + left are not read, nor the `X-RateLimit-*` headers some APIs use: only `Retry-After` is; + - the request the API turned away is not sent again; the operation's next turn builds a new one; + - the requests for an operation already on their way when it is told to wait still go out, at most + `schedule.workAheadFactor` × `engine.maxConcurrency` of them; a wait of the whole API holds back + every one of them, and those it still holds when the run stops sending are never sent, and are + counted with the requests that got no reply; + - a limit over the whole API is only known as one once a second operation is told to wait, so each + of the first two operations draws a refusal of its own. + + Each 429 the document does not declare is still a warning, `F920`, and each 503 a fault, `F121`, + like any other reply — there are few of them once the run waits. `schedule.waitWhenToldTo=false` + sends to such an API as RESTest did before it waited; against an API that limits its callers, a + lower `engine.maxConcurrency` ([the settings](settings.md)) still keeps the run further from its + limit. - **There is no ceiling on requests per second.** What bounds a run is the number of requests in flight, sixteen at most by default (`engine.maxConcurrency`). The limiter halves it whenever a request goes unanswered and sends fewer as answers slow down, but no setting promises an API's diff --git a/docs/manual/05-plans.md b/docs/manual/05-plans.md index f8d25d1ce..dcc6fc777 100644 --- a/docs/manual/05-plans.md +++ b/docs/manual/05-plans.md @@ -98,10 +98,11 @@ others. ### A run that the seed repeats A run prints its **seed**, the number every choice it makes by chance comes from. `--seed` with the -same number makes the same choices. But three things in the plan RESTest carries make a run depend on -the API's answers as well — the `observed` source, the changes to accepted requests and the series — -so with them the seed makes a similar run rather than the same one. [Getting the seed -back](../switches.md#getting-the-seed-back) has the plan and the settings that leave all three out, +same number makes the same choices. But four things make a run depend on the API's answers as well — +the `observed` source, the changes to accepted requests and the series, in the plan RESTest carries, +and leaving alone an operation the API asks to be left alone (`schedule.waitWhenToldTo`) — so with +them the seed makes a similar run rather than the same one. [Getting the seed +back](../switches.md#getting-the-seed-back) has the plan and the settings that leave all four out, and with them `--seed` repeats a run exactly. `--store`, which keeps the run you actually had, is usually the better record. diff --git a/docs/manual/10-troubleshooting.md b/docs/manual/10-troubleshooting.md index 32de27120..55153eff5 100644 --- a/docs/manual/10-troubleshooting.md +++ b/docs/manual/10-troubleshooting.md @@ -211,6 +211,94 @@ as a fault, `F121`, which a run whose pace is to blame can stop with `--set checks.noReplyInTime=false` stops. An API that is simply slow gets longer to answer with `--set engine.readTimeout=2m`. [Chapter 7](07-settings.md) has more. +## The API asks the run to wait + +The summary's first line gives, after the idle share, how much of it was `waiting as the API asked`, +and a line under the replies starts `told to wait:`. The API answered some requests with `429 Too +Many Requests`, or with `503` and a `Retry-After` header, which is how an API asks a caller to slow +down. RESTest then sends nothing to that operation for as long as the header says, or, when it says +nothing, for a second that doubles each time the operation is told again, up to a minute; the other +operations go on. When a second operation is told to wait while another still waits, the limit is +the whole API's, and nothing at all is sent until the wait is over. The pet shop of [chapter +8](08-an-api-that-asks-for-a-key.md), run for 20 seconds against a stand-in that lets 20 requests a +second through and asks for a second's quiet after that (below), + +```bash +restest run https://petstore3.swagger.io/api/v3/openapi.json --url http://localhost:8429 --budget 20s +``` + +begins its summary like this: + +``` +541 requests to 19 operations in 20.0s, 99% of it idle (91% waiting as the API asked) + opening lap: 19 requests in 181ms, 19 of 19 operations answered 2xx + 381 2xx, 148 4xx, 12 no reply + 46 of them were pushing at the API with values nobody sensible would send, which accounts for some of the 148 refusals above + 86 of them changed one thing in a request the API had accepted + 38 of them were steps of 23 series about a thing the run created: createTwice 30, putTwice 2, deleteTwice 3, readAfterDelete 2, safeGet 1; 8 creation(s) meant to begin one were not accepted + told to wait: the whole API 19 times, 18.1s in all; addPet 8 times, 7.1s in all, 0 turns skipped; findPetsByStatus 7 times, 6.1s in all, 0 turns skipped; findPetsByTags 7 times, 7.0s in all, 0 turns skipped; and 16 more; the run's report names every one +``` + +Nearly all of the run was spent waiting, as the API asked, and the idle share says so: the brackets +are the API's doing, not the tool's. The `12 no reply` are requests RESTest had already handed over +when the last wait began, held back and never sent once the run's time was up. `report.json` names every operation under +[`toldToWait`](../report.md#toldtowait). Each 429 the document does not declare is still a warning, +*Undeclared status code* (`F920`), and each 503 an error (`F121`); there are few of them once the run +waits. Against a quota that lasts longer than the run — so many creations an hour — that operation is +simply tried again once a minute while the others are tested. To send to such an API as RESTest did +before it waited, turn the waiting off: + +```bash +restest run openapi.yaml --set schedule.waitWhenToldTo=false +``` + +The stand-in is a few lines of Java: + +```java +import com.sun.net.httpserver.HttpServer; +import java.net.InetSocketAddress; +import java.util.concurrent.Executors; + +/** Lets 20 requests a second through and asks for a second's quiet after that. */ +public class Limit { + private static long second, inThisSecond, quietUntil; + + public static void main(String[] args) throws Exception { + HttpServer server = HttpServer.create(new InetSocketAddress(8429), 0); + server.setExecutor(Executors.newVirtualThreadPerTaskExecutor()); + server.createContext("/", exchange -> { + long now = System.currentTimeMillis(); + boolean let; + synchronized (Limit.class) { + if (now / 1000 != second) { + second = now / 1000; + inThisSecond = 0; + } + let = now >= quietUntil && ++inThisSecond <= 20; + if (!let && now >= quietUntil) { + quietUntil = now + 1000; + } + } + if (let) { + byte[] body = "[]".getBytes(); + exchange.getResponseHeaders().add("Content-Type", "application/json"); + exchange.sendResponseHeaders(200, body.length); + exchange.getResponseBody().write(body); + } else { + exchange.getResponseHeaders().add("Retry-After", "1"); + exchange.sendResponseHeaders(429, -1); + } + exchange.close(); + }); + server.start(); + System.out.println("letting 20 requests a second through at http://localhost:8429"); + } +} +``` + +Saved as `Limit.java`, `java Limit.java` starts it, and `--url http://localhost:8429` points a run at +it. + ## The run left data behind It did: RESTest creates, changes and deletes whatever the document says it can. Point it at a copy diff --git a/docs/report.md b/docs/report.md index 3c02904d5..41356b228 100644 --- a/docs/report.md +++ b/docs/report.md @@ -41,7 +41,7 @@ RESTest testing Swagger Petstore - OpenAPI 3.0 at https://petstore3.swagger.io/a | ` of them ask for an API key that was not given (, in ): --auth gives it` | The document asks for a key and none was given | Those operations will most likely be refused, and how to hand the key over. [Handing over a key or a token](command-line.md#handing-over-a-key-or-a-token) has the forms `--auth` takes | | `the document declares an API key it asks for on no operation (, in ): --auth sends it with of them` | The document declares a key without saying which operations need it | That a key given with `--auth` would go with every operation | | ` left alone by the plan` | The plan's filter set operations aside | Operations the plan keeps the run away from, on purpose: [Which operations a run may touch](campaign-format.md#which-operations-a-run-may-touch) | -| `what it sends depends on the API's own replies, so the seed alone does not repeat this run` | The plan draws on what the API returned, or changes accepted requests, or sends series | Running the same command again makes a similar run rather than the same one. `--store` keeps the run you had. [Getting the seed back](switches.md#getting-the-seed-back) has the files that make a run repeatable from its seed | +| `what it sends depends on the API's own replies, so the seed alone does not repeat this run` | The plan draws on what the API returned, or changes accepted requests, or sends series, or the run leaves alone what the API asks it to (`schedule.waitWhenToldTo`) | Running the same command again makes a similar run rather than the same one. `--store` keeps the run you had. [Getting the seed back](switches.md#getting-the-seed-back) has the files that make a run repeatable from its seed | | ` setting(s) are not what RESTest does by default; --print-settings lists them with where each came from` | A setting was changed, by a file, the environment or `--set` — counting those worked out from it, such as the longest wait for a whole request once `engine.readTimeout` is changed | That the run is not configured the way RESTest ships, so that a setting given through an environment variable is not invisible. `report.json` lists every setting under `settings` | | ` part(s) of the document could not be read:` | Some of the document could not be read | Up to five of the problems, then how many more. An operation a problem was in is tested with what could be read, or, when nothing could be built from it, not tested and named at the end. One whose name the document gives to an operation RESTest reads first is called by its method and path, which costs it nothing else | @@ -84,7 +84,7 @@ some replies could not be checked; report.json names their operations: | Line | When | What it says | |---|---|---| -| ` requests to operations in