diff --git a/README.md b/README.md index 3f2d432be..77d19130a 100644 --- a/README.md +++ b/README.md @@ -305,9 +305,10 @@ restest run openapi.yaml --url http://localhost:8080 --set engine.maxConcurrency RESTEST_ENGINE_MAX_CONCURRENCY=1 restest run openapi.yaml --url http://localhost:8080 ``` -That one is the answer to an API that falls over when it is asked two things at once. [The -settings](docs/settings.md), and [the switches](docs/switches.md) — the settings that each turn off -one thing a run does. +That one is the answer to an API that falls over when it is asked two things at once; +`--set engine.mostRequestsPerMinute=300` keeps a run to five requests a second, evenly spaced, for a +server whose owner wants a rate. [The settings](docs/settings.md), and [the +switches](docs/switches.md) — the settings that each turn off one thing a run does. **An API that asks for a key** refuses every request without it. `--auth` hands it over, and RESTest sends it where the document says, with the operations that ask for it: diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 8c72b7256..338832a59 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -69,7 +69,7 @@ Terms used throughout the repository, in commit messages and in pull requests. | **SPI** (Service Provider Interface) | A deliberately small Java interface that the core defines and *somebody else* implements. The core says "give me values for this parameter" without knowing who answers. Adding a capability then means writing one small class and putting it on the classpath — no change to the core. This is the mechanism that makes the tool extensible. | | **Event stream** | An internal broadcast: the engine announces what it is doing ("request sent", "response received", "failure found") and any number of independent listeners react. Adding an eighth report format does not touch the engine. | | **AUC** (Area Under the Curve) | A measure of *how fast* a tool achieves something, not just how much it achieves in the end. A tool that covers 15 operations in the first minute scores far better than one covering 15 in the last minute. | -| **Idle time** | The fraction of the test budget during which the tool had no request in flight — time spent computing instead of testing. Reported on every run. | +| **Idle time** | The fraction of the test budget during which the tool had no request in flight — time spent computing instead of testing, except for what the API asked for (a 429) or the rate the run was given made, which a run reports apart. Reported on every run. | | **Opening lap** | The first round of a run: every operation sent once, each with the request it is most likely to accept, in steps — lists, then creations, then reads of one thing, then changes, then deletions — each waiting for the answers to the one before. It is what gets operations answered in the first seconds, which is what AUC rewards. Charged to the budget, reported as a phase of its own, and switched off with `schedule.openingLap=false`. | | **Scheduler** | The part of the tool that holds the deadline and decides which operation is sent next. It sends nothing itself: the request loop asks it what to do. | | **Fuzzing** | Sending deliberately malformed or extreme inputs to see whether the API handles them correctly. | diff --git a/docs/adr/0009-non-blocking-engine.md b/docs/adr/0009-non-blocking-engine.md index b8b625670..a4b6ea41f 100644 --- a/docs/adr/0009-non-blocking-engine.md +++ b/docs/adr/0009-non-blocking-engine.md @@ -1,6 +1,6 @@ # ADR-0009: The engine never blocks on computation; idle time replaces a throughput floor -**Status:** Accepted, amended on 29 October 2026 +**Status:** Accepted, amended twice on 29 October 2026 **Date:** 2026-09-11 ## Context @@ -64,3 +64,12 @@ in flight. It is not the tool being slow, which is what idle time exists to catc much of its idle time came while it waited as the API asked, in `report.json` (`toldToWait.idle`) and in brackets after the idle share on the first line of its summary. The tool's own idleness is the rest. + +## Amendment (29 October 2026): idle time a rate made + +A run can be given a rate to keep to ([ADR-0043](0043-a-rate-to-keep-to.md)), and against a fast API +most of its time is then spent with nothing in flight, waiting for the next turn. That is not the +tool being slow either. The engine counts the idle time that falls after a request went out and +before the next was allowed to, in which nothing could have been sent however quick the tool was, +and a run says it in `report.json` (`idleKeepingToTheRate`, under `engine`) and in the same brackets after +the idle share, beside the API's asking. The tool's own idleness is what is left. diff --git a/docs/adr/0025-settings.md b/docs/adr/0025-settings.md index 4da7558b0..16b542ea9 100644 --- a/docs/adr/0025-settings.md +++ b/docs/adr/0025-settings.md @@ -813,3 +813,16 @@ skips depends on when the API asked, the file for getting the seed back turns it it on says under its seed that the seed alone does not repeat it. A first wait of no time is refused, since an API that does not say how long would be asked again at once, and so is a ceiling below the first wait. One hundred and five settings in nine groups. + +## Amendment (29 October 2026): a rate to keep to, and a switch a named rate turns on + +`engine.limitTheRate`, off by default, and `engine.mostRequestsPerMinute` (`600`) beside it +([ADR-0043](0043-a-rate-to-keep-to.md)): at most that many requests a minute, never two closer than +a minute divided by it. They are in `engine.*` because the HTTP engine keeps the turns. The switch +is the first whose default follows whether another setting was named rather than its value: when +nobody names it, it is on once a rate is named anywhere, and recorded as *worked out*, so that +`--set engine.mostRequestsPerMinute=300` promises a rate in one line. It is not among the settings +printed as comments, which follow other settings' values, so a printed file writes it out; a run +whose rate is off while its number is not the default says so under its first lines. A rate of zero +is refused, saying that the switch is how a rate is left alone. The switch is listed among those +that are not levers. One hundred and seven settings in nine groups. diff --git a/docs/adr/0042-wait-when-told-to-wait.md b/docs/adr/0042-wait-when-told-to-wait.md index 663024dc2..71339568e 100644 --- a/docs/adr/0042-wait-when-told-to-wait.md +++ b/docs/adr/0042-wait-when-told-to-wait.md @@ -1,6 +1,6 @@ # ADR-0042: An operation the API asks to be left alone is left alone, and the whole API once a second one is -**Status:** Accepted +**Status:** Accepted, amended on 29 October 2026 **Date:** 2026-10-29 ## Context @@ -261,3 +261,13 @@ operation of either waited. depends on the values RESTest invented — a wrong password — and a rule that reacts to an answer should not count such a refusal. Nothing here tells it apart: the sign-in operation is left alone for a while, as any other. + +## Amendment (29 October 2026): the hold and a rate to keep to + +A run can now be given a rate ([ADR-0043](0043-a-rate-to-keep-to.md)), which the engine keeps under +the same lock as the hold: a hold of the whole API still comes first, and a turn the rate gives is +waited for only once no hold is in force. The loop asks the engine how long a request would wait for +its turn right after it asks whether the whole API waits, and sits that out as it sits out a wait of +the whole API, without counting it as quiet the API asked for. `HttpEngine.dropWhatIsHeld()` also +lets go a request waiting for a turn that has not come, so that, when the run stops sending, nothing +the rate held back is sent either. diff --git a/docs/adr/0043-a-rate-to-keep-to.md b/docs/adr/0043-a-rate-to-keep-to.md new file mode 100644 index 000000000..f4ea7f873 --- /dev/null +++ b/docs/adr/0043-a-rate-to-keep-to.md @@ -0,0 +1,224 @@ +# ADR-0043: A run can be given a rate to keep to, in one line, and keeps to it evenly where it sends + +**Status:** Accepted +**Date:** 2026-10-29 + +## Context + +What bounds a run today is how many requests it has in flight: sixteen at most +(`engine.maxConcurrency`), fewer when answers slow down or go missing. Against an API on the same +machine that is thousands of requests a second: blogapi, run without its key for 20 s in +[ADR-0042](0042-wait-when-told-to-wait.md)'s evidence, received 149,242. That is right for an API +somebody runs to test, and wrong for a staging server somebody else runs and lets them test, whose +owner wants to be promised a rate. Nothing in RESTest could make that promise: +[known limitations](../known-limitations.md) said so, and pointed at `engine.maxConcurrency=1` as the +nearest thing, which bounds how many requests wait at once, not how often one goes. + +Among the related tools, a rate is common and off by default: + +- **Schemathesis**, `--rate-limit 5/s`, `100/m` or `1000/h`, through the pyrate-limiter library's + sliding window: at most that many in any window of that length, which lets all of them go at once + and then waits. One limit per host the run sends to. +- **EvoMaster**, `--ratePerMinute 600`, a whole number, zero or less for none: it sleeps a minute + divided by the number between one action and the next, so the actions go evenly. +- **ZAP** has a rate in its network options. **CATS** alone keeps one always on, at 10,000 requests + a minute. + +## Decision + +### 1. Two settings, and naming the rate turns it on + +- `engine.limitTheRate`, a switch, off by default: a rate on by default would slow every run to + protect the few pointed at somebody else's server. A default that switched on for any address + but the tool's own machine was considered and set aside: APIs under test often run in containers + at addresses that are not the machine's. +- `engine.mostRequestsPerMinute`, a whole number of at least one, `600` by default. + +Per minute and a whole number, as EvoMaster writes it: exact for any limit of so many a minute, and +five a second is `300`. The maintainer chose it on 29 October 2026 over Schemathesis's `5/s`, +`100/m`, `1000/h` and over requests a second with decimals, which writes a limit of a hundred a +minute as `1.6667`, slightly over it. Ten a second by default, of the order of what APIs commonly +allow one client: slow enough for any staging server, quick enough for the opening lap of an API of +thirty-five operations, as pet-clinic is, to take under four seconds. + +**Naming a rate limits the rate.** When nobody names the switch, it is on once +`engine.mostRequestsPerMinute` is named in any of the four places a setting is given +([ADR-0025](0025-settings.md)), and recorded as *worked out*, as `engine.callTimeout` is when one of +the waits inside it is named. So `--set engine.mostRequestsPerMinute=300` is the one line that +promises a rate. A rate typed and then ignored because the switch was not typed too would let the run +go as fast as it can while whoever typed it believed the promise made. The switch named outright +still decides, either way: `limitTheRate: false` beside a rate leaves the rate alone, and +`limitTheRate: true` alone keeps to ten a second. + +A printed settings file writes the switch out, `limitTheRate: false` beside +`mostRequestsPerMinute: 600`, so changing only the number in it keeps no rate. For that case the run +says so under its first lines whenever the rate is off and the number is not the default: +`engine.mostRequestsPerMinute is 300, but engine.limitTheRate is false, so no rate is kept to`. The +switch is not added to the settings that follow others and are printed as comments: that list holds +settings that follow other settings' values, and this one follows whether another was named. + +A rate of `0` is refused, saying that `engine.limitTheRate=false` is how a rate is left alone: zero +is what somebody tries for "none", and a number set to zero says nothing about switching +([ADR-0025](0025-settings.md)'s M11.2 amendment). + +### 2. The promise: never two requests closer than a minute divided by the rate + +At `600`, never two closer than a tenth of a second. The first request goes at once, and each turn +is counted from the request before it, never from the turn that was due, so a quiet spell is never +made up for with a burst: never six hundred in the first seconds and then nothing, which is what a +sliding window allows and what the maintainer asked about. The promise is not "at most so many in +any one minute": a window that includes both of its ends holds one more. + +### 3. One schedule, in the engine, and a loop that builds a request when it can go + +- **The engine keeps the turns.** `ConcurrencyLimiter.acquire()` gives no slot to a request before + its turn, however many slots are free, under the same lock as ADR-0042's hold, which comes first. + A request waiting for its turn holds no slot and earns nothing for the limiter, as one waiting out a + hold does. The first turn is the moment the limiter is built, so the first request goes at once + wherever the clock happens to begin. +- **It says how long one more request would wait.** `HttpEngine.untilItsTurn()`: until the next turn, + and one turn more for each request handed over and not yet given a slot. The engine counts a + request handed over on the thread that hands it over, before it leaves, so that the next question + already counts it. Room among the requests in flight is not counted: that depends on answers + nobody can foresee. +- **The loop asks, right before it builds anything**, beside the whole API's quiet: if the answer is + more than nothing, it gives its slot back, sits the wait out, to the deadline or a stop at most, and + looks again. Requests are then built when they can go, and do not pile up inside the engine with + values that were fresh when they were made. When the API answers more slowly than the rate allows, + requests wait in the engine for room among those in flight, as they do with no rate; the loop's + answer counts them, so it hands over one more only a turn per request already waiting, and that + queue stays about as long as the API is behind, up to `schedule.workAheadFactor` × + `engine.maxConcurrency`. It empties once the API catches up. + +### 4. At the end of the run, what waits for its turn is not sent + +When the run stops sending, `HttpEngine.dropWhatIsHeld()` lets go every request waiting for a turn +that has not come, now or later, as it already lets go what a hold of the whole API holds, so that +nothing the rate held back reaches the API after the run's time is up. Each comes back as one that +never went out, and is counted with the requests that got no reply. A request waiting only for a +slot, its turn already come, still goes, as it does today. With the loop of §3 there is hardly ever +more than one. Decided by the maintainer on 29 October 2026. + +### 5. What a run says + +- **The engine counts the idle time the rate made**: the time with nothing in flight after a request + went out and before the next was allowed to, in which nothing could have been sent however quick + the tool was. Idle time after the turn came is the tool's own. +- **`report.json`**: `idleKeepingToTheRate`, under `engine`, `PT0S` with no rate given. +- **The summary**: the idle share on its first line is followed by that part, beside what the API + asked for. Against pet-clinic for 30 s at 300 a minute it read + `145 requests to 43 operations in 30.0s, 85% of it idle (81% keeping to the rate it was given)`; + with both, `(% waiting as the API asked, % keeping to the rate it was given)`. A part that + rounds to none is left out. The reasoning is + [ADR-0009](0009-non-blocking-engine.md)'s amendment of 29 October: idle time exists to catch a slow + tool, and quiet somebody asked for is not that. The two parts can overlap, by at most one turn for + each time the run sits out the API's asking. +- **The page** shows the idle share as before. +- **The exit code** does not change. + +### 6. A switch, but not a lever + +The switch is listed among those that are not levers: a rate is a courtesy to the API, not a way of +finding more, so it plays no part in an ablation. It changes when requests go, and so how many fit +in a run's time, and makes nothing a run sends depend on the API's replies, so the file for getting +the seed back leaves it as it is. + +## The evidence + +Off by default, so no run changes unless somebody names a rate, and there were no recorded runs +with a rate to replay before it was built. Measured once built, on one +machine, against two APIs of the dataset of local APIs, run from scratch for 20 s each with the +same seed and every request stored. The gaps are between the moments RESTest recorded sending one +request and the next: + +| Run | Requests | Shortest gap | Most in any one second | Idle share, and the part the rate made | Last request sent, against the time being up | Never sent at the end | +|---|---|---|---|---|---|---| +| blogapi, 30 a minute | 10 | 2,000.6 ms | 1 | 98% (96%) | 1.448 s before | 0 | +| blogapi, 300 a minute | 95 | 200.7 ms | 5 | 92% (87%) | 0.205 s before | 0 | +| blogapi, 3,000 a minute | 810 | 20.3 ms | 43 | 75% (56%) | 0.003 s before | 0 | +| petclinic-rest, 300 a minute | 95 | 201.2 ms | 5 | 92% (86%) | 0.174 s before | 0 | +| blogapi, nothing named | 70,196 | under 0.1 ms | 4,388 | 3% | 0.003 s after | — | +| petclinic-rest, nothing named | 98,112 | under 0.1 ms | 7,722 | 5% | 0.003 s after | — | + +At 3,000 a minute blogapi got 810 of the 1,000 turns its time held: the opening lap waits for the +answers to each of its steps, and some answers took longer than a turn. + +With nothing named, the run is as it was. Against `master`, in pairs with the same seed and +alternating order: + +| | `master` | This change | +|---|---|---| +| petclinic-rest, seeds 1, 2, 3: requests | 115,061; 107,686; 104,623 | 104,566; 105,596; 110,442 | +| petclinic-rest: idle share | 2.3%; 2.3%; 2.1% | 2.3%; 2.2%; 2.3% | +| blogapi, seeds 1, 2, 3: requests | 70,008; 75,433; 68,661 | 59,958; 74,322; 68,663 | +| blogapi: idle share | 1.9%; 1.9%; 2.1% | 1.6%; 1.7%; 2.2% | + +Requests vary between runs of one version as much as between the two: petclinic-rest's three runs +of `master` span 104,623 to 115,061. blogapi's first pair is the widest apart, and its third the +closest, two requests in 68,661. An engine with no rate takes no lock it did not take before, and +reads the clock once more, when it is built. + +**Why the engine keeps the turns, and not the loop alone.** A stand-in with four requests in flight +held their answers for 0.3 s and then gave them all at once, to an engine keeping to 600 a minute +with eight requests handed over (the engine's own test, `KeepingToARateTest`). With the engine's turn, the four that had waited went out 102 to +106 ms apart. With the turn taken out of the engine, as a rate kept by the loop alone would leave +it, they went out 12 to 157 microseconds apart. + +**Where the API sees them.** The rate is kept where RESTest sends. Seen from a stand-in that records +when each request arrives, the first two of a run kept to 600 a minute arrived 41 ms apart, every +later pair 101 to 107 ms: the first request opens the connection and arrives late, and the second, +sent a turn after it, does not. + +## Alternatives considered + +- **One number and no switch, with `0` or a negative number for "none"**, as EvoMaster does. One line + and no trap, but the only setting of the hundred and seven with a value that means "off", against + the rule given with the switches on 29 September 2026, and missing from the page of switches, + whose test finds a switch by its kind. +- **A plain switch and a number, as `schedule.stopAfterRequests` is**, where the number alone does + nothing. Two `--set` options for one promise, and a rate typed without the switch would be ignored. +- **Both lines printed as comments until somebody names one**, so that a printed file whose number is + changed keeps the rate too. A new rule in how settings are printed, for a case the line under the + first lines already catches. +- **`N/s`, `N/m`, `N/h`, as Schemathesis writes it**, which copies a limit as an API publishes it. A + new kind of value in the settings, with its own reading and writing. **Requests a second with + decimals**, which needs no new kind but writes a limit of a hundred a minute as `1.6667`. The + maintainer chose per minute. +- **A sliding window, as Schemathesis's**, or a token bucket. Either lets requests go together — a + window all of them at once, a bucket up to twice the rate across the edge of a minute — and a rate + promised to an owner should not. +- **The engine alone.** The loop works up to `schedule.workAheadFactor` × `engine.maxConcurrency` + requests ahead, thirty-two by default, and they would wait inside the engine for their turns: built + up to thirty-two turns before they go, and, at the end, sent after the run's time or dropped by the + dozen. +- **The loop alone.** Requests handed over evenly still go out together when every slot was taken + and several answers come back at once — an API that stalls and recovers. Measured on a stand-in that + does that (above), the engine's turn is what keeps them apart. +- **A turn in the loop and another in the engine, each with its own clock.** Proposed first, and + found wrong in review: after a stall both sides run at the rate, so a queue that formed inside the + engine never drains, and every request after it is built long before it goes. + +## Consequences + +- **`--set engine.mostRequestsPerMinute=300` keeps a run to five requests a second, evenly.** With + nothing named, nothing changes. +- **`HttpEngine` gains `untilItsTurn()`**, which an engine with no rate answers with nothing, and + `dropWhatIsHeld()` also lets go what waits for a turn. +- **`EngineStatistics` gains `idleKeepingToTheRate`**, `report.json` its field and the summary its + part of the brackets. +- **Two settings join `engine.*`**: one hundred and seven in nine groups. +- **The known limitations** lose "there is no ceiling on requests per second" and gain what the rate + does not do. + +## Still open + +- **The page.** It shows the idle share without saying how much of it the rate made: a change of + layout needs a mock-up first, as ADR-0042 left its own section. +- **A rate per operation**, or one taken from the `RateLimit` fields an API may send, which RESTest + does not read. +- **Redirections.** With `engine.followRedirects` on, which it is not by default, a redirection is + followed inside the request that drew it, without a turn of its own, so two requests can reach + the API closer together than the rate allows. +- **The document fetched over the network** is one request the rate does not count. +- **Requests waiting for their turn are not served in order**: several may wake at once and the lock + decides which takes it. With the loop of §3 there is hardly ever more than one. diff --git a/docs/adr/README.md b/docs/adr/README.md index b09481aa6..d253437d9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -61,7 +61,7 @@ so records written from now on cite none. | [0006](0006-event-stream-and-store.md) | One event stream; every interaction persisted | Accepted, amended at M1.4, M1.6, M1.7, M11.1, M9.1 and M11.3, and in #325, #327 and #334, and on 10, 19 and 25 October 2026 | | [0007](0007-specification-parser-boundary.md) | The parser sits behind our own interface; OAS 2.0, 3.0.x and 3.1.x | Accepted, amended at M0.2, in #303 and in #328, reversed at M1.2 | | [0008](0008-extension-points-no-ai-abstractions.md) | Extension points, and no AI-specific abstractions | Accepted | -| [0009](0009-non-blocking-engine.md) | The engine never blocks; idle time replaces a throughput floor | Accepted, amended on 29 October 2026 | +| [0009](0009-non-blocking-engine.md) | The engine never blocks; idle time replaces a throughput floor | Accepted, amended twice on 29 October 2026 | | [0010](0010-idl-strategy.md) | IDL: relicensed assets, ANTLR4 parser, solver behind an interface | Accepted, not yet built (14 October 2026) | | [0011](0011-evaluation-harness.md) | Evaluation harness in `evaluation/`, outside the Maven build — amended at M1.9: in a repository of its own, so `evaluation/` was never created | Accepted, amended at M1.9 and on 9 October 2026 | | [0012](0012-canonical-model.md) | The canonical model: immutable records, sealed types, recorded gaps | Accepted, amended at M1.8, M2.1a and M2.1b, and on 17 October 2026 | @@ -94,4 +94,5 @@ so records written from now on cite none. | [0039](0039-the-html-report.md) | The HTML report: one page beside report.json, drawn in the browser from the whole of it, fetching nothing | Accepted | | [0040](0040-homebrew.md) | RESTest is installed with Homebrew from a tap of its own, by a formula that brings its Java and is written into the tap once a release is published | Accepted | | [0041](0041-stop-when-the-api-refuses-the-whole-opening-lap.md) | A run stops sending when the API answers 401 to every request of its opening lap, and ends with `5` | Accepted | -| [0042](0042-wait-when-told-to-wait.md) | An operation the API asks to be left alone is left alone, and the whole API once a second one is | Accepted | +| [0042](0042-wait-when-told-to-wait.md) | An operation the API asks to be left alone is left alone, and the whole API once a second one is | Accepted, amended on 29 October 2026 | +| [0043](0043-a-rate-to-keep-to.md) | A run can be given a rate to keep to, in one line, and keeps to it evenly where it sends | Accepted | diff --git a/docs/known-limitations.md b/docs/known-limitations.md index cee8ff47e..f61dc9f21 100644 --- a/docs/known-limitations.md +++ b/docs/known-limitations.md @@ -24,13 +24,24 @@ current release: an entry leaves it in the release that lifts it, and that relea Each 429 the document does not declare is still a warning, `F920`, and each 503 a fault, `F121`, like any other reply — there are few of them once the run waits. `schedule.waitWhenToldTo=false` - sends to such an API as RESTest did before it waited; against an API that limits its callers, a - lower `engine.maxConcurrency` ([the settings](settings.md)) still keeps the run further from its - limit. -- **There is no ceiling on requests per second.** What bounds a run is the number of requests in - flight, sixteen at most by default (`engine.maxConcurrency`). The limiter halves it whenever a - request goes unanswered and sends fewer as answers slow down, but no setting promises an API's - owner a rate. Lowering `engine.maxConcurrency` is the nearest thing. + sends to such an API as RESTest did before it waited; against an API whose limit you know, + `engine.mostRequestsPerMinute` ([the settings](settings.md#five-things-worth-knowing)) set below it + keeps the run under it rather than finding it by going over. +- **A rate is kept to only when it is given, for the whole run, and where RESTest sends.** By + default nothing bounds a run but its requests in flight, sixteen at most (`engine.maxConcurrency`), + and against a fast API that is thousands of requests a second. `engine.mostRequestsPerMinute` + keeps a run to that many a minute, evenly spaced. What it does not do: + - it is one rate for the whole API, not one per operation; + - it counts the requests of the test, not the one that fetches the document when its address is + given rather than a file; + - it is kept to where RESTest sends: the network on the way may bring two requests a little closer + together, most of all the first, behind the connection it opens; + - with `engine.followRedirects` on, which it is not by default, a redirection is followed within + the request that drew it, without waiting for a turn of its own; + - the requests already built and waiting for their turn when the run's time is up are never sent, + and are counted with the requests that got no reply; there is hardly ever more than one; + - the page shows the idle share without saying how much of it the rate made; the summary and + `report.json` (`idleKeepingToTheRate`, under `engine`) do. - **A run stops for a refused key only when the API refuses the whole opening lap.** When every request of the opening lap is answered and every answer is 401, the run stops sending there, says which key it held, or how the document's is handed over, and ends with `5` @@ -70,7 +81,7 @@ of its own. Whether you may test an API at all is not something any tool can che - **Optional parameters and optional bodies share one number.** With `generation.optionalParametersBySize` off, each optional parameter is decided on a coin weighted by `generation.optionalBodyChance`, the same number that decides whether a request sends a body it - may leave out, so changing one changes the other ([the settings](settings.md#four-things-worth-knowing)). + may leave out, so changing one changes the other ([the settings](settings.md#five-things-worth-knowing)). Leave `optionalParametersBySize` on, which is the default, if you want to change only bodies. - **A value can be offered back to the operation that first used it.** A value an accepted request carried is offered to the other operations and not back to its own, but a value seen again replaces diff --git a/docs/manual/07-settings.md b/docs/manual/07-settings.md index 984e71247..8fcd703ae 100644 --- a/docs/manual/07-settings.md +++ b/docs/manual/07-settings.md @@ -93,6 +93,22 @@ used and where each came from. flight, and slows down by itself when the API does. An API that cannot take even that, or one you do not want loaded, gets one request at a time with `--set engine.maxConcurrency=1`. +**Somebody else's API.** Against an API on your own machine RESTest sends thousands of requests a +second. Whoever runs a staging server you were allowed to test usually wants a rate instead, and one +line promises it: + +```bash +restest run http://localhost:9966/petclinic/v3/api-docs --budget 30s --set engine.mostRequestsPerMinute=300 +``` + +That is five requests a second, evenly spaced: never two closer than a fifth of a second, and never +three hundred in the first seconds and then nothing. The quiet it makes is not RESTest being slow, +and the first line of the summary says how much of the idle share it was: + +``` +145 requests to 43 operations in 30.0s, 85% of it idle (81% keeping to the rate it was given) +``` + **A slow API.** RESTest waits thirty seconds for an answer once connected. For an API that takes longer, `--set engine.readTimeout=2m`. Lengths of time are written `500ms`, `30s`, `5m`, `2h`, or as a plain number of seconds. diff --git a/docs/manual/10-troubleshooting.md b/docs/manual/10-troubleshooting.md index 55153eff5..7f75a0260 100644 --- a/docs/manual/10-troubleshooting.md +++ b/docs/manual/10-troubleshooting.md @@ -205,8 +205,9 @@ The same goes for *Undeclared content type* (`F921`) and a document that declare RESTest keeps several requests in flight at once, and slows down by itself when the API does. An API that cannot take that — it answers 503, or stops answering, or its log fills with errors about -connections — gets one request at a time with `--set engine.maxConcurrency=1`. Each 503 is reported -as a fault, `F121`, which a run whose pace is to blame can stop with +connections — gets one request at a time with `--set engine.maxConcurrency=1`, and one whose owner +allows so many requests a minute keeps to them with `--set engine.mostRequestsPerMinute=300`. Each +503 is reported as a fault, `F121`, which a run whose pace is to blame can stop with `--set checks.otherServerError=false`; each request it stops answering, as `F312`, which `--set checks.noReplyInTime=false` stops. An API that is simply slow gets longer to answer with `--set engine.readTimeout=2m`. [Chapter 7](07-settings.md) has more. diff --git a/docs/report.md b/docs/report.md index 41356b228..faeb4de07 100644 --- a/docs/report.md +++ b/docs/report.md @@ -43,6 +43,7 @@ RESTest testing Swagger Petstore - OpenAPI 3.0 at https://petstore3.swagger.io/a | ` left alone by the plan` | The plan's filter set operations aside | Operations the plan keeps the run away from, on purpose: [Which operations a run may touch](campaign-format.md#which-operations-a-run-may-touch) | | `what it sends depends on the API's own replies, so the seed alone does not repeat this run` | The plan draws on what the API returned, or changes accepted requests, or sends series, or the run leaves alone what the API asks it to (`schedule.waitWhenToldTo`) | Running the same command again makes a similar run rather than the same one. `--store` keeps the run you had. [Getting the seed back](switches.md#getting-the-seed-back) has the files that make a run repeatable from its seed | | ` setting(s) are not what RESTest does by default; --print-settings lists them with where each came from` | A setting was changed, by a file, the environment or `--set` — counting those worked out from it, such as the longest wait for a whole request once `engine.readTimeout` is changed | That the run is not configured the way RESTest ships, so that a setting given through an environment variable is not invisible. `report.json` lists every setting under `settings` | +| `engine.mostRequestsPerMinute is , but engine.limitTheRate is false, so no rate is kept to` | A rate was given, but the switch says not to keep to it: a rate other than the default with the switch off, or any rate given in the environment or with `--set` over a file that turns the switch off, as a printed settings file does | That the run will go as fast as it can, not at the rate. Take `limitTheRate: false` out of the file, or add `--set engine.limitTheRate=true`. [The settings](settings.md#five-things-worth-knowing) say why | | ` part(s) of the document could not be read:` | Some of the document could not be read | Up to five of the problems, then how many more. An operation a problem was in is tested with what could be read, or, when nothing could be built from it, not tested and named at the end. One whose name the document gives to an operation RESTest reads first is called by its method and path, which costs it nothing else | Problems with a file you handed over — a plan, a list of values, a key in `RESTEST_AUTH` — are @@ -84,7 +85,7 @@ some replies could not be checked; report.json names their operations: | Line | When | What it says | |---|---|---| -| ` requests to operations in