Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,9 +305,10 @@ restest run openapi.yaml --url http://localhost:8080 --set engine.maxConcurrency
RESTEST_ENGINE_MAX_CONCURRENCY=1 restest run openapi.yaml --url http://localhost:8080
```

That one is the answer to an API that falls over when it is asked two things at once. [The
settings](docs/settings.md), and [the switches](docs/switches.md) — the settings that each turn off
one thing a run does.
That one is the answer to an API that falls over when it is asked two things at once;
`--set engine.mostRequestsPerMinute=300` keeps a run to five requests a second, evenly spaced, for a
server whose owner wants a rate. [The settings](docs/settings.md), and [the
switches](docs/switches.md) — the settings that each turn off one thing a run does.

**An API that asks for a key** refuses every request without it. `--auth` hands it over, and RESTest
sends it where the document says, with the operations that ask for it:
Expand Down
2 changes: 1 addition & 1 deletion docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Terms used throughout the repository, in commit messages and in pull requests.
| **SPI** (Service Provider Interface) | A deliberately small Java interface that the core defines and *somebody else* implements. The core says "give me values for this parameter" without knowing who answers. Adding a capability then means writing one small class and putting it on the classpath — no change to the core. This is the mechanism that makes the tool extensible. |
| **Event stream** | An internal broadcast: the engine announces what it is doing ("request sent", "response received", "failure found") and any number of independent listeners react. Adding an eighth report format does not touch the engine. |
| **AUC** (Area Under the Curve) | A measure of *how fast* a tool achieves something, not just how much it achieves in the end. A tool that covers 15 operations in the first minute scores far better than one covering 15 in the last minute. |
| **Idle time** | The fraction of the test budget during which the tool had no request in flight — time spent computing instead of testing. Reported on every run. |
| **Idle time** | The fraction of the test budget during which the tool had no request in flight — time spent computing instead of testing, except for what the API asked for (a 429) or the rate the run was given made, which a run reports apart. Reported on every run. |
| **Opening lap** | The first round of a run: every operation sent once, each with the request it is most likely to accept, in steps — lists, then creations, then reads of one thing, then changes, then deletions — each waiting for the answers to the one before. It is what gets operations answered in the first seconds, which is what AUC rewards. Charged to the budget, reported as a phase of its own, and switched off with `schedule.openingLap=false`. |
| **Scheduler** | The part of the tool that holds the deadline and decides which operation is sent next. It sends nothing itself: the request loop asks it what to do. |
| **Fuzzing** | Sending deliberately malformed or extreme inputs to see whether the API handles them correctly. |
Expand Down
11 changes: 10 additions & 1 deletion docs/adr/0009-non-blocking-engine.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ADR-0009: The engine never blocks on computation; idle time replaces a throughput floor

**Status:** Accepted, amended on 29 October 2026
**Status:** Accepted, amended twice on 29 October 2026
**Date:** 2026-09-11

## Context
Expand Down Expand Up @@ -64,3 +64,12 @@ in flight. It is not the tool being slow, which is what idle time exists to catc
much of its idle time came while it waited as the API asked, in `report.json`
(`toldToWait.idle`) and in brackets after the idle share on the first line of its summary. The
tool's own idleness is the rest.

## Amendment (29 October 2026): idle time a rate made

A run can be given a rate to keep to ([ADR-0043](0043-a-rate-to-keep-to.md)), and against a fast API
most of its time is then spent with nothing in flight, waiting for the next turn. That is not the
tool being slow either. The engine counts the idle time that falls after a request went out and
before the next was allowed to, in which nothing could have been sent however quick the tool was,
and a run says it in `report.json` (`idleKeepingToTheRate`, under `engine`) and in the same brackets after
the idle share, beside the API's asking. The tool's own idleness is what is left.
13 changes: 13 additions & 0 deletions docs/adr/0025-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -813,3 +813,16 @@ skips depends on when the API asked, the file for getting the seed back turns it
it on says under its seed that the seed alone does not repeat it. A first wait of no time is refused,
since an API that does not say how long would be asked again at once, and so is a ceiling below the
first wait. One hundred and five settings in nine groups.

## Amendment (29 October 2026): a rate to keep to, and a switch a named rate turns on

`engine.limitTheRate`, off by default, and `engine.mostRequestsPerMinute` (`600`) beside it
([ADR-0043](0043-a-rate-to-keep-to.md)): at most that many requests a minute, never two closer than
a minute divided by it. They are in `engine.*` because the HTTP engine keeps the turns. The switch
is the first whose default follows whether another setting was named rather than its value: when
nobody names it, it is on once a rate is named anywhere, and recorded as *worked out*, so that
`--set engine.mostRequestsPerMinute=300` promises a rate in one line. It is not among the settings
printed as comments, which follow other settings' values, so a printed file writes it out; a run
whose rate is off while its number is not the default says so under its first lines. A rate of zero
is refused, saying that the switch is how a rate is left alone. The switch is listed among those
that are not levers. One hundred and seven settings in nine groups.
12 changes: 11 additions & 1 deletion docs/adr/0042-wait-when-told-to-wait.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ADR-0042: An operation the API asks to be left alone is left alone, and the whole API once a second one is

**Status:** Accepted
**Status:** Accepted, amended on 29 October 2026
**Date:** 2026-10-29

## Context
Expand Down Expand Up @@ -261,3 +261,13 @@ operation of either waited.
depends on the values RESTest invented — a wrong password — and a rule that reacts to an answer
should not count such a refusal. Nothing here tells it apart: the sign-in operation is left alone
for a while, as any other.

## Amendment (29 October 2026): the hold and a rate to keep to

A run can now be given a rate ([ADR-0043](0043-a-rate-to-keep-to.md)), which the engine keeps under
the same lock as the hold: a hold of the whole API still comes first, and a turn the rate gives is
waited for only once no hold is in force. The loop asks the engine how long a request would wait for
its turn right after it asks whether the whole API waits, and sits that out as it sits out a wait of
the whole API, without counting it as quiet the API asked for. `HttpEngine.dropWhatIsHeld()` also
lets go a request waiting for a turn that has not come, so that, when the run stops sending, nothing
the rate held back is sent either.
Loading
Loading