Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 46 additions & 1 deletion docs/adr/0012-canonical-model.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ADR-0012: The canonical model is immutable records and sealed types, and says so when it cannot represent something

**Status:** Accepted, amended at M1.8, M2.1a and M2.1b, and on 17 and 26 October 2026
**Status:** Accepted, amended at M1.8, M2.1a and M2.1b, and on 17, 26 and 29 October 2026
**Date:** 2026-09-12

## Context
Expand Down Expand Up @@ -145,6 +145,51 @@ shape, and it is lossless in both directions: a 3.0 document's `minimum: 5` plus

---

## Amendment (29 October 2026): only a reply that could not be read makes an operation read in part

**Date:** 2026-10-29

**`ApiModel.readInPart` counts a `DEGRADED` issue only when it is about one of the operation's
replies, whole, and the reader says which ones are by where it puts them: a reply it cannot read is
located at the reply itself, `paths./pets.get.responses.404`, rather than at the operation.** The
checks of a status code and of a content type the document does not declare leave alone an
operation read in part, since a reply the document declares and RESTest could not read would
otherwise look undeclared. Until now every `DEGRADED` issue counted, and most hide no reply: a parameter, a request
body or the body of a reply built from a shape RESTest does not fully understand — a
`discriminator`, a shape the document never defines — is read with less than the document holds,
but every status code and content type the operation declares is still read. Such an operation lost
both checks with nothing missing from its replies. Of the 3,627 operations in the 98 documents read
for the amendment of 26 October, 378, in 12 documents, counted as read in part; 4 do now, all in
catwatch, which declares a reply under the status code `0`. On ohsome-api 126 of 134 operations were
left alone, and on jasper 59 of 61.

**Where the issue is, rather than what it cost**, because the two kinds cost the same: the operation
is tested with less than the document holds, which is what `DEGRADED` says. Three other answers
were weighed:

- **A sixth effect, for a reply that could not be read.** It would say two things in one field, the
cost and the place, and it would change the word `report.json` has written since 2.2.0 for the
very issues the two checks were built around.
- **A field on the issue naming the part of the operation it is about.** The location already says
where an issue is, which is what it is for, and the field would change the constructor of a record
`restest-core` publishes, as the amendment of 17 October declined to.
- **Counting whatever is inside a reply too**: a shape not understood in its body, a header given as
a reference that leads nowhere. Neither hides a status code or a content type, which is all the
two checks look for; the check of a body against its shape has a reason of its own for a shape it
cannot use, `shapeUnusable`.

**A location counts only when it is a reply itself**: what follows its last `.responses.` is a
status code, which holds no dot. An issue placed later inside a reply — a header's, at
`…responses.200.headers.X-Rate` — then does not count, and `SpecificationIssue` says where a reply
that could not be read goes, for whoever else writes issues.

**What changes in a run**: those operations are held to the status codes and content types they
declare, and a reply they do not declare is reported as on any other operation; with
`checks.failOnWarnings` on, such a run can now end with `1`. The screen, `documentIssues` in
`report.json` and the page give a reply that could not be read its own location, and the words for
`readInPart` on the screen, in `notChecked[].why` and on the page say that a reply the document
declares could not be read.

## Amendment (26 October 2026): what no request can carry is left out, and said

**Date:** 2026-10-26
Expand Down
8 changes: 8 additions & 0 deletions docs/adr/0016-how-a-fault-is-presented.md
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,14 @@ scout-api no renamed operation is also one RESTest read in part, so the replay's
two rules judge those eight operations as they are defined to; that what could not be read is said
of the right operation is shown by RESTest's tests, not by this run.

**A correction of 29 October: only a reply RESTest could not read leaves an operation alone.** That
`F920` and `F921` leave alone any operation a `DEGRADED` issue names no longer holds either. A shape
RESTest does not fully understand, in a parameter, a request body or the body of a reply it did
read, hides no status code and no content type the operation declares, and such an operation is
judged; only a reply the document declares and RESTest could not read at all still keeps it from
both rules ([ADR-0012](0012-canonical-model.md)'s amendment of 29 October, with what it changes
across 98 documents). The reason a reply gets for it, `readInPart`, now says so in its words too.

## Amendment (22 October 2026): what a person reads first

**Date:** 2026-10-22
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ so records written from now on cite none.
| [0009](0009-non-blocking-engine.md) | The engine never blocks; idle time replaces a throughput floor | Accepted, amended twice on 29 October 2026 |
| [0010](0010-idl-strategy.md) | IDL: relicensed assets, ANTLR4 parser, solver behind an interface | Accepted, not yet built (14 October 2026) |
| [0011](0011-evaluation-harness.md) | Evaluation harness in `evaluation/`, outside the Maven build — amended at M1.9: in a repository of its own, so `evaluation/` was never created | Accepted, amended at M1.9 and on 9 October 2026 |
| [0012](0012-canonical-model.md) | The canonical model: immutable records, sealed types, recorded gaps | Accepted, amended at M1.8, M2.1a and M2.1b, and on 17 October 2026 |
| [0012](0012-canonical-model.md) | The canonical model: immutable records, sealed types, recorded gaps | Accepted, amended at M1.8, M2.1a and M2.1b, and on 17, 26 and 29 October 2026 |
| [0013](0013-input-generation.md) | Where input values come from, and how a campaign is put together | Accepted, amended at M1.11, M2.7a, M2.4, M2.10a, M9.1, M9.3, M10.1 and M10.3, and on 10, 12, 13 and 25 October 2026 |
| [0014](0014-response-conformance.md) | A reply is judged against the specification document itself | Accepted, amended at M1.8 and on 10, 17, 19 and 22 October 2026 |
| [0015](0015-command-line-contract.md) | One command, a budget spent in full, and what an exit code means | Accepted, amended at M1.7, M1.8, M2.7a, M2.10a, M11.1, M9.1, M11.3, M12.1a, M12.1c and M12.1b, in #314 and #344, and on 1, 11, 12, 19, 22, 25 and 28 October 2026 |
Expand Down
26 changes: 14 additions & 12 deletions docs/faults.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,16 +319,18 @@ and `default` included.
counts each such reply as one this check could not judge (`noReplyDeclared`, under
`notChecked` in `report.json`), unless the operation is one of those below, read in part,
which comes first (`readInPart`).
- On an operation RESTest could not read in full, a status code it could not find among what it
read: a reply the document declares, and RESTest could not read — one that points at a shape the
document never defines, say — would otherwise look undeclared. The run lists what it could not
read before it starts, and counts each such reply as one this check could not judge
(`readInPart`). A status code it did read is held to as on any other operation. A reply the
library RESTest reads documents with drops on its own, such as one written as a word rather than
an object, is not known to be missing, and does look undeclared ([known
limitations](known-limitations.md#what-a-run-says)). An operation RESTest only had to call by
its method and path, because the document gives its name to an operation RESTest reads first,
is not one of these: the new name costs it nothing else.
- On an operation one of whose replies RESTest could not read, a status code it could not find
among what it read: a reply the document declares, and RESTest could not read — one given as a
reference to a reply the document never defines, say, or under a status code that is not one —
would otherwise look undeclared. The run lists what it could not read before it starts, and
counts each such reply as one this check could not judge (`readInPart`). A status code it did
read is held to as on any other operation. A reply the library RESTest reads documents with
drops on its own, such as one written as a word rather than an object, is not known to be
missing, and does look undeclared ([known limitations](known-limitations.md#what-a-run-says)).
Nothing else RESTest says it could not read hides a reply, so nothing else makes an operation one
of these: not a shape it does not fully understand, in a parameter, a request body or the body of
a reply it did read, not a header of a reply it could not read, and not a name of RESTest's own,
given because the document gives the operation's name to an operation RESTest reads first.
- A reply to a request whose body RESTest sent, on purpose, under a media type the operation does
not accept: a document is not expected to describe how an API refuses a body it never promised
to take.
Expand Down Expand Up @@ -369,8 +371,8 @@ warning Undeclared content type (F921)
- A document that declares `*/*`, which promises no type in particular — and is what a Swagger 2.0
document that says nothing about types is read as, so such a reply counts as checked.
- A server error, and a reply to a body sent under a media type the operation does not accept, as
for `F920`; and, on an operation RESTest could not read in full, a type it could not find among
what it read, counted as for `F920` (`readInPart`).
for `F920`; and, on an operation one of whose replies RESTest could not read, a type it could
not find among what it read, counted as for `F920` (`readInPart`).
- **What the API wrote is quoted cut short.** The fault quotes the content type as the API sent it,
cut at 100 characters, with anything that would move a terminal's cursor or change its colours,
and invisible characters that would reorder or hide what is shown around them, replaced by `?`.
Expand Down
7 changes: 4 additions & 3 deletions docs/known-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,10 @@ of its own. Whether you may test an API at all is not something any tool can che
- **A reply the document declares in a form RESTest cannot read is reported as undeclared.** The
library RESTest reads documents with drops a reply written as something other than an object —
`"404": "not found"` — and says so before the run, without naming the operation. That operation's
404s are then `F920`, though the document meant to declare them. On an operation RESTest knows it
read only in part, such as one with a reply pointing at a shape the document never defines, a
status code it could not find among what it read is left alone, and counted under `notChecked`.
404s are then `F920`, though the document meant to declare them. On an operation one of whose
replies RESTest knows it could not read, such as one given as a reference to a reply the document
never defines, a status code it could not find among what it read is left alone, and counted
under `notChecked`.
- **A body the document describes only in words is not checked against a type.** A reply with a body
whose status code the document declares with no content type at all is left alone, whatever type
it comes as, and counted under `notChecked`. A Swagger 2.0 document that names no type at all is
Expand Down
2 changes: 1 addition & 1 deletion docs/report.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,7 +526,7 @@ those, `F920` and `F921`.
| `shapeUnusable` | shape | The document gives a shape RESTest cannot use: one pointing at another file, at something the document does not define, or round in a circle, one with a pattern the checker cannot read, or a document that could not be kept to check against |
| `keyHidden` | shape | A key handed over with `--auth` came back in the body and was hidden there, and the body could then no longer be read as JSON, so what is wrong with it may be the hiding's doing |
| `noReplyDeclared` | status code | The document declares no reply at all for the operation, so it promised none |
| `readInPart` | status code, content type | RESTest could not read all the document says about the operation — a reply pointing at a shape it never defines, say — and what came back is not among what it could read, so it may be declared in the part it could not. What it could read is held to as on any other operation |
| `readInPart` | status code, content type | RESTest could not read a reply the document declares for the operation — one given as a reference to a reply it never defines, say — and what came back is not among what it could read, so it may be declared in the reply it could not. What it could read is held to as on any other operation |
| `noContentTypeDeclared` | content type | The document declares the status code with no content type, describing that reply only in words |

### `dictionaries`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@
* whatever it is sent: nothing is broken on purpose in a request to it, since the identifier is the
* only thing such a request could have changed. The other answers 404, which it does not declare,
* and is reported: having nothing left out that a request could carry, it is held to its document.
*
* <p>The fourth document builds a parameter of one operation, and the body of a reply of another,
* from a shape it never defines. RESTest says so before the run, and both operations are still held
* to the replies they declare, since every one of those was read: the 404 neither declares is
* reported.
*/
class AReplyUnreadIsNotUndeclaredTest {

Expand Down Expand Up @@ -149,6 +154,38 @@ class AReplyUnreadIsNotUndeclaredTest {
description: the profile
""";

/** A shape the document never defines, in a request of one operation and a reply of another. */
private static final String A_SHAPE_NEVER_DEFINED = """
openapi: 3.0.3
info:
title: Night animals
version: "1.0"
paths:
/moths:
get:
operationId: listMoths
parameters:
- name: kind
in: query
schema:
$ref: '#/components/schemas/NoSuchShape'
responses:
"200":
description: every moth
/larks:
get:
operationId: listLarks
responses:
"200":
description: every lark
content:
application/json:
schema:
$ref: '#/components/schemas/NoSuchShape'
components:
schemas: {}
""";

private static WireMockServer api;

@BeforeAll
Expand Down Expand Up @@ -203,14 +240,39 @@ void a_renamed_operation_is_held_to_its_document(@TempDir Path directory) throws
.describedAs("what RESTest could not read, and the new names, are said before the "
+ "run")
.contains(
"paths./bats.get: the response '404' is a reference that does not resolve",
"paths./bats.get.responses.404: the response '404' is a reference that "
+ "does not resolve",
"paths./bats.get: another operation is already called 'listOwls'; "
+ "renamed 'GET /bats' to stay unique",
"renamed 'GET /moths' to stay unique");
assertThat(codesIn(report)).containsExactly(920);
assertThat(operationsWith(920, report)).containsExactlyInAnyOrder("listOwls", "GET /moths");
}

@Test
@DisplayName("a shape RESTest cannot read, in a request or inside a reply it read, hides no "
+ "reply the document declares, so a 404 it does not declare is reported")
void a_shape_it_cannot_read_hides_no_reply(@TempDir Path directory) throws IOException {
StringWriter screen = new StringWriter();

int answer = ARun.run(A_SHAPE_NEVER_DEFINED, api.baseUrl(), directory, screen);

JsonValue.JsonObject report = report(directory.resolve("out"));
assertThat(answer)
.describedAs("an undeclared status code is a warning, which ends no run with 1 "
+ "unless asked; the run said:%n%s", screen)
.isZero();
assertThat(screen.toString())
.describedAs("what RESTest could not read is said before the run")
.contains("paths./moths.get: this operation's data includes a shape RESTest does "
+ "not fully understand yet",
"paths./larks.get: this operation's data includes a shape RESTest does "
+ "not fully understand yet");
assertThat(codesIn(report)).containsExactly(920);
assertThat(operationsWith(920, report))
.containsExactlyInAnyOrder("listMoths", "listLarks");
}

@Test
@DisplayName("an identifier a path has no place for is left out, and said, so that no request "
+ "sent as it was accepted is reported as broken data accepted, and the operation is "
Expand Down
Loading
Loading