Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ docker build --tag restest . && docker run --rm restest version # the containe
| Why the tool is shaped as it is, one decision per file | `docs/adr/` |
| Continuous integration and how to reproduce it | `docs/ci.md` |
| How a release is made and what it carries: the image, the archives and their launchers, the Homebrew formula, the modules on Maven Central, and the jbang catalog | `docs/ci.md`, `docs/adr/0032-how-a-release-is-published.md`, `docs/adr/0040-homebrew.md`, `docs/adr/0044-maven-central.md`, `docs/adr/0045-jbang.md`, `Dockerfile`, `dist/` |
| The documentation as a website, for the latest release, and the shape of `docs/` it reads, which a test holds | `isa-group/restest-site`, `docs/adr/0047-the-documentation-site.md`, `.github/workflows/documentation-site.yml`, `DocumentationSiteTest` in `restest-arch-tests` |
| The documentation as a website, for the latest release, and the shape of `docs/` it reads, which a test holds | `isa-group/restest-site`, `docs/adr/0047-the-documentation-site.md`, `.github/workflows/documentation-site.yml`, `DocumentationSiteTest` in `restest-arch-tests`; the run its first page shows, `docs/manual/images/clinic-run/`, held by `DocumentedRunTest` |
| How a change is proposed and reviewed | `CONTRIBUTING.md` and `.github/PULL_REQUEST_TEMPLATE.md` |
| The documents the evaluation platform may be named in | `BENCHMARK_MAY_BE_NAMED_IN` in `restest-arch-tests/src/test/java/io/restest/arch/SourceTreeRulesTest.java` |

Expand Down
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,10 @@ Use the template, and fill in every section. Four carry the most weight:

Titles say what became possible, not what was edited.

A pull request that changes how `report.html` looks, or the kinds of fault a run reports, says so in
its description: the pictures of the report in the manual and the README are made again for the next
release ([`docs/ci.md`](docs/ci.md#the-pictures-of-the-report)).

## Review

Continuous integration must be green ([`docs/ci.md`](docs/ci.md) says what it checks). A maintainer
Expand Down
4 changes: 2 additions & 2 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,8 +307,8 @@ how to reproduce it locally.
- **Documentation held to the tool**: the reference pages that list the command line, the settings,
the switches, the keys of `report.json` and the kinds of fault are checked against the tool on
every build, every relative link is checked to lead somewhere, the pages are held to the shape the
documentation's website reads, and the JSON Schemas of a dictionary and a plan are held to
RESTest's readers.
documentation's website reads, the run the manual pictures is held to being a whole default run of
a version numbered as a release, and the JSON Schemas of a dictionary and a plan are held to RESTest's readers.
- **Mutation testing** on the oracles — deliberately corrupt our own code and
check that the tests notice. "Our tool finds bugs" is more credible when our own oracle logic has
a measured score.
Expand Down
5 changes: 5 additions & 0 deletions docs/adr/0047-the-documentation-site.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ leads to it on GitHub at that tag. The one thing here that names the site's part
- every page has a title, a line starting with `# `, and none starts with front matter.
- **Held here by `DocumentedLinksTest`**: every Markdown link, and the heading it names, leads
somewhere; a page's own title is no such heading, since the site shows it above the page instead.
- **Held here by `DocumentedRunTest`**: the run the manual pictures, in
`docs/manual/images/clinic-run/`, is a whole default run of a version numbered as a release, not
a snapshot, with its six pictures: `report.json`, and `summary-`, `fault-` and `found-`, each
`-light.png` and `-dark.png`. The site's first page shows them and takes its figures and faults from the run's
`report.json`; a version without the folder, such as 2.4.0, has a first page without them.
- **Checked only by the site's own build**, which names the file: links in raw HTML (`src`, `srcset`,
`href`), reference-style links, links that leave the repository, a link to a heading the site
names differently from GitHub, such as one with the same words as the page's title, and two files
Expand Down
47 changes: 45 additions & 2 deletions docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ itself, in its first stage, and needs nothing the smoke run built, so the two ru
| What Maven Central gets | `dist/central.sh`, on the `ubuntu-latest` / Java 25 row and in `release.yml` | a module's Javadoc or sources that do not build; a file Central asks for missing, or a POM without a name; a module whose POM would name a library at another version than `restest-cli` runs with; a program depending on `restest-cli` getting any jar but those the release archive carries |
| End-to-end smoke run | `SmokeRunTest`, in the `smoke` job | the command failing against two real containerised APIs, or answering 2, 3 or 4 rather than "ran, and here is what I found". Runs only where a container runtime exists, and fails rather than skips when it was asked for |
| How RESTest starts | `check-restest.sh`, in the `smoke` and `image` jobs, in `release.yml` and, through `check-homebrew.sh`, in `homebrew.yml` | the image, a launcher or Homebrew's command failing to start, saying the wrong version, or answering with a number other than RESTest's own: `2` for a wrong option, `3` when nothing answers, `127` from a launcher or the image when there is no Java it can use — and anything but `0` from Homebrew's command, which brings its Java — and `143` when `docker stop` ends a run in the image; the image, or a release's folder, without the JSON Schemas of a dictionary and a plan, without the skill for coding agents, or without `THIRD-PARTY.txt`; Homebrew's formula failing `brew style` or, once a release is published, `brew audit --strict --online`, failing to install or its own test, or installing anything but the archive as it is |
| Documentation held to the tool | `DocumentedCommandLineTest`, `DocumentedSettingsTest`, `DocumentedSwitchesTest`, `DocumentedReportTest`, `DocumentedSkillTest`, `DocumentedLinksTest`, `DocumentationSiteTest` | a reference page that disagrees with the tool: the command line, its options and exit codes against the help; the settings template against `--print-settings`; a switch, a file of settings or a plan on the switches page the tool does not have or does not accept; a key of `report.json`, a kind of fault, its title, severity or advice, or a reason a reply was not checked, against what a run writes and the code; the skill for coding agents giving a command, an option, a setting, an exit code or a key of `report.json` RESTest does not have, or finding a run's summary by a line a run does not print, with a heading agents cannot read, a link into the repository, or a copy at the end of `docs/for-coding-agents.md` that differs from it; a relative link that leads nowhere, or to a page's own title; a page the documentation's website could not build: the manual's index without its numbered list of chapters, a chapter missing from it or one not starting with its link, a section of `docs/README.md` without a table of pages, a page that is no entry of either index, a page with no title, or with front matter |
| Documentation held to the tool | `DocumentedCommandLineTest`, `DocumentedSettingsTest`, `DocumentedSwitchesTest`, `DocumentedReportTest`, `DocumentedSkillTest`, `DocumentedLinksTest`, `DocumentationSiteTest`, `DocumentedRunTest` | a reference page that disagrees with the tool: the command line, its options and exit codes against the help; the settings template against `--print-settings`; a switch, a file of settings or a plan on the switches page the tool does not have or does not accept; a key of `report.json`, a kind of fault, its title, severity or advice, or a reason a reply was not checked, against what a run writes and the code; the skill for coding agents giving a command, an option, a setting, an exit code or a key of `report.json` RESTest does not have, or finding a run's summary by a line a run does not print, with a heading agents cannot read, a link into the repository, or a copy at the end of `docs/for-coding-agents.md` that differs from it; a relative link that leads nowhere, or to a page's own title; a page the documentation's website could not build: the manual's index without its numbered list of chapters, a chapter missing from it or one not starting with its link, a section of `docs/README.md` without a table of pages, a page that is no entry of either index, a page with no title, or with front matter; the run the manual pictures, and the website's first page shows, made by a snapshot build, cut short, given another time than the default minute, given a plan, settings or values, without one of its six pictures, or by a version chapter 4 does not name |
| Runs on a plain runtime | `StockJreRunTest`, in the `smoke` job | the tool reaching for a part of Java that a runtime carrying only the compulsory modules does not have, which stops it before its first request; a seed meaning a different run there than it does here; and a key left in `RESTEST_AUTH` failing to reach an API that refuses every request without it, or being written into any file the run leaves behind — the one place the variable is read from a real environment rather than handed to the command by a test. Pinned to Java 21, the oldest release supported and one of the two where the generators RESTest once named are optional; the gate refuses to run on an image where they are not |

### Coverage
Expand Down Expand Up @@ -394,7 +394,10 @@ The steps of every release, in order, each taken by a maintainer:
and the pages that name the version to install — the README's installation section and chapters 2
and 9 of the manual — with output from the image and the archive built from that commit, and, for
chapter 2's Homebrew and jbang sections, what `check-homebrew` and `check-jbang` print in that pull
request's run of `release.yml`.
request's run of `release.yml`; and [the pictures of the report](#the-pictures-of-the-report),
made again if `git diff --stat v<previous> -- restest-report/src/main/resources/io/restest/report/report.html
restest-core/src/main/java/io/restest/core/oracle/FaultCategory.java` lists either file, or a pull
request since the previous release says the page or the kinds of fault changed.
3. **The tag**, pushed once that pull request is merged and the documentation site builds from
`master` as it now is: start the site's workflow by hand in `isa-group/restest-site` (Actions →
Site → Run workflow, from `main`) and wait until it is green — its weekly run is older than that
Expand All @@ -420,6 +423,46 @@ The steps of every release, in order, each taken by a maintainer:
the `$id` of both schemas to the next release, since the schemas on `master` name the release
being prepared; `PublishedSchemasTest` requires them to be no older than that snapshot.

### The pictures of the report

The manual and the README show what `report.html` looks like with pictures of two real runs, and the
documentation site's first page shows the manual's, with figures it reads from that run's
`report.json`. They are made again when a release changes how the page looks, or the kinds of fault
a run reports. That happens in the pull request that prepares the release, with the archive built
from it, so that they show a version numbered as a release: `DocumentedRunTest` holds the manual's
run to being a whole default run of such a version, not of a snapshot, that chapter 4 names.

1. Build RESTest as the release will be (`./mvnw -q package -DskipTests && dist/assemble.sh
target/dist`); it is then `target/dist/restest-<v>/bin/restest`.
2. **The manual's run.** Start the clinic as the Web Fuzzing Dataset builds it, which runs natively
on Apple silicon as well as on Intel and AMD, unlike the image chapter 3 uses, pinned to the image
the kept run was made with:

```bash
docker run -d --name clinic -p 9967:8080 webfuzzing/wfd-petclinic-rest:FINAL@sha256:88331a1fdf244ae7bbbf5a7f3f434d17b9b8c0b3f839d193007851d213e90a88
```

Once `curl -s http://localhost:9967/petclinic/api/pettypes` prints a list of pet types, run
`restest run http://localhost:9967/petclinic/v3/api-docs` with no other option and no `RESTEST_`
variable set. Copy its `restest-out/report.json` into `docs/manual/images/clinic-run/`, and change
the version chapter 4 names.
3. **Its pictures**, from its `report.html` in Chrome, 1280 pixels wide at a device pixel ratio of 2
(Developer Tools' device toolbar), so that they are 2560 pixels wide and sharp on any screen, once
light and once dark (Rendering, "Emulate CSS media feature prefers-color-scheme"), each saved as
`<view>-light.png` and `<view>-dark.png` in the same folder:
- `summary`: the page from its top to the end of the timeline;
- `fault`: the operation where the first kind under "What was found" was first written out,
opened with that kind chosen, from its row to the end of what opened —
`jq -r '.findings[] | select(.category.title == "<that kind>") | .operation' report.json | head -1`
names it;
- `found`: the section "What was found".
4. **The README's**: the pet shop's ten-second run that [the README](../README.md#your-first-run)
gives, and the top of its page to the end of the timeline, as
`docs/images/petstore-report-light.png` and `-dark.png`.

Developer Tools' "Capture node screenshot" takes one element, and "Capture full size screenshot" the
whole page, to cut.

### Homebrew

`brew install isa-group/tap/restest` reads RESTest's formula from the repository
Expand Down
Binary file modified docs/images/petstore-report-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/petstore-report-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
39 changes: 33 additions & 6 deletions docs/manual/04-reading-what-a-run-says.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,22 @@ Read it from the top:

`report.html`, beside `report.json`, is the same run as a page to open in a browser: from what a
build kept, from an e-mail, on a laptop with no network. It is one file, needs nothing beside it,
fetches nothing, and shows nothing that `report.json` does not hold. The clinic's first run, from
[chapter 3](03-a-first-run.md#the-first-run), opens like this:
fetches nothing, and shows nothing that `report.json` does not hold. Here is the clinic as the [Web
Fuzzing Dataset](https://github.com/WebFuzzing/Dataset) builds it, as an image for Apple silicon as well
as Intel and AMD, tested by RESTest 2.4.0 for the minute it takes when no `--budget` is given, with
nothing but its document. To make such a run yourself:

![The top of report.html for a run of the clinic: under the header and the menu, four cards counting the replies by class of status code, the errors and the warnings by kind, and the operations by what was found on them](images/the-page.png)
```bash
docker run -d --name clinic -p 9967:8080 webfuzzing/wfd-petclinic-rest:FINAL@sha256:88331a1fdf244ae7bbbf5a7f3f434d17b9b8c0b3f839d193007851d213e90a88
restest run http://localhost:9967/petclinic/v3/api-docs
```

The page opens like this:

<picture>
<source media="(prefers-color-scheme: dark)" srcset="images/clinic-run/summary-dark.png">
<img src="images/clinic-run/summary-light.png" alt="The top of report.html for a one-minute run of the clinic: under the header and the menu, four cards counting the replies by class of status code, the errors and the warnings by kind, and the operations by what was found on them, then the timeline of the run">
</picture>

From the top:

Expand All @@ -124,10 +136,22 @@ From the top:
a search. Open one to see how its replies divided and, for each kind of fault found on it, the
first request of that kind written out whole and the reply to it. **Copy curl** copies the command
that sends the request again; **Copy as issue** copies a title, the request, the reply and the
command, in Markdown, ready for an issue tracker.
command, in Markdown, ready for an issue tracker. Here one is open at its first kind of fault:

<picture>
<source media="(prefers-color-scheme: dark)" srcset="images/clinic-run/fault-dark.png">
<img src="images/clinic-run/fault-light.png" alt="An operation opened in report.html: how its replies divided by class of status code, a button for each kind of fault found on it, and the first fault of the first kind written out whole, the request beside the reply, with Copy curl and Copy as issue above them">
</picture>

- **What was found**: each kind of fault, with its number in the catalogue testing tools share —
point at the number, or reach it with the keyboard, to read what it means — what to look at, in
which classes of status code it came back, and the operations it was found on.
which classes of status code it came back, and the operations it was found on:

<picture>
<source media="(prefers-color-scheme: dark)" srcset="images/clinic-run/found-dark.png">
<img src="images/clinic-run/found-light.png" alt="What was found in report.html for the same run: each kind of fault, errors first, with its number in the catalogue, what to look at, the classes of status code it came back in, how many times and on how many operations">
</picture>

- **What could not be tested or read**: operations no request went to, replies no check could judge,
parts of the document that could not be read, and keys the document asks for that nobody gave.
- **How it was run**: the number the run ended with and what it means, as [chapter
Expand All @@ -142,7 +166,10 @@ report.html=false` ([the settings](../settings.md#report)).

`report.json` holds all of it for a program to read. [`jq`](https://jqlang.org/), a small
command-line tool for JSON, answers most questions in one line. These assume the report of the last
run is in `restest-out/`.
run is in `restest-out/`; the answers shown are those of an earlier run of the clinic, so yours will
differ. To try them without running RESTest, put the report of the run pictured above,
[`docs/manual/images/clinic-run/report.json`](images/clinic-run/report.json), in place of
`restest-out/report.json`.

**Which operations broke, and how often?**

Expand Down
Binary file added docs/manual/images/clinic-run/fault-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/manual/images/clinic-run/fault-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/manual/images/clinic-run/found-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/manual/images/clinic-run/found-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/manual/images/clinic-run/report.json

Large diffs are not rendered by default.

Binary file added docs/manual/images/clinic-run/summary-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/manual/images/clinic-run/summary-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/manual/images/the-page.png
Binary file not shown.
Loading
Loading