Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 137 additions & 0 deletions .github/workflows/documentation-site.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Moves the documentation site, isa-group/restest-site, on to a release once it is published, so that
# https://isa-group.github.io/restest-site/docs/ documents the version people install. ADR-0047 says
# why it is shaped this way; docs/ci.md says how to use it.
#
# version checks the tag, and that it is a published release rather than a draft
# push writes the tag into the site's restest-version, never over a higher version
# check-published waits until the site, which rebuilds itself on that commit, documents the version
#
# Started by hand from master with a tag, it moves the site on to a release already published, one
# made before this workflow existed included, and is how a failed run is repeated.
#
# Every action is pinned to a commit SHA, with the version in a trailing comment, for the reasons
# ci.yml gives.
name: Documentation site

on:
release:
types: [published]
workflow_dispatch:
inputs:
tag:
description: The published release the site is to document, such as v2.5.0
required: true

# One run at a time, never cancelled half-way, so that two never both read the site's version before
# either writes.
concurrency:
group: documentation-site
cancel-in-progress: false

permissions:
contents: read

defaults:
run:
shell: bash

jobs:
version:
name: Check the release
runs-on: ubuntu-latest
outputs:
tag: ${{ steps.what.outputs.tag }}
version: ${{ steps.what.outputs.version }}
steps:
# The tag reaches the script through the environment and is checked before it is used: never
# pasted into a script, where a name chosen to be a command would run. The whole value is
# matched, not each of its lines: a tag given by hand could hold a newline, and with it a second
# line of the job's outputs. A draft is no release for the site, which reads the tag's files.
- name: Work out which release
id: what
env:
TAG: ${{ inputs.tag || github.event.release.tag_name || '' }}
GH_TOKEN: ${{ github.token }}
run: |
[[ $TAG =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] \
|| { echo "::error::$TAG is not a version tag such as v2.5.0"; exit 1; }
draft=$(gh release view "$TAG" --repo isa-group/RESTest --json isDraft --jq .isDraft) \
|| draft=missing
[ "$draft" = false ] \
|| { echo "::error::$TAG is not a published release of RESTest"; exit 1; }
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"

# The deploy key writes to isa-group/restest-site and nothing else. It is kept in the environment
# restest-site, which only master and version tags may use, so a workflow pushed to another branch
# cannot read it. While it is loaded, this job checks out nothing of RESTest and runs only git.
push:
name: Write the version into the site
needs: version
runs-on: ubuntu-latest
environment: restest-site
outputs:
pushed: ${{ steps.write.outputs.pushed }}
env:
TAG: ${{ needs.version.outputs.tag }}
steps:
- name: Check out the site
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: isa-group/restest-site
ssh-key: ${{ secrets.RESTEST_SITE_KEY }}
path: site

# restest-version holds one tag with its v, such as v2.4.0, the only form the site reads. The
# versions are compared without the v, which sort -V would otherwise rank above any number. A
# lower version is never written over a higher one; the same one is not written again. A push
# made with a deploy key, unlike one made with the workflow's own token, starts the site's
# workflow, which builds the site from the new tag and publishes it.
- name: Write the version, never over a higher one
id: write
working-directory: site
run: |
current=$(tr -d '[:space:]' 2> /dev/null < restest-version || true)
highest=$(printf '%s\n' "${current#v}" "${TAG#v}" | sort -V | tail -n 1)
if [ "$highest" != "${TAG#v}" ]; then
echo "::notice::The site documents RESTest $current, higher than $TAG, and is left as it is"
echo "pushed=no" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$current" = "$TAG" ]; then
echo "::notice::The site documents RESTest $TAG already"
echo "pushed=already" >> "$GITHUB_OUTPUT"
exit 0
fi
printf '%s\n' "$TAG" > restest-version
git add restest-version
git -c user.name="github-actions[bot]" \
-c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit --quiet --message "restest-version $TAG"
git push --quiet origin HEAD
echo "pushed=yes" >> "$GITHUB_OUTPUT"

# Also when the site held this very version already, which is how a run whose check failed is made
# again; not when it held a higher one.
check-published:
name: Check that the site documents it
needs: [version, push]
if: needs.push.outputs.pushed == 'yes' || needs.push.outputs.pushed == 'already'
runs-on: ubuntu-latest
env:
VERSION: ${{ needs.version.outputs.version }}
PUBLISHED: https://isa-group.github.io/restest-site/llms.txt
steps:
# The site builds and publishes itself within a few minutes of the push, and GitHub Pages'
# servers may keep the file they had for ten minutes more. The sentence is the one the site's
# llms.txt opens with. Should the site fail to build the version, its own run says why.
- name: Wait until the site documents the new version
run: |
for _ in $(seq 30); do
if curl -fsSL "$PUBLISHED" | grep -qF "The documentation of RESTest $VERSION,"; then
exit 0
fi
sleep 30
done
echo "::error::After fifteen minutes $PUBLISHED does not document RESTest $VERSION; the site's own run, in isa-group/restest-site, says why"
exit 1
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,7 @@ docker build --tag restest . && docker run --rm restest version # the containe
| Why the tool is shaped as it is, one decision per file | `docs/adr/` |
| Continuous integration and how to reproduce it | `docs/ci.md` |
| How a release is made and what it carries: the image, the archives and their launchers, the Homebrew formula, the modules on Maven Central, and the jbang catalog | `docs/ci.md`, `docs/adr/0032-how-a-release-is-published.md`, `docs/adr/0040-homebrew.md`, `docs/adr/0044-maven-central.md`, `docs/adr/0045-jbang.md`, `Dockerfile`, `dist/` |
| The documentation as a website, for the latest release, and the shape of `docs/` it reads, which a test holds | `isa-group/restest-site`, `docs/adr/0047-the-documentation-site.md`, `.github/workflows/documentation-site.yml`, `DocumentationSiteTest` in `restest-arch-tests` |
| How a change is proposed and reviewed | `CONTRIBUTING.md` and `.github/PULL_REQUEST_TEMPLATE.md` |
| The documents the evaluation platform may be named in | `BENCHMARK_MAY_BE_NAMED_IN` in `restest-arch-tests/src/test/java/io/restest/arch/SourceTreeRulesTest.java` |

Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,9 @@ the faults, the plan, dictionaries, the JSON Schemas of a dictionary and a plan,
switches, coding agents and the known limitations — and, for working on RESTest itself, the design,
continuous integration and the decision records.

The same pages, for the latest release and with search, are at
https://isa-group.github.io/restest-site/docs/.

## Acknowledgements

RESTest names the faults it finds by the catalogue of
Expand Down
5 changes: 3 additions & 2 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,8 +306,9 @@ how to reproduce it locally.
is measured on every run and reported, so that a change that makes the loop wait shows in it.
- **Documentation held to the tool**: the reference pages that list the command line, the settings,
the switches, the keys of `report.json` and the kinds of fault are checked against the tool on
every build, every relative link is checked to lead somewhere, and the JSON Schemas of a dictionary
and a plan are held to RESTest's readers.
every build, every relative link is checked to lead somewhere, the pages are held to the shape the
documentation's website reads, and the JSON Schemas of a dictionary and a plan are held to
RESTest's readers.
- **Mutation testing** on the oracles — deliberately corrupt our own code and
check that the tests notice. "Our tool finds bugs" is more credible when our own oracle logic has
a measured score.
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ One page per subject. The [README](../README.md) at the root of the repository i
makes a first run, and [the manual](manual/README.md) is read from beginning to end. These pages are
what to look things up in afterwards.

## Using RESTest
## Reference

| Page | What it covers |
|---|---|
Expand Down
8 changes: 8 additions & 0 deletions docs/adr/0031-the-manual-is-markdown-only.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,11 @@ or by hand, and a release carries none.
has no release workflow yet to put it in.
- **One HTML page instead of a PDF.** The same pipeline with a different output, and a documentation
site would make it redundant.

## Amendment (5 November 2026): a second rendering after all

**The documentation is also a website**, built from these same files at a release's tag, as §3 left
room for ([ADR-0047](0047-the-documentation-site.md)). The manual so has two renderings, GitHub's
and the site's, and whatever GitHub draws the site has to draw too. A test holds the pages to the
shape the site reads, and the site is built every week from `master`, so that the second rendering
is checked before a release rather than after it.
125 changes: 125 additions & 0 deletions docs/adr/0047-the-documentation-site.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# ADR-0047: The documentation is a website, built from a released version in a repository of its own

**Status:** Accepted
**Date:** 2026-11-05

## Context

RESTest's documentation is Markdown in this repository: the README, the manual, a page per subject,
the design and the decision records ([ADR-0031](0031-the-manual-is-markdown-only.md)). GitHub shows
each file well, but it offers no search across the pages and no menu, and what it shows by default
is `master`, which describes the next version rather than the one people install. There is no
address for the documentation as a whole that the README, the skill for coding agents or a paper
can give.

ADR-0031 §3 left room for a site: "A documentation site, if one is built, reads the same files."

## Decision

### 1. A site of its own, in `isa-group/restest-site`

The documentation is published as a website by the repository `isa-group/restest-site`, built with
Astro and its documentation theme, Starlight, and published by GitHub Pages at
`https://isa-group.github.io/restest-site/docs/`. Its search runs in the browser, with no service,
and `/llms.txt` lists the Markdown of every page in its menu, for coding agents. The same site is to hold a landing
page and a showcase.

RESTest's build knows nothing of the site: no Node, no npm, nothing more for a contributor to
install. The site's repository stays public while the organisation is on GitHub's free plan, under
which GitHub Pages publishes public repositories only.

### 2. It reads RESTest's files as they are, at a release's tag

Before each build the site takes RESTest at a version's tag and reads the Markdown under `docs/`,
`README.md`, `AGENTS.md` and `CONTRIBUTING.md`. Nothing is copied into
RESTest, and the pages are not written for the site: each page's first `#` heading becomes its
title, links to other pages lead to the site's pages, images are copied, and a link to any other file
leads to it on GitHub at that tag. The one thing here that names the site's parts is
`docs/README.md`: its sections are the site's groups (§6).

### 3. What it depends on

- **Held here by `DocumentationSiteTest`**, on every pull request:
- `docs/manual/README.md` has a numbered list, each item starting with a link to a page, and
every chapter of `docs/manual/` is in it;
- `docs/README.md` has `##` sections, each with a table — a header, the line of dashes under it
and at least one row — and each row's first cell links a page;
- every page is an entry of one of the two indexes — a list item's first link, or a row's — but
the two indexes, the README and the decision records, which have an index of their own; a link
in an index's prose is no entry;
- every page has a title, a line starting with `# `, and none starts with front matter.
- **Held here by `DocumentedLinksTest`**: every Markdown link, and the heading it names, leads
somewhere; a page's own title is no such heading, since the site shows it above the page instead.
- **Checked only by the site's own build**, which names the file: links in raw HTML (`src`, `srcset`,
`href`), reference-style links, links that leave the repository, a link to a heading the site
names differently from GitHub, such as one with the same words as the page's title, and two files
that would be published at one address, such as `AGENTS.md` and `docs/agents.md`, or two chapters
of the manual whose names differ only by their number.
- **Also relied on:** the `<img>` of `RESTLogo_Black.png`, at the root, is left out by that file name,
since the site's header shows the logo already; a `<picture>` with a source for a dark colour scheme
follows the theme the reader picks; and whatever GitHub draws, Mermaid diagrams included, the site
has to draw too, which is the site's work rather than a rule for the pages.
- **Between the two repositories:** the site's file `restest-version` holds the tag the site
documents, with its `v` (`v2.4.0`), the only form it reads; and the site's `llms.txt` opens with
"The documentation of RESTest 2.4.0,", the sentence the workflow of §5 waits for.

### 4. The latest release, not `master`

The site documents what people install. What is merged into `docs/` reaches the site with the next
release, and until then GitHub shows it. Every week the site is also built from `master`, without
publishing it, so that a page the next version would not build with is known before its tag.

### 5. Moved on by a release's publication

`.github/workflows/documentation-site.yml`, started by a release's publication or by hand with a tag,
checks that the tag is a published release rather than a draft, writes it into the site's
`restest-version` — never over a higher version, and not again when it is there already — and waits
up to fifteen minutes for the site's `llms.txt` to name it. It writes with a **deploy key**, kept in
an environment of RESTest's, `restest-site`, which only `master` and version tags may use, as
Homebrew's tap and the jbang catalog are ([ADR-0040](0040-homebrew.md) §5,
[ADR-0045](0045-jbang.md) §5). A push made with a deploy key, unlike one made with a workflow's own
token, starts the site's workflow, which builds the site from the new tag and publishes it.

A tag never changes. Should a release's pages not build on the site, **the site is changed until it
builds that tag**; `restest-version` is never put back to an earlier version, and the site publishes
nothing new meanwhile (maintainer). This is why, before a version is tagged, the site's build from
`master` has to be green ([Continuous integration](../ci.md#releases)).

### 6. The menu is named by the indexes

The site's menu has the manual's chapters, in their order and with their numbers, and then a group for
each `##` section of `docs/README.md`, named by it, with a row for each page (maintainer). That is why
the first section is called "Reference".

## Consequences

- A page added to `docs/` needs an entry in an index, and a chapter one in the manual's list, or
RESTest's build fails, saying which.
- A change to `docs/` is on the site with the next release, not when it is merged.
- A second rendering to check, which ADR-0031 counted against a PDF of the manual. What keeps it in
step is `DocumentationSiteTest` on every pull request, and the site's weekly build from `master`.
- Two repositories and a key between them: the environment, the deploy key and its secret are set up
once, by an owner of the organisation ([Continuous integration](../ci.md#the-documentation-site)).
- The address is GitHub's until the project has a domain of its own. GitHub then sends the old
address on to the new one, and this record, the README, the skill and the workflow change with it.

## Alternatives considered

- **The site in this repository**, built from the same commit as the pages, with no second repository
and no key: but Node and npm in a repository built with Maven, which every contributor would meet,
and the site's showcase, which shows what RESTest finds on real APIs, is built from measurements
made outside this repository, which keeps none (principle 9, [ADR-0011](0011-evaluation-harness.md)).
- **The site's own build, run on every pull request of RESTest**, in place of
`DocumentationSiteTest`: one set of rules rather than two that may drift apart, and the checks that
only the site makes would come before a merge too; but Node in RESTest's continuous integration,
and every pull request here depending on the site's `main`, so that a change to the site could
fail a pull request that changed no page.
- **GitHub alone**, as now: no search across pages, no menu, no `llms.txt`, and unreleased
documentation by default.
- **`release.yml` telling the site to rebuild**, with a token: `release.yml` only drafts a release, so
the site would have built the version before it.
- **The site asking GitHub which release is the latest**, at each build: the same commit of the site
would build different pages on different days, and the version it documents would be written
nowhere.
- **`repository_dispatch` with a personal token**: a token that expires, held by one person.
- **Publishing `master`**: the documentation would describe what no release does yet.
Loading
Loading