Skip to content

Ask a coding agent to follow the page for coding agents on the documentation site, in the same words everywhere - #441

Merged
ssegura merged 3 commits into
masterfrom
feat/m12-7-agent-prompt
Nov 8, 2026
Merged

ssegura merged 3 commits into
masterfrom
feat/m12-7-agent-prompt

Conversation

@ssegura

@ssegura ssegura commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Related: isa-group/restest-site#4, the documentation site's first page, which shows the sentence this pull request gives, read from the page for coding agents; #440, the run that page shows. This pull request is merged while 2.5.0 is prepared, before its tag: the address it gives answers only once 2.5.0 is published.

What you can do now that you could not before

Give a coding agent two sentences — "Use RESTest to test the REST APIs you develop in this project. Follow the setup and usage instructions at https://isa-group.github.io/restest-site/docs/for-coding-agents/" — and have it read, on the documentation site, the page for coding agents of the version it will install, rather than the page as master has it. The README, that page and the site's first page give the same sentence, and the build fails if the README and the page ever differ.

Try it yourself

With Java and git, from a clean clone:

git clone --branch feat/m12-7-agent-prompt https://github.com/isa-group/RESTest.git restest-agent-prompt
cd restest-agent-prompt
sed -n '/^## Asking your agent/,/^```$/p' docs/for-coding-agents.md | tail -4
./mvnw verify -pl restest-arch-tests -am -Dtest='io.restest.arch.DocumentedSkillTest' \
  -Dsurefire.failIfNoSpecifiedTests=false | grep -E 'Tests run:.*in io|BUILD'
Two sentences are enough:

```
Use RESTest to test the REST APIs you develop in this project. Follow the setup and usage instructions at https://isa-group.github.io/restest-site/docs/for-coding-agents/
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.385 s -- in io.restest.arch.DocumentedSkillTest
[INFO] BUILD SUCCESS

Now have the README say something else than the page:

sed -i.bak 's/develop in this project\. Follow/develop. Follow/' README.md && rm README.md.bak
./mvnw verify -pl restest-arch-tests -am -Dtest='io.restest.arch.DocumentedSkillTest' \
  -Dsurefire.failIfNoSpecifiedTests=false | grep -E '^\[ERROR\]   DocumentedSkillTest|BUILD'
[ERROR]   DocumentedSkillTest.the_readme_asks_as_the_page_does:199 [what the README tells an agent under "From a coding agent" is what docs/for-coding-agents.md tells it under "Asking your agent", which the website's first page shows too: change the two together]
[INFO] BUILD FAILURE

Undo it with git checkout -- README.md. The address itself answers once 2.5.0 is published; today the site documents 2.4.0, which has no page for coding agents.

How it works

The README, under "From a coding agent", and the page for coding agents, under "Asking your agent", each give a person a sentence to paste into their agent. It used to point at the page on GitHub as master has it, which may describe options the version the agent installs does not have yet. It now asks the agent to follow the setup and usage instructions on the documentation site, which documents the latest release, the one Homebrew, jbang and the image install; the page tells the agent how to install RESTest and the skill, and ends with the skill itself.

The sentence is now given in three places: the README, the page, and the documentation site's first page, which reads it from the page at the version it documents, so the two on the site cannot differ. A test holds the README to the page: the first block of code under each heading must be the same, or the build fails saying where to change it.

What changed in the code

  • README.md and docs/for-coding-agents.md — the new sentence, "two sentences" rather than "one", and "the second can go" once the skill is installed.
  • restest-arch-tests/.../DocumentedSkillTest.java — the_readme_asks_as_the_page_does, with firstBlockUnder(page, heading), which reads the first block of code under a heading and stops at the next one.
  • docs/adr/0047-the-documentation-site.md §3 — the site's first page relies on that block of the page, held by the test.
  • docs/ci.md — the table of gates says what the test now catches.

Evidence

  • Tests: one added to DocumentedSkillTest, now 10; it fails when the README's sentence differs from the page's, as shown above. Architecture and documentation rules: all green. — the checks of this pull request
  • Architecture tests: none weakened.
  • Mutation score: not applicable; no oracle or constraint touched.
  • Per-request overhead: not applicable.
  • Smoke run: not applicable.
  • Idle time: not applicable.
  • Artefact: the documentation site's first page built from this branch shows the same sentence under "Coding agent" and in its band for coding agents (The first page: what RESTest is and how to start, with a real run, for humans and for agents restest-site#4).

Decisions taken

  • The page on the documentation site, not the skill, and not the page on GitHub (maintainer): the page gives both the setup — installing RESTest and the skill, so that a later session needs no address — and the use, the skill word for word at its end; and on the site it describes the version the agent installs, where master may describe the next one.
  • A pull request of its own, merged with 2.5.0 (maintainer): until 2.5.0 is published, the site documents 2.4.0 and the address answers "page not found".
  • The site reads the sentence from the page rather than keeping a copy, and RESTest holds the README to the page, so one sentence is changed in one place and checked in the other.
  • restest --help still names the page on GitHub at the running version's tag, which is right for the version a person has; ADR-0046, a record, still speaks of the one sentence it decided then.

Open questions


  • Targets master
  • English throughout, including comments and test names
  • The hard rules in AGENTS.md hold

🤖 Generated with Claude Code

ssegura and others added 3 commits November 6, 2026 15:12
…ite, for the version it installs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… coding agents, which the site shows too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura
ssegura merged commit 50f1e91 into master Nov 8, 2026
11 checks passed
@ssegura
ssegura deleted the feat/m12-7-agent-prompt branch November 8, 2026 14:54
ssegura added a commit that referenced this pull request Nov 10, 2026
…n at one address (#442)

**Related:** None. This is the release of 2.5.0, with what `master`
holds on 10 November: #432 to #441. **This pull request is merged before
the tag `v2.5.0`, which goes on its merge commit.**

## What you can do now that you could not before

- **Get 2.5.0.** Once this is merged and tagged, it installs five ways:
  - `brew install isa-group/tap/restest`;
- `jbang app install
restest@https://isa-group.github.io/jbang-catalog/jbang-catalog.json`,
new in this version, with no Java installed first;
  - `docker run ghcr.io/isa-group/restest:2.5.0`;
  - the release's archives, with Java alone;
  - or built from the source.
- **What 2.5.0 adds:**
- RESTest runs with jbang on macOS, Linux and Windows, which downloads a
Java when there is none (#434);
- every archive carries a skill that teaches a coding agent to use
RESTest, `skills/restest/SKILL.md` (#437), and the page for coding
agents gives the sentence to hand one, the same in the README (#441);
- the documentation can be read and searched at one address, moved on to
each release once it is published (#439);
- installing takes a few lines, with the details in the manual (#435),
and the README and the architecture's diagrams are simpler (#438);
- the manual shows `report.html` as a real run of the clinic draws it,
light and dark, with that run's `report.json` beside it (#440);
- a fix: an API that answers in HTTP/1.0, as Python's `http.server`
does, gets a reply read for every request (#436).
- **What stays the same as in 2.4.0:** the command line, its exit codes,
plans, dictionaries, settings and both reports.

## Try it yourself

From a clean clone, with only Java and git:

```bash
git clone --quiet --branch release/2.5.0 https://github.com/isa-group/RESTest.git restest-2.5.0 && cd restest-2.5.0
./mvnw -q install -DskipTests
./restest version
./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://127.0.0.1:1 --budget 3s; echo "exit $?"
ls restest-out
grep -o '"verdict" *: *"[A-Za-z]*"' restest-out/report.json
dist/assemble.sh ../release --archives > /dev/null && ls ../release
tar -xzf ../release/restest-2.5.0.tgz -C ../release && ../release/restest-2.5.0/bin/restest version
ls ../release/restest-2.5.0/skills/restest
```

Below is the real output, from a fresh clone of this branch on 10
November; the Java line will be your own. Nothing listens on port 1, so
the run ends with `3`, and it still writes both files; the archive
carries the skill:

```
RESTest 2.5.0
Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64
RESTest testing Pet Shelter at http://127.0.0.1:1

4 of 4 operations can be tested, seed 2125531189854063678, budget 3s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent

64 requests to 4 operations in 361ms, 90% of it idle
  opening lap: 4 requests in 46ms, 0 of 4 operations answered 2xx
  64 no reply
  1 of them were pushing at the API with values nobody sensible would send
  3 creation(s) meant to begin a series about a thing the run created were not accepted, so no series began
no faults found
report written to restest-out/report.json (9.6 KiB)
page written to restest-out/report.html (90.7 KiB)
the run itself was not kept; pass --store to keep every request and reply
restest: nothing at http://127.0.0.1:1 answered any of the 64 requests, so the API was never actually tested. Check the address, and that it is running.
exit 3
report.html
report.json
"verdict":"nothingAnswered"
restest-2.5.0
restest-2.5.0.tgz
restest-2.5.0.zip
RESTest 2.5.0
Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64
SKILL.md
```

With Docker as well, the image is built from the same clone, as the
release builds it:

```bash
docker build --quiet --tag restest:2.5.0 . > /dev/null && docker run --rm restest:2.5.0 version
```

```
RESTest 2.5.0
Java 21.0.13 (Eclipse Adoptium), Linux 7.0.14-linuxkit aarch64
```

What the tag will leave on the Central Portal, laid out from the same
clone:

```bash
./mvnw -q package -Pcentral -DskipTests -pl restest-cli -am && dist/central.sh target/central
ls target/central/maven-central/es/us/isa/restest
grep -c "<dependency>" target/central/maven-central/es/us/isa/restest/restest-cli/2.5.0/restest-cli-2.5.0.pom
ls restest-cli/target/lib | wc -l
```

```
/…/restest-2.5.0/target/central/maven-central
restest-cli
restest-core
restest-exec
restest-gen
restest-oracles
restest-report
restest-spec
restest-store
70
      70
```

## How it works

**The version.** `-SNAPSHOT` is how Maven writes "on the way to this
version", and this pull request removes it from every `pom.xml`. After
the merge, the tag `v2.5.0` on the merge commit starts the release
workflow ([`docs/ci.md`](docs/ci.md#releases)):
- it builds the image and the archives, and checks them on three
operating systems and two processors;
- it installs the Homebrew formula for the archive on macOS and Linux,
and runs the jbang catalog for this version;
- it signs the eight modules and uploads them to the Central Portal,
once a maintainer approves the `central` job; the Portal validates them
and publishes nothing;
- it pushes the image, and leaves a draft release for a maintainer to
write the notes in and publish.

A maintainer publishes the deployment on the Portal before publishing
the release. Publishing the release starts the Homebrew workflow, which
writes 2.5.0 into the tap, the jbang workflow, which writes it into the
catalog, and the documentation site's, which moves the site on to 2.5.0.
Before the tag, the documentation site is built by hand from `master`,
so that a tag the site cannot build is never pushed.

**The base images.** Step 1 of every release moves the `Dockerfile`'s
two base images to the newest patch; they are the newest already, so the
`Dockerfile` does not change, nor the Java the image prints.

**The pages.** The README and chapters 2 and 9 of the manual name the
version to install, and now name 2.5.0. Every other page was written for
2.5.0 as each change landed, from #432 to #441. Those that give a
minimum version (the skill's "2.4.0 or later"), speak of a past release,
or name the version the manual's kept run was made with (chapter 4's,
2.4.0) stay as they are.

## What changed in the code

No Java code changed, and nothing under `src/`.

- **Version:** every `pom.xml`, from `2.5.0-SNAPSHOT` to `2.5.0`, ten
files. The JSON Schemas' `$id` already names `v2.5.0` since #432.
- **`README.md`:** the Docker command runs `2.5.0`.
- **`docs/manual/02-installing.md`:** Homebrew's and jbang's `restest
version`, Docker, the archive, the checksums, the source and Updating
name 2.5.0. The Docker and archive outputs are those above, unchanged
but for the version. Homebrew's and jbang's lines and sizes are what
this pull request's [run of
`release.yml`](https://github.com/isa-group/RESTest/actions/runs/42861198585)
printed, as `docs/ci.md` asks: `check-homebrew` installed 628 MB on
macOS and 824 MB on Ubuntu (625 and 823 before) and printed `Java
21.0.13.0.3 (Homebrew), Mac OS X 26.7.2 aarch64`; `check-jbang` started
RESTest on Java 27.0.1.0.2, where the page said `27`, with 636 MB of
Javas, where it said some 600. jbang's libraries, 37 MB, stay as they
are: `check-jbang` counts the sources and the Javadoc Maven Central
gets, which jbang does not download, and the published release's run of
`jbang.yml` is what they are checked against.
- **`docs/manual/09-exit-codes-and-ci.md`:** the workflow fetches and
runs the 2.5.0 archive.

## Evidence

- [x] **Tests:** `./mvnw verify` is green at `edfb1510`, with version
2.5.0: 4,981 tests, 0 failures, BUILD SUCCESS in 2 min 36 s. 162 of them
are the architecture and documentation rules, `PublishedSchemasTest`
among them, which on a release requires the schemas' `$id` to name it;
they pass again with chapter 2's sizes. The first push's
[Release](https://github.com/isa-group/RESTest/actions/runs/42861198585)
run is green, `central` and `publish` skipped, as on every pull request.
On the second push,
[CI](https://github.com/isa-group/RESTest/actions/runs/42862469536)'s
*Java 26 on ubuntu-latest* failed on its first attempt in
`RunLoopTest.an_operation_told_to_wait_is_skipped`, which counted
`getPet`'s wait twice where it expects once: a wait the API asks for
that lasts past the end of the budget ends at the budget's end, and a
request sent at that very moment is let through and refused again,
beginning a second wait. The row passed when re-run, with the two rows
the failure had cancelled, and every check is green. Nothing under
`src/` changes here; the race is put right in a pull request of its own,
after this release.
- [x] **Architecture tests:** unchanged, none weakened.
- [x] **Mutation score:** no oracle or constraint touched.
- [x] **Per-request overhead:** unchanged; nothing under `src/` changed.
- [x] **Smoke run:** unchanged, for the same reason; CI runs it on this
pull request.
- [x] **Idle time:** unchanged.
- [x] **Artefact:** `check-restest.sh 2.5.0` passes on both the archive
and the image built here, every check `ok`: `--launcher` on
`restest-2.5.0/bin/restest`, and `--image restest:2.5.0`, built again
without Docker's cache. `dist/central.sh` passes on the build above;
`restest-cli`'s POM names the 70 libraries the archive's `lib/` carries
beside `restest-cli` itself.

## Decisions taken

- **The pictures of the report are not made again.** Neither
`report.html` nor `FaultCategory.java` changed since v2.4.0, and #440,
which added the manual's kept run, says they are made again only when
the page or the kinds of fault change. Chapter 4 keeps naming 2.4.0, the
version that run was made with.
- **No release evaluation on the Web Fuzzing Dataset in this pull
request.** 2.5.0 changes no request but HTTP/1.0 replies (#436),
measured in its own pull request. The showcase campaign of 2.5.0, one
minute per API on the dataset's 52 APIs, runs once the release is
published.
- **Guava is not moved in this release** (#430, Dependabot alert 48,
raised in 2.4.0's release): the maintainer decided it waits. RESTest's
own code uses neither Guava nor Java serialisation.
- **The release notes are not in this pull request.** They go into the
draft release after the tag, as `docs/ci.md` says.

## Open questions

None. After the merge, each step taken by a maintainer: the
documentation site built by hand from `master`; the tag `v2.5.0` on the
merge commit, and the approval of the `central` job; the notes in the
draft; the deployment published on the Central Portal, then the release;
the installation commands tried from outside; and the pull request that
moves `master` to `2.6.0-SNAPSHOT`.

---

- [x] Targets `master`
- [x] English throughout, including comments and test names
- [x] The hard rules in `AGENTS.md` hold

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant