From 3150478192bf28dff8181ff01e35353e0a1733aa Mon Sep 17 00:00:00 2001 From: Sergio Segura Date: Tue, 13 Oct 2026 22:34:12 +0200 Subject: [PATCH 1/4] Give a run a size, and the command's tests a size instead of a second of the clock A run can now be told to stop once it has sent so many requests: two settings in the schedule group, `stopAfterRequests` (off) and `requestsBeforeStopping` (1000). On, the loop stops asking for more at that count and waits for the answers it is owed, as it does at a deadline; the run ends as it was meant to, not cut short. A run of a known size sends the same requests on a slow machine as on a fast one, which is what RESTest's own tests of the command line needed: 65 of their runs were given a second or two of the clock, and twice this week the slowest Windows machine continuous integration runs on spent that second starting up and answered 3, "nothing was tested" (AuthCommandTest, on master). Fifteen test classes now give their runs a size; the module takes 1 min 20 s instead of 3 min. Two constants of the command move to the same group, by the rule of ADR-0025: how long a run stopped from outside may spend writing what it found once it has stopped waiting for its answers (`interruptWritingGrace`, 5 s) and how long it then gives its directory to say what was written (`interruptLookingGrace`, 1 s). ARunCutShortTest sets the first to a second and the second to ten: the test took eight seconds and failed on a slow Windows machine whose directory did not answer within the one second it was given (the other red run on master this week); it now takes one second and does not depend on the machine. The settings page, the switches page (a second switch that is not a lever) and the command-line page say so; ADR-0025 records the four settings. RunLoopTest proves a run of twelve requests sends twelve, waits for their answers and is not cut short. Co-Authored-By: Claude Fable 5.1 --- docs/adr/0025-settings.md | 24 ++++++ docs/command-line.md | 13 +-- docs/settings.md | 12 +++ docs/switches.md | 3 +- .../main/java/io/restest/cli/RunCommand.java | 44 +++++----- .../src/main/java/io/restest/cli/RunLoop.java | 17 +++- .../cli/ABrokenRunAnswersFourTest.java | 27 ++++--- .../restest/cli/AKeyIsWrittenNowhereTest.java | 11 ++- .../java/io/restest/cli/ARunCutShortTest.java | 19 +++-- .../java/io/restest/cli/AuthCommandTest.java | 14 ++-- .../cli/BodiesOfTheWrongShapeTest.java | 10 ++- .../io/restest/cli/CampaignCommandTest.java | 35 +++++--- .../restest/cli/CheckSwitchesCommandTest.java | 11 ++- .../FuzzingFindsWhatNominalMissesTest.java | 22 +++-- ...tationFindsWhatNothingElseReachesTest.java | 10 ++- .../io/restest/cli/OpeningLapCommandTest.java | 21 +++-- .../test/java/io/restest/cli/RestestTest.java | 80 +++++++++++++------ .../test/java/io/restest/cli/RunLoopTest.java | 60 +++++++++++--- ...quencesFindWhatSingleRequestsMissTest.java | 10 ++- .../io/restest/cli/SettingsCommandTest.java | 36 ++++++--- .../java/io/restest/cli/WholeRunTest.java | 17 ++-- .../core/settings/ScheduleSettings.java | 47 ++++++++++- .../io/restest/core/settings/SettingKey.java | 10 +++ .../io/restest/core/settings/Settings.java | 17 +++- .../restest/core/settings/SettingsTest.java | 15 +++- .../java/io/restest/gen/RunOrderTest.java | 2 +- .../java/io/restest/gen/SchedulerTest.java | 4 +- .../java/io/restest/gen/SequencesTest.java | 4 +- 28 files changed, 449 insertions(+), 146 deletions(-) diff --git a/docs/adr/0025-settings.md b/docs/adr/0025-settings.md index 95296071d..c3e5c2a33 100644 --- a/docs/adr/0025-settings.md +++ b/docs/adr/0025-settings.md @@ -656,3 +656,27 @@ place asked before lists written for a whole kind of value, is gone: since [ADR-0033](0033-a-dictionary-file-is-written-by-operation-and-place.md) every list handed over is written for places of one API, and the lists are asked in the order they were handed over. Two are left, and the page of switches says so. + +## Amendment (13 October 2026): a run of a known size, and how long a stopped run may write + +**Date:** 2026-10-13 + +Four settings join the `schedule` group. `stopAfterRequests` (off) and `requestsBeforeStopping` +(1000) let a run be given a size rather than only a length: on, it stops asking for more once it has +sent that many requests, waits for their answers as it does at a deadline, and ends. A run of a +known size sends the same requests on a slow machine as on a fast one, which is what RESTest's own +tests of the command line need: two in three of them ran for a second or two of the clock, and the +slowest machines continuous integration runs on spent that second starting up and answered `3`, +"nothing was tested". A pair rather than one number with zero for "off", by the rule given with the +switches: a number set to zero says nothing about switching. + +`interruptWritingGrace` (5 s) and `interruptLookingGrace` (1 s) were two constants in the command: +how long a run stopped from outside may spend writing what it found once it has stopped waiting for +its answers, and how long it then gives its directory to say what was written. They are decisions, +not facts - the sum, with `interruptGrace`, has to fit inside the ten seconds a stopped container +is given - and §5 says such constants move when the code around them is next touched. It was, so +that a test of a run that cannot finish writing could set the first to a second and the second to +ten: the test took eight seconds and failed on a slow Windows machine whose directory did not +answer within the second it was given; it now takes one and does not depend on the machine. + +Eighty-seven settings in nine groups. diff --git a/docs/command-line.md b/docs/command-line.md index dd091c38f..fd5332887 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -125,12 +125,13 @@ $ echo $? An answer that has not come back by the end of that wait counts as never answered, and the run says how many there were. A second Ctrl-C does nothing: the run is already stopping, and is done in -a few seconds. If it has not finished writing `schedule.interruptGrace` and five seconds after it -was stopped — a report that got stuck, say — it says what it did not leave behind, and ends. -`report.json` is only ever there whole. A `run.sqlite` that was not closed has lost the last -interactions it had not yet saved, and keeps some of what it did save in `run.sqlite-wal` beside -it, so the three files go together. If the directory itself does not answer, the run says that -instead, a second later. +a few seconds. If it has not finished writing `schedule.interruptGrace` and +`schedule.interruptWritingGrace` — five seconds unless set otherwise — after it was stopped, a +report that got stuck, say, it says what it did not leave behind, and ends. `report.json` is only +ever there whole. A `run.sqlite` that was not closed has lost the last interactions it had not yet +saved, and keeps some of what it did save in `run.sqlite-wal` beside it, so the three files go +together. If the directory itself does not answer within `schedule.interruptLookingGrace`, a second +unless set otherwise, the run says that instead. A run stopped while it is still reading the document writes nothing, since it has found nothing yet, and says so: the directory it would have written to is left as it was, with whatever an diff --git a/docs/settings.md b/docs/settings.md index d6f2da7dc..471d06558 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -120,6 +120,14 @@ schedule: openingLap: true # default # how long each step of that opening lap waits for the answers to the one before openingLapPatience: "2s" # default + # whether a run stops once it has sent a set number of requests, before its budget is spent + stopAfterRequests: false # default + # how many requests that is + requestsBeforeStopping: 1000 # default + # how long a run stopped from outside is then given to write what it found + interruptWritingGrace: "5s" # default + # how long it then gives its directory to say what was written, when writing did not finish + interruptLookingGrace: "1s" # default generation: # below this depth, only what the description insists on is built @@ -366,6 +374,10 @@ spelling `--budget` takes. | `interruptGrace` | `2s` | how long a run stopped from outside waits for answers already asked for | | `openingLap` | `true` | whether a run starts by sending every operation once, the request likeliest to work | | `openingLapPatience` | `2s` | how long each step of that opening lap waits for the answers to the one before | +| `stopAfterRequests` | `false` | whether a run stops once it has sent a set number of requests, before its budget is spent | +| `requestsBeforeStopping` | `1000` | how many requests that is | +| `interruptWritingGrace` | `5s` | how long a run stopped from outside is then given to write what it found | +| `interruptLookingGrace` | `1s` | how long it then gives its directory to say what was written, when writing did not finish | ### `generation.*` diff --git a/docs/switches.md b/docs/switches.md index dbe81f837..3605cd9b3 100644 --- a/docs/switches.md +++ b/docs/switches.md @@ -97,11 +97,12 @@ No check decides what a run sends: switching one off changes which replies are c so the exit code. [The faults](faults.md#switching-a-check-off) says what else it changes, and what it does not. -### And one that is not a lever +### And two that are not levers | Switch | By default | Part of | Off, a run… | |---|---|---|---| | `engine.followRedirects` | `false` | Redirections | reports a redirection rather than following it. It decides what a 3XX answer means rather than turning off something a run could do without, which is why it is off by default and plays no part in an ablation | +| `schedule.stopAfterRequests` | `false` | A run of a known size | ends when its budget is spent, as every run did until now. On, it ends once it has sent `schedule.requestsBeforeStopping` requests and their answers are in, which is how a short run is made the same on a slow machine as on a fast one. It changes when a run ends, not what it sends, so it plays no part in an ablation either | ## What leaving each one on costs diff --git a/restest-cli/src/main/java/io/restest/cli/RunCommand.java b/restest-cli/src/main/java/io/restest/cli/RunCommand.java index a5c32a7f3..963d5ebc0 100644 --- a/restest-cli/src/main/java/io/restest/cli/RunCommand.java +++ b/restest-cli/src/main/java/io/restest/cli/RunCommand.java @@ -28,6 +28,7 @@ import io.restest.core.model.OperationId; import io.restest.core.model.SecurityScheme; import io.restest.core.model.SpecificationIssue; +import io.restest.core.settings.ScheduleSettings; import io.restest.core.settings.Settings; import io.restest.core.settings.SettingsException; import io.restest.core.settings.SettingsInEffect; @@ -163,27 +164,6 @@ final class RunCommand implements Callable { "run.sqlite-shm", "report.json", JsonReport.whileBeingWritten(Path.of("report.json")).toString()); - /** - * How long a run stopped from outside is given to write what it found, once it has stopped - * waiting for its answers, before the program is let end without it. - * - *

Writing takes a fraction of a second: the reports catching up with the last answers, the - * report file, the last interactions into the stored run. This is not a pause anybody waits - * through. It is how long to wait for writing that has got stuck - a report that stopped making - * progress, a disk that stopped answering - before saying so, because the program will be ended - * outright soon after whoever stopped it runs out of patience: ten seconds after - * {@code docker stop}, with nothing said. With the two seconds a stopped run waits for its - * answers by default, and the moment {@link #LOOKING_ALLOWED} takes, this leaves two in hand. - */ - private static final Duration WRITING_ALLOWED = Duration.ofSeconds(5); - - /** - * How long a run stopped from outside that ran out of time to write gives its directory to say - * what is in it, before saying that it did not answer. A directory answers at once, unless it - * is the reason the writing ran out of time. - */ - private static final Duration LOOKING_ALLOWED = Duration.ofSeconds(1); - @Parameters( index = "0", // Not required, so that --print-campaign can answer a question about the tool without @@ -410,9 +390,9 @@ public Integer call() { // is still reading the document does not begin; one told while it tests is given the time // to leave behind what it found - and no more. StopFromOutside stop = new StopFromOutside(); - Duration grace = settings.schedule().interruptGrace(); + ScheduleSettings schedule = settings.schedule(); try (StopsFromOutside.Arrangement heard = stops.whenStopped( - () -> stoppedFromOutside(stop, grace, err))) { + () -> stoppedFromOutside(stop, schedule, err))) { try { int answer = readAndTest(configuration, stop, out, err); // Stopped while it was still reading, a run can go on to find something else to @@ -523,9 +503,17 @@ private int readAndTest(SettingsInEffect configuration, StopFromOutside stop, Pr * the program ends. When it is not done in time, this says what it has not left behind, and * the program ends all the same: whoever stopped it is about to end it outright. A run stopped * before it began testing is not waited for, and this says it wrote nothing. + * + *

How long the run is waited for is the schedule's: the time it may spend waiting for its + * answers, then the time it may spend writing. Writing takes a fraction of a second, so the + * second is how long to wait for writing that has got stuck - a report that stopped making + * progress, a disk that stopped answering - before saying so. The two, with the moment the + * directory is then given to say what is in it, have to fit inside the patience of whoever + * stopped the run: ten seconds, when a container is stopped. */ - private void stoppedFromOutside(StopFromOutside stop, Duration grace, PrintWriter err) { - Duration atMost = grace.plus(WRITING_ALLOWED); + private void stoppedFromOutside(StopFromOutside stop, ScheduleSettings schedule, + PrintWriter err) { + Duration atMost = schedule.interruptGrace().plus(schedule.interruptWritingGrace()); switch (stop.askAndWait(atMost)) { case DONE -> { return; @@ -548,7 +536,8 @@ private void stoppedFromOutside(StopFromOutside stop, Duration grace, PrintWrite Thread.ofVirtual().name("restest-looking").start(look); String missing; try { - List found = look.get(StopFromOutside.nanosIn(LOOKING_ALLOWED), + List found = look.get( + StopFromOutside.nanosIn(schedule.interruptLookingGrace()), TimeUnit.NANOSECONDS); missing = found.isEmpty() ? "" : ": " + String.join("; ", found); } catch (TimeoutException notAnswering) { @@ -676,6 +665,9 @@ private int testing(ApiModel model, RandomTestCaseGenerator generator, settings.schedule().workAheadFactor() * settings.engine() .maxConcurrency(), settings.schedule().announcementsAllowedToPileUp(), + settings.schedule().stopAfterRequests() + ? settings.schedule().requestsBeforeStopping() + : Long.MAX_VALUE, settings.engine().readTimeout() .plus(settings.schedule().stragglerGrace()), settings.schedule().interruptGrace(), stop, sending, drained); diff --git a/restest-cli/src/main/java/io/restest/cli/RunLoop.java b/restest-cli/src/main/java/io/restest/cli/RunLoop.java index b78f739c1..c1f3b7e70 100644 --- a/restest-cli/src/main/java/io/restest/cli/RunLoop.java +++ b/restest-cli/src/main/java/io/restest/cli/RunLoop.java @@ -149,6 +149,9 @@ boolean nothingCouldBeBuilt() { * faster than all three together; without a limit the backlog grows for as long as the run * lasts, and a long run against a fast API ends by running out of memory rather than out of * time. Pausing is the honest response, and is counted as time the tool wasted + * @param mostRequests how many requests the loop may send before it stops asking for more, as a + * run of a known size asks; the answers to them are waited for as at the deadline. The + * largest number there is when only the deadline ends the run * @param howLongToWaitForStragglers how long to go on waiting, past the deadline, for answers to * requests that had already gone out * @param interruptGrace how long to wait for them instead, from the moment the run is stopped @@ -159,10 +162,15 @@ boolean nothingCouldBeBuilt() { * @return what the loop did with the time */ static Outcome run(Scheduler scheduler, String baseUrl, int workAhead, - int announcementsAllowedToPileUp, Duration howLongToWaitForStragglers, - Duration interruptGrace, StopFromOutside stop, HttpEngine engine, EventStream events) { + int announcementsAllowedToPileUp, long mostRequests, + Duration howLongToWaitForStragglers, Duration interruptGrace, StopFromOutside stop, + HttpEngine engine, EventStream events) { Objects.requireNonNull(scheduler, "scheduler"); Objects.requireNonNull(baseUrl, "baseUrl"); + if (mostRequests < 1) { + throw new IllegalArgumentException("a run that may send no request at all tests " + + "nothing: " + mostRequests); + } Objects.requireNonNull(howLongToWaitForStragglers, "howLongToWaitForStragglers"); Objects.requireNonNull(interruptGrace, "interruptGrace"); Objects.requireNonNull(stop, "stop"); @@ -225,6 +233,11 @@ static Outcome run(Scheduler scheduler, String baseUrl, int workAhead, } } case Scheduler.Step.Send send -> { + // A run of a known size has sent what it was asked to: it stops asking, + // and goes on below to wait for the answers it is owed, as at a deadline. + if (sent >= mostRequests) { + break deciding; + } if (!waitForASlot(slots, deadline, stop)) { break deciding; } diff --git a/restest-cli/src/test/java/io/restest/cli/ABrokenRunAnswersFourTest.java b/restest-cli/src/test/java/io/restest/cli/ABrokenRunAnswersFourTest.java index d107c992e..29ad02d6e 100644 --- a/restest-cli/src/test/java/io/restest/cli/ABrokenRunAnswersFourTest.java +++ b/restest-cli/src/test/java/io/restest/cli/ABrokenRunAnswersFourTest.java @@ -72,11 +72,14 @@ class ABrokenRunAnswersFourTest { /** - * Long enough for a slow machine to read the document, start and still send: a run that sends - * nothing answers 3, which is not what these runs are about. One second was not, on one of the - * machines the build runs on. + * The runs here are given a size, sixty requests, rather than a length: a run that sends + * nothing answers 3, which is not what these runs are about, and on the slowest machines the + * build runs on a second or two of the clock was spent starting up. The budget is only the + * ceiling a run that went wrong could not get past. */ - private static final String BUDGET = "3s"; + private static final String BUDGET = "30s"; + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; private static WireMockServer api; @@ -106,7 +109,8 @@ static void stopTheApi() { void losing_some_requests_answers_four(@TempDir Path directory) { int answer = run(settings -> new Losing(new OkHttpEngine(settings), testCase -> testCase.operation().value().equals("listPets")), - "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", BUDGET, "--seed", + "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", BUDGET, + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "7", "--out", directory.toString()); assertThat(answer).describedAs("%s%n%s", screen, problems).isEqualTo(4); @@ -155,7 +159,8 @@ void an_exchange_kept_without_its_details_answers_four(@TempDir Path directory) // picking the key out of it runs out of stack, so the exchange is kept with everything that // could hold a key blanked. The run is handed a key, or there is nothing to hide at all. int answer = run(settings -> new NestingTooDeep(new OkHttpEngine(settings)), - "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", BUDGET, "--seed", + "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", BUDGET, + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "7", "--out", directory.toString(), "--auth", "header:X-Shelter-Key=Zk9-leakprobe-4f"); @@ -181,11 +186,13 @@ void a_request_handed_back_as_nothing_is_lost(@TempDir Path directory) { testCase -> testCase.operation().value().equals("listPets"), true); int plain = run(nothingForListPets, "run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", BUDGET, "--seed", "7", "--out", directory.resolve("plain").toString()); + "--budget", BUDGET, "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, + "--seed", "7", "--out", directory.resolve("plain").toString()); String saidWithoutAKey = problems.toString(); problems.getBuffer().setLength(0); int keyed = run(nothingForListPets, "run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", BUDGET, "--seed", "7", "--out", directory.resolve("keyed").toString(), + "--budget", BUDGET, "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, + "--seed", "7", "--out", directory.resolve("keyed").toString(), "--auth", "header:X-Shelter-Key=Zk9-leakprobe-4f"); String saidWithAKey = problems.toString(); @@ -221,8 +228,8 @@ void the_failure_shown_has_the_key_hidden(@TempDir Path directory) { int answer = run(settings -> new Losing(new OkHttpEngine(settings), testCase -> testCase.operation().value().equals("listPets"), false, "sent with " + key), "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", - BUDGET, "--seed", "7", "--out", directory.toString(), "--auth", - "header:X-Shelter-Key=" + key); + BUDGET, "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "7", + "--out", directory.toString(), "--auth", "header:X-Shelter-Key=" + key); assertThat(answer).describedAs("%s%n%s", screen, problems).isEqualTo(4); assertThat(problems.toString()) diff --git a/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java b/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java index 9d6b16a3f..cb258e15c 100644 --- a/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java +++ b/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java @@ -69,6 +69,14 @@ */ class AKeyIsWrittenNowhereTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String THREE_HUNDRED_REQUESTS = "schedule.requestsBeforeStopping=300"; + private static final String IN_A_HEADER = "Zk9-leakprobe-4f+q/Rw=="; private static final String IN_THE_QUERY = "Qm7-leakprobe-8x+z/Pt=="; private static final String IN_A_COOKIE = "Hs2-leakprobe-5c-w_Ls="; @@ -187,7 +195,8 @@ void no_key_is_written_anywhere(@TempDir Path directory) throws IOException { try (PrintWriter printed = new PrintWriter(screen); PrintWriter complained = new PrintWriter(problems)) { answer = Restest.run(new String[] {"run", document.toString(), "--url", - api.baseUrl(), "--budget", "3s", "--seed", "20260930", "--out", out.toString(), + api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", THREE_HUNDRED_REQUESTS, "--seed", "20260930", "--out", out.toString(), "--store", "--auth", "inAHeader=" + IN_A_HEADER, "--auth", "inTheQuery=" + IN_THE_QUERY}, printed, complained, Map.of("RESTEST_AUTH", "inACookie=" + IN_A_COOKIE)); diff --git a/restest-cli/src/test/java/io/restest/cli/ARunCutShortTest.java b/restest-cli/src/test/java/io/restest/cli/ARunCutShortTest.java index a4390bdb3..f7a8c9eef 100644 --- a/restest-cli/src/test/java/io/restest/cli/ARunCutShortTest.java +++ b/restest-cli/src/test/java/io/restest/cli/ARunCutShortTest.java @@ -164,20 +164,25 @@ void a_run_that_cannot_finish_writing_says_so(@TempDir Path directory) { StopWhen stop = new StopWhen(() -> asked() >= 20); // Stuck where the run says it is over, so nothing after that - the report, the stored run - // being closed - is done while the program waits for it. + // being closed - is done while the program waits for it. Writing is given a second rather + // than the usual five, so that the test takes one; the directory is given ten rather than + // one to say what is in it, so that a slow build machine, whose directory has taken longer + // than a second to answer, cannot turn this into a test of the machine. int answer = run(settings -> new StuckAtTheEnd(new OkHttpEngine(settings), - Duration.ofSeconds(7)), stop, "run", "pet-shelter.yaml", "--url", api.baseUrl(), + Duration.ofSeconds(3)), stop, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", BUDGET, "--seed", "7", "--store", "--out", directory.toString(), - "--set", "schedule.interruptGrace=0s"); + "--set", "schedule.interruptGrace=0s", + "--set", "schedule.interruptWritingGrace=1s", + "--set", "schedule.interruptLookingGrace=10s"); stop.awaitDone(); assertThat(Duration.between(stop.setOffAt, stop.doneAt)) - .describedAs("the grace, none here, and the five seconds writing is given") - .isGreaterThanOrEqualTo(Duration.ofMillis(4900)) - .isLessThan(Duration.ofSeconds(7)); + .describedAs("the grace, none here, and the second writing is given") + .isGreaterThanOrEqualTo(Duration.ofMillis(900)) + .isLessThan(Duration.ofSeconds(3)); assertThat(problems.toString()) .contains("restest: the run was stopped from outside and had not finished writing " - + "what it found 5s later, so it ends here: report.json was not written; " + + "what it found 1s later, so it ends here: report.json was not written; " + "run.sqlite was not closed, so the last interactions it had not yet saved " + "are lost, and some of what it did save is in run.sqlite-wal beside it: " + "keep the three files together"); diff --git a/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java b/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java index befbbccda..a4712598f 100644 --- a/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java @@ -175,7 +175,7 @@ void in_a_header(@TempDir Path directory) throws IOException { .willReturn(aResponse().withStatus(200))); int answer = run(Map.of(), "run", document(directory, PETSTORE), "--url", - api.baseUrl(), "--budget", "2s", "--seed", "20260930", "--out", + api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString(), "--auth", KEY); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); @@ -209,7 +209,7 @@ void in_the_query_and_in_a_cookie(@TempDir Path directory) throws IOException { .atPriority(1).willReturn(aResponse().withStatus(200))); int answer = run(Map.of(), "run", document(directory, TWO_KEYS), "--url", - api.baseUrl(), "--budget", "2s", "--seed", "20260930", "--out", + api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString(), "--auth", "token=" + KEY, "--auth", "session=" + cookie); @@ -233,7 +233,7 @@ void with_its_place(@TempDir Path directory) throws IOException { .atPriority(1).willReturn(aResponse().withStatus(200))); int answer = run(Map.of(), "run", document(directory, WORDS), "--url", api.baseUrl(), - "--budget", "2s", "--seed", "20260930", "--out", + "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString(), "--auth", "query:apiKey=" + KEY); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); @@ -280,7 +280,7 @@ void a_bearer_token_already_held(@TempDir Path directory) throws IOException { Path out = directory.resolve("out"); int answer = run(Map.of(), "run", document(directory, signedIn), "--url", - api.baseUrl(), "--budget", "2s", "--seed", "20260930", "--out", out.toString(), + api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", out.toString(), "--store", "--auth", "header:Authorization=Bearer " + token); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); @@ -312,7 +312,7 @@ void it_is_sent_and_a_typed_one_wins(@TempDir Path directory) throws IOException .willReturn(aResponse().withStatus(200))); int alone = run(Map.of("RESTEST_AUTH", KEY), "run", document(directory, PETSTORE), - "--url", api.baseUrl(), "--budget", "1s", "--seed", "20260930", "--out", + "--url", api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("alone").toString()); assertThat(alone).describedAs("%s%n%s", screen, problems).isBetween(0, 1); assertThat(api.findAll(getRequestedFor(urlPathEqualTo("/pets")))).isNotEmpty() @@ -321,7 +321,7 @@ void it_is_sent_and_a_typed_one_wins(@TempDir Path directory) throws IOException api.resetRequests(); int overruled = run(Map.of("RESTEST_AUTH", "api_key=the-one-left-behind"), "run", - document(directory, PETSTORE), "--url", api.baseUrl(), "--budget", "1s", + document(directory, PETSTORE), "--url", api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("overruled").toString(), "--auth", KEY); assertThat(overruled).isBetween(0, 1); @@ -336,7 +336,7 @@ void one_that_cannot_be_used_only_warns(@TempDir Path directory) throws IOExcept api.stubFor(any(anyUrl()).atPriority(1).willReturn(aResponse().withStatus(200))); int answer = run(Map.of("RESTEST_AUTH", KEY), "run", document(directory, NO_SCHEME), - "--url", api.baseUrl(), "--budget", "1s", "--seed", "20260930", "--out", + "--url", api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString()); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); diff --git a/restest-cli/src/test/java/io/restest/cli/BodiesOfTheWrongShapeTest.java b/restest-cli/src/test/java/io/restest/cli/BodiesOfTheWrongShapeTest.java index 79568cbcf..153f180ee 100644 --- a/restest-cli/src/test/java/io/restest/cli/BodiesOfTheWrongShapeTest.java +++ b/restest-cli/src/test/java/io/restest/cli/BodiesOfTheWrongShapeTest.java @@ -57,6 +57,14 @@ */ class BodiesOfTheWrongShapeTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String FIFTEEN_HUNDRED_REQUESTS = "schedule.requestsBeforeStopping=1500"; + private static final String SPECIFICATION = """ openapi: 3.0.3 info: {title: Careless notes, version: "1.0"} @@ -135,7 +143,7 @@ private static String run(Path out, Path document, String... extra) { List arguments = new ArrayList<>(List.of( "run", document.toString(), "--url", api.baseUrl(), - "--budget", "2s", + "--budget", "30s", "--set", STOP_AFTER_REQUESTS, "--set", FIFTEEN_HUNDRED_REQUESTS, "--seed", "20260928", "--out", out.toString())); for (String other : THE_OTHER_CHANGES) { diff --git a/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java b/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java index 8b7612213..ab1cd2264 100644 --- a/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java @@ -44,6 +44,14 @@ */ class CampaignCommandTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; + private static WireMockServer api; private final StringWriter screen = new StringWriter(); @@ -105,7 +113,8 @@ void what_is_printed_can_be_handed_back(@TempDir Path directory) throws Exceptio Path saved = directory.resolve("mine.yaml"); Files.writeString(saved, screen.toString()); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", saved.toString(), "--out", directory.resolve("out").toString())) .isBetween(0, 1); assertThat(problems.toString()).doesNotContain("could not be read"); @@ -120,7 +129,8 @@ void a_plan_narrows_which_operations_are_touched(@TempDir Path directory) throws methods: [GET, HEAD, OPTIONS, TRACE] """); - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", plan.toString(), "--out", directory.resolve("out").toString()); assertThat(screen.toString()) @@ -138,7 +148,8 @@ void a_plan_that_matches_nothing_says_so(@TempDir Path directory) throws Excepti only: [thereIsNoSuchOperation] """); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", plan.toString(), "--out", directory.resolve("out").toString())) .isEqualTo(3); assertThat(problems.toString()) @@ -168,7 +179,8 @@ void what_the_api_returned_goes_back_out(@TempDir Path directory) throws Excepti - source: random """); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "2s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", plan.toString(), "--out", directory.resolve("out").toString())) .isBetween(0, 1); @@ -200,7 +212,8 @@ void a_plan_that_cannot_be_read_ends_the_run(@TempDir Path directory) throws Exc methods: [GET] """); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", plan.toString(), "--out", directory.resolve("out").toString())) .isEqualTo(2); assertThat(problems.toString()).contains("does not recognise (sourses)"); @@ -212,7 +225,8 @@ void a_plan_that_cannot_be_read_ends_the_run(@TempDir Path directory) throws Exc @Test @DisplayName("nor does a plan named that is not there, or one with a mistake inside a group") void the_other_two_ways_a_plan_is_refused(@TempDir Path directory) throws Exception { - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", directory.resolve("nope.yaml").toString(), "--out", directory.resolve("out").toString())) .isEqualTo(2); @@ -228,7 +242,8 @@ void the_other_two_ways_a_plan_is_refused(@TempDir Path directory) throws Except sources: - weighted: {random: 100, example: 0} """); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", zero.toString(), "--out", directory.resolve("out2").toString())) .describedAs("a person's typo answers 2, not the 4 that means RESTest broke") .isEqualTo(2); @@ -239,7 +254,8 @@ void the_other_two_ways_a_plan_is_refused(@TempDir Path directory) throws Except @DisplayName("how much of a run pushes at the API is said in a plan: --fuzzing is not an " + "option, and answers the way any option that does not exist answers") void a_share_of_pushing_is_said_in_a_plan(@TempDir Path directory) throws Exception { - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--fuzzing", "10", "--out", directory.resolve("out").toString())) .isEqualTo(2); assertThat(problems.toString()).contains("Unknown option: --fuzzing"); @@ -259,7 +275,8 @@ void a_plan_that_matches_nothing_says_so_beside_what_could_not_be_read(@TempDir """); assertThat(run("run", "pet-shop-with-an-operation-it-cannot-read.yaml", - "--url", api.baseUrl(), "--budget", "1s", "--campaign", plan.toString(), + "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", plan.toString(), "--out", directory.resolve("out").toString())) .isEqualTo(3); assertThat(problems.toString()) diff --git a/restest-cli/src/test/java/io/restest/cli/CheckSwitchesCommandTest.java b/restest-cli/src/test/java/io/restest/cli/CheckSwitchesCommandTest.java index c9e61c379..1edcccf3f 100644 --- a/restest-cli/src/test/java/io/restest/cli/CheckSwitchesCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/CheckSwitchesCommandTest.java @@ -48,6 +48,14 @@ */ class CheckSwitchesCommandTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; + private static final String DOCUMENT = """ openapi: 3.0.3 info: @@ -143,7 +151,8 @@ private static int run(Path directory, StringWriter screen, String... settings) throws IOException { Path document = Files.writeString(directory.resolve("openapi.yaml"), DOCUMENT); List arguments = new ArrayList<>(List.of("run", document.toString(), - "--url", api.baseUrl(), "--budget", "1s", "--seed", "1", + "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.resolve("out").toString())); arguments.addAll(List.of(settings)); PrintWriter out = new PrintWriter(screen); diff --git a/restest-cli/src/test/java/io/restest/cli/FuzzingFindsWhatNominalMissesTest.java b/restest-cli/src/test/java/io/restest/cli/FuzzingFindsWhatNominalMissesTest.java index dd8f57408..f62b8274f 100644 --- a/restest-cli/src/test/java/io/restest/cli/FuzzingFindsWhatNominalMissesTest.java +++ b/restest-cli/src/test/java/io/restest/cli/FuzzingFindsWhatNominalMissesTest.java @@ -132,11 +132,13 @@ void the_awkward_values_are_what_find_it(@TempDir Path directory) throws IOExcep // The changes made to accepted requests switched off in both runs, so that the list of // awkward values is the only thing that differs between them. - // Longer than the other runs here: the empty search term is one of the many awkward values - // a run holds, drawn by chance, and two seconds on a slow build machine may not reach it. - String withAwkwardValues = runFor("5s", directory.resolve("with"), document, + // Larger than the other runs here: the empty search term is one of the many awkward values + // a run holds, drawn by chance, and a run of a few hundred requests may not reach it. With + // the seed the run draws for itself, what is reached differs from run to run; the size is + // what makes the draw reach it every time. + String withAwkwardValues = runOf(1_200, directory.resolve("with"), document, "--set", "mutation.violations=false"); - String withoutThem = runFor("5s", directory.resolve("without"), document, + String withoutThem = runOf(1_200, directory.resolve("without"), document, "--campaign", plan(directory, NEVER_PUSHES).toString(), "--set", "mutation.violations=false"); @@ -321,14 +323,20 @@ private static Path plan(Path directory, String text) throws IOException { } private static String run(Path out, Path document, String... extra) { - return runFor("2s", out, document, extra); + return runOf(300, out, document, extra); } - private static String runFor(String budget, Path out, Path document, String... extra) { + /** + * A run of this many requests. A size rather than a length of time, so that what the run sends + * does not depend on how fast the machine is; the budget is only a ceiling. + */ + private static String runOf(int requests, Path out, Path document, String... extra) { java.util.List arguments = new java.util.ArrayList<>(java.util.List.of( "run", document.toString(), "--url", api.baseUrl(), - "--budget", budget, + "--budget", "30s", + "--set", "schedule.stopAfterRequests=true", + "--set", "schedule.requestsBeforeStopping=" + requests, "--out", out.toString())); arguments.addAll(java.util.List.of(extra)); StringWriter screen = new StringWriter(); diff --git a/restest-cli/src/test/java/io/restest/cli/MutationFindsWhatNothingElseReachesTest.java b/restest-cli/src/test/java/io/restest/cli/MutationFindsWhatNothingElseReachesTest.java index 886bcbaa3..610d8dbc5 100644 --- a/restest-cli/src/test/java/io/restest/cli/MutationFindsWhatNothingElseReachesTest.java +++ b/restest-cli/src/test/java/io/restest/cli/MutationFindsWhatNothingElseReachesTest.java @@ -52,6 +52,14 @@ */ class MutationFindsWhatNothingElseReachesTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String FIFTEEN_HUNDRED_REQUESTS = "schedule.requestsBeforeStopping=1500"; + private static final String SPECIFICATION = """ openapi: 3.0.3 info: {title: Careless orders, version: "1.0"} @@ -122,7 +130,7 @@ private static String run(Path out, Path document, String... extra) { List arguments = new ArrayList<>(List.of( "run", document.toString(), "--url", api.baseUrl(), - "--budget", "2s", + "--budget", "30s", "--set", STOP_AFTER_REQUESTS, "--set", FIFTEEN_HUNDRED_REQUESTS, "--seed", "20260927", "--out", out.toString())); arguments.addAll(List.of(extra)); diff --git a/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java b/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java index 3197462a7..8407d2145 100644 --- a/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java @@ -53,6 +53,14 @@ */ class OpeningLapCommandTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; + private static WireMockServer api; @BeforeAll @@ -79,9 +87,10 @@ static void stopTheApi() { void a_run_begins_with_every_operation_once(@TempDir Path directory) throws IOException { StringWriter screen = new StringWriter(); - // Five seconds, not two: the budget starts with the command, before the document is read, - // and a cold machine must still see the whole round through before the time is up. - run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "5s", + // Sixty requests see the whole round through, however cold the machine: the round's own + // requests are the first of them. + run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", "--out", directory.toString(), "--store"); // In the order they went out. Within a step the requests are sent together, so the two @@ -106,7 +115,8 @@ void a_run_begins_with_every_operation_once(@TempDir Path directory) throws IOEx void switched_off_there_is_no_first_round(@TempDir Path directory) throws IOException { StringWriter screen = new StringWriter(); - run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "2s", + run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", "--out", directory.toString(), "--store", "--set", "schedule.openingLap=false"); @@ -135,7 +145,8 @@ void nothing_declared_means_anything_accepted(@TempDir Path directory) throws IO """); run(new StringWriter(), "run", document.toString(), "--url", api.baseUrl(), - "--budget", "2s", "--seed", "1", "--out", directory.resolve("out").toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.resolve("out").toString()); List pinged = api.findAll(getRequestedFor(urlEqualTo("/ping"))); assertThat(pinged).isNotEmpty(); diff --git a/restest-cli/src/test/java/io/restest/cli/RestestTest.java b/restest-cli/src/test/java/io/restest/cli/RestestTest.java index a1d872417..60a7372f0 100644 --- a/restest-cli/src/test/java/io/restest/cli/RestestTest.java +++ b/restest-cli/src/test/java/io/restest/cli/RestestTest.java @@ -59,6 +59,14 @@ */ class RestestTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; + private static WireMockServer api; private final StringWriter screen = new StringWriter(); @@ -91,12 +99,13 @@ static void stopTheApi() { @DisplayName("a run that finds something wrong says so, and answers 1") void faults_found_answer_one(@TempDir Path directory) throws Exception { int answer = run("run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--seed", "20260914", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", "--out", directory.toString()); assertThat(answer).isEqualTo(1); assertThat(screen.toString()) .contains("RESTest testing Pet Shelter at " + api.baseUrl()) - .contains("4 of 4 operations can be tested, seed 20260914, budget 1s") + .contains("4 of 4 operations can be tested, seed 20260914, budget 30s") .contains("F100") .contains("listPets - GET " + api.baseUrl() + "/pets") .containsPattern("\\d+ requests to \\d+ operations in .+, \\d+% of it idle"); @@ -142,7 +151,8 @@ void the_directory_the_document_declares_is_where_the_requests_go(@TempDir Path .withBody("[{\"id\": 7, \"name\": \"Rex\"}]"))); int answer = run("run", "pet-shelter-under-a-directory.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--seed", "20260918", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260918", "--out", directory.toString()); assertThat(answer) .describedAs("nothing here is broken - though this says only that the run finished, " @@ -158,7 +168,8 @@ void the_directory_the_document_declares_is_where_the_requests_go(@TempDir Path @Test @DisplayName("a run keeps nothing unless asked to, and says so rather than leaving you to look") void a_run_keeps_nothing_unless_asked(@TempDir Path directory) { - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", "--out", directory.toString()); assertThat(directory.resolve("run.sqlite")) @@ -175,7 +186,8 @@ void a_run_keeps_nothing_unless_asked(@TempDir Path directory) { @Test @DisplayName("a run asked to keep itself does, and says where and how big") void a_run_asked_to_keep_itself_does(@TempDir Path directory) { - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", "--out", directory.toString(), "--store"); assertThat(directory.resolve("run.sqlite")).exists(); @@ -189,14 +201,16 @@ void a_run_asked_to_keep_itself_does(@TempDir Path directory) { @Test @DisplayName("a second run replaces a kept run rather than leaving two runs in one directory") void a_second_run_replaces_a_kept_run(@TempDir Path directory) { - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.toString(), "--store"); assertThat(directory.resolve("run.sqlite")).exists(); StringWriter second = new StringWriter(); PrintWriter out = new PrintWriter(second); Restest.run(new String[] {"run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--seed", "2", "--out", directory.toString()}, + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "2", "--out", directory.toString()}, out, new PrintWriter(problems)); out.flush(); @@ -213,7 +227,8 @@ void a_run_leaves_other_files_alone(@TempDir Path directory) throws Exception { Path somebodysWork = directory.resolve("notes.txt"); Files.writeString(somebodysWork, "not ours"); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", "--out", directory.toString())).isEqualTo(1); assertThat(somebodysWork) @@ -227,7 +242,8 @@ void a_run_leaves_other_files_alone(@TempDir Path directory) throws Exception { @DisplayName("a run that finds nothing wrong says so, and answers 0") void a_clean_run_answers_zero(@TempDir Path directory) throws Exception { int answer = run("run", "pet-shelter.yaml", "--url", api.baseUrl() + "/shelters-only", - "--budget", "1s", "--seed", "1", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.toString()); // Nothing is stubbed under that prefix, so every reply is a 404 - which no oracle in this // version complains about, because a document that does not describe 404 has not been @@ -291,7 +307,8 @@ void an_unreadable_document_answers_three(@TempDir Path directory) throws Except Path nothingThere = directory.resolve("not-a-document.yaml"); Files.writeString(nothingThere, "this file is not an API description\n"); - assertThat(run("run", nothingThere.toString(), "--url", api.baseUrl(), "--budget", "1s")) + assertThat(run("run", nothingThere.toString(), "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS)) .isEqualTo(3); assertThat(problems.toString()).contains("no operation that could be tested"); } @@ -299,7 +316,8 @@ void an_unreadable_document_answers_three(@TempDir Path directory) throws Except @Test @DisplayName("a document with nowhere to send requests asks for --url, and answers 3") void nowhere_to_send_answers_three() { - assertThat(run("run", "pet-shelter.yaml", "--budget", "1s")).isEqualTo(3); + assertThat(run("run", "pet-shelter.yaml", "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS)).isEqualTo(3); assertThat(problems.toString()) .contains("there is nowhere to send the requests") .contains("--url"); @@ -324,7 +342,8 @@ void nothing_listening_answers_three(@TempDir Path directory) throws Exception { void a_get_insisting_on_a_body_is_named_rather_than_blamed_on_the_address( @TempDir Path directory) { int answer = run("run", "pet-search-with-a-body.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()); assertThat(answer).isEqualTo(3); assertThat(problems.toString()) @@ -349,7 +368,8 @@ void a_get_merely_accepting_a_body_is_sent_without_it(@TempDir Path directory) { .withBody("[]"))); int answer = run("run", "pet-search-with-an-optional-body.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--seed", "20260923", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", "--out", directory.toString()); assertThat(answer).isZero(); assertThat(screen.toString()) @@ -377,14 +397,15 @@ void the_operations_a_run_cannot_try_are_named_on_the_screen_and_in_the_report( + "is not assembled yet"); int answer = run("run", "pet-shop-with-operations-it-cannot-try.yaml", - "--url", api.baseUrl(), "--budget", "1s", "--seed", "20260923", + "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", "--out", directory.toString()); assertThat(answer).describedAs("an operation that had to be skipped is not an error").isZero(); List lines = screen.toString().lines().toList(); assertThat(lines.subList(0, 3)).containsExactly( "RESTest testing Pet Shop at " + api.baseUrl(), "", - "1 of 4 operations can be tested, seed 20260923, budget 1s"); + "1 of 4 operations can be tested, seed 20260923, budget 30s"); int verdict = lines.indexOf("no faults found"); assertThat(verdict).describedAs("the verdict is there").isNotNegative(); assertThat(lines.subList(verdict + 1, lines.size())) @@ -412,13 +433,14 @@ void an_operation_that_could_not_be_read_is_counted_and_named(@TempDir Path dire + "greater than maxProperties (2), so no value can satisfy both (paths./pets.get)"; int answer = run("run", "pet-shop-with-an-operation-it-cannot-read.yaml", - "--url", api.baseUrl(), "--budget", "1s", "--seed", "20260923", + "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", "--out", directory.toString()); assertThat(answer).isZero(); assertThat(screen.toString()) .describedAs("the document describes two operations, whether or not both could be read") - .contains("1 of 2 operations can be tested, seed 20260923, budget 1s"); + .contains("1 of 2 operations can be tested, seed 20260923, budget 30s"); List lines = screen.toString().lines().toList(); int verdict = lines.indexOf("no faults found"); assertThat(verdict).describedAs("the verdict is there").isNotNegative(); @@ -450,7 +472,8 @@ void a_document_nothing_of_which_could_be_read_says_why(@TempDir Path directory) responses: {'200': {description: the pets that match}} """); - int answer = run("run", document.toString(), "--url", api.baseUrl(), "--budget", "1s", + int answer = run("run", document.toString(), "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.resolve("out").toString()); assertThat(answer).isEqualTo(3); @@ -486,7 +509,8 @@ void the_reason_quoted_is_one_given_for_an_operation_that_was_read(@TempDir Path responses: {'200': {description: every pet}} """); - int answer = run("run", document.toString(), "--url", api.baseUrl(), "--budget", "1s", + int answer = run("run", document.toString(), "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.resolve("out").toString()); assertThat(answer).isEqualTo(3); @@ -525,7 +549,8 @@ void an_unreadable_operation_never_shadows_a_readable_one_of_the_same_name( // which the awkward values include, makes the stub itself answer 500. api.stubFor(any(anyUrl()).atPriority(10).willReturn(aResponse().withStatus(404))); - int answer = run("run", document.toString(), "--url", api.baseUrl(), "--budget", "1s", + int answer = run("run", document.toString(), "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260924", "--out", directory.resolve("out").toString()); assertThat(answer).isZero(); @@ -560,7 +585,8 @@ void an_unwritable_output_directory_answers_three(@TempDir Path parent) throws E try { int answer = run("run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()); assertThat(answer) .describedAs("nowhere to write the results is one of the ways a run cannot " @@ -599,7 +625,8 @@ void an_error_in_a_run_answers_four(@TempDir Path directory) { // program with the number Java gives any program that crashes, which is 1. int answer = run(anEnvironmentThatBreaks(() -> { throw new StackOverflowError("a shape too deep to walk"); - }), "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + }), "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()); assertThat(answer) @@ -618,7 +645,8 @@ void an_error_in_a_run_answers_four(@TempDir Path directory) { void an_exception_in_a_run_answers_four(@TempDir Path directory) { int answer = run(anEnvironmentThatBreaks(() -> { throw new IllegalStateException("something RESTest did not expect"); - }), "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + }), "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()); assertThat(answer).isEqualTo(4); @@ -685,7 +713,8 @@ void a_run_that_breaks_while_sending_answers_four_and_leaves_nothing_open( assertThat(threadsNamed("restest-events")) .describedAs("the thread that delivered the run's events ended with it") .isSubsetOf(deliveringBefore); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", "--out", directory.resolve("again").toString())) .describedAs("and the same program runs another, which finds what it always finds") .isEqualTo(1); @@ -697,7 +726,8 @@ void a_run_that_cannot_say_it_broke_still_answers_four(@TempDir Path directory) // A program that has run out of memory may not have enough left to write a stack trace. // Somewhere to write problems that breaks when it is written to stands in for that. int answer = Restest.run(new String[] {"run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--out", directory.toString()}, + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()}, new PrintWriter(screen), aScreenThatBreaks(), anEnvironmentThatBreaks(() -> { throw new StackOverflowError("a shape too deep to walk"); diff --git a/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java b/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java index 3fd4fd5da..9caba365a 100644 --- a/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java +++ b/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java @@ -105,7 +105,11 @@ class RunLoopTest { private static final ScheduleSettings WITHOUT_A_FIRST_ROUND = new ScheduleSettings( WITH_A_FIRST_ROUND.workAheadFactor(), ANNOUNCEMENTS_ALLOWED, WITH_A_FIRST_ROUND.stragglerGrace(), WITH_A_FIRST_ROUND.interruptGrace(), false, - WITH_A_FIRST_ROUND.openingLapPatience()); + WITH_A_FIRST_ROUND.openingLapPatience(), false, 1_000, + WITH_A_FIRST_ROUND.interruptWritingGrace(), WITH_A_FIRST_ROUND.interruptLookingGrace()); + + /** No limit on how many requests a run here may send: its budget alone ends it. */ + private static final long NO_CAP = Long.MAX_VALUE; private final ApiModel model = new SwaggerSpecificationParser().parse("pet-shelter.yaml"); private final ApiEngine engine = new ApiEngine(); @@ -134,6 +138,30 @@ void the_whole_budget_is_spent() { .isEqualTo(outcome.sent()); } + @Test + @DisplayName("told its size, a run stops once it has sent that many requests, having waited for their answers") + void a_run_of_a_known_size_stops_after_that_many_requests() { + engine.takes(request -> Duration.ofMillis(1)); + long began = System.nanoTime(); + + RunLoop.Outcome outcome = run(Duration.ofSeconds(30), WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, + 12, WITH_A_FIRST_ROUND, new StopFromOutside(), listening -> { }); + + assertThat(outcome.sent()) + .describedAs("the first round's requests count towards the size: %s", outcome) + .isEqualTo(12); + assertThat(outcome.answered()).isEqualTo(12); + assertThat(outcome.stillOwed()) + .describedAs("the answers to the last requests were waited for: %s", outcome) + .isZero(); + assertThat(outcome.cutShort()) + .describedAs("a run that ends at its size ended as it was meant to") + .isFalse(); + assertThat(Duration.ofNanos(System.nanoTime() - began)) + .describedAs("and it ended long before its budget of thirty seconds") + .isLessThan(Duration.ofSeconds(10)); + } + @Test @DisplayName("one slow operation does not stop the rest of the API being tested") void a_slow_operation_does_not_hold_up_the_run() { @@ -322,7 +350,7 @@ void a_lost_step_of_a_series_is_heard_and_its_series_goes_on() { Scheduler scheduler = new Scheduler(generator, WITHOUT_A_FIRST_ROUND, Instant.now().plus(BUDGET), InstantSource.system(), events::publish); outcome = RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, - ANNOUNCEMENTS_ALLOWED, PATIENT, GRACE, new StopFromOutside(), engine, events); + ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, events); } assertThat(outcome.lost()).describedAs("list reads were sent and lost: %s", outcome) @@ -459,7 +487,7 @@ void each_step_is_built_from_what_the_step_before_brought_back() { // Three seconds: the whole first round has to fit, on the slowest machine CI has. Scheduler scheduler = new Scheduler(generator, WITH_A_FIRST_ROUND, Instant.now().plusSeconds(3), InstantSource.system(), events::publish); - RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, + RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, events); } @@ -508,7 +536,7 @@ void a_series_waits_for_each_answer() { deadline = System.nanoTime() + BUDGET.toNanos(); Scheduler scheduler = new Scheduler(generator, WITHOUT_A_FIRST_ROUND, Instant.now().plus(BUDGET), InstantSource.system(), events::publish); - RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, + RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, events); } @@ -553,7 +581,8 @@ void a_request_never_answered_holds_the_first_round_up_for_one_wait_at_most() { ScheduleSettings patientForAFifth = new ScheduleSettings( WITH_A_FIRST_ROUND.workAheadFactor(), ANNOUNCEMENTS_ALLOWED, WITH_A_FIRST_ROUND.stragglerGrace(), WITH_A_FIRST_ROUND.interruptGrace(), true, - Duration.ofMillis(200)); + Duration.ofMillis(200), false, 1_000, WITH_A_FIRST_ROUND.interruptWritingGrace(), + WITH_A_FIRST_ROUND.interruptLookingGrace()); long began = System.nanoTime(); RunLoop.Outcome outcome = run(Duration.ofMillis(2500), patientForAFifth, listening -> { }); @@ -650,7 +679,9 @@ void a_stop_during_the_first_round_cuts_it_short() { engine.takes(request -> Duration.ofSeconds(20)); ScheduleSettings patient = new ScheduleSettings(WITH_A_FIRST_ROUND.workAheadFactor(), ANNOUNCEMENTS_ALLOWED, WITH_A_FIRST_ROUND.stragglerGrace(), - WITH_A_FIRST_ROUND.interruptGrace(), true, Duration.ofSeconds(10)); + WITH_A_FIRST_ROUND.interruptGrace(), true, Duration.ofSeconds(10), false, 1_000, + WITH_A_FIRST_ROUND.interruptWritingGrace(), + WITH_A_FIRST_ROUND.interruptLookingGrace()); List ended = new CopyOnWriteArrayList<>(); StopFromOutside stop = stoppedAfter(Duration.ofMillis(400)); @@ -685,7 +716,7 @@ void an_answer_after_the_wait_is_not_announced() throws InterruptedException { Scheduler scheduler = new Scheduler(generator(), WITHOUT_A_FIRST_ROUND, Instant.now().plusSeconds(30), InstantSource.system(), events::publish); outcome = RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, - ANNOUNCEMENTS_ALLOWED, PATIENT, GRACE, stop, engine, events); + ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, stop, engine, events); Thread.sleep(1500); events.awaitDelivery(PATIENT); } @@ -711,7 +742,7 @@ void a_loop_that_fails_announces_nothing_afterwards() throws InterruptedExceptio Scheduler scheduler = new Scheduler(generator(), WITHOUT_A_FIRST_ROUND, Instant.now().plusSeconds(30), InstantSource.system(), events::publish); org.assertj.core.api.Assertions.assertThatThrownBy(() -> RunLoop.run(scheduler, - "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, PATIENT, GRACE, + "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, events)) .isInstanceOf(IllegalStateException.class); Thread.sleep(1200); @@ -733,7 +764,7 @@ void a_wait_too_long_to_count_is_still_a_wait() { Scheduler scheduler = new Scheduler(generator(), WITHOUT_A_FIRST_ROUND, Instant.now().plusMillis(300), InstantSource.system(), events::publish); outcome = RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, - ANNOUNCEMENTS_ALLOWED, Duration.ofMillis(Long.MAX_VALUE), + ANNOUNCEMENTS_ALLOWED, NO_CAP, Duration.ofMillis(Long.MAX_VALUE), Duration.ofMillis(Long.MAX_VALUE), new StopFromOutside(), engine, events); } @@ -768,12 +799,19 @@ private RunLoop.Outcome run(Duration budget, int workAhead, int announcementsAll private RunLoop.Outcome run(Duration budget, int workAhead, int announcementsAllowed, ScheduleSettings schedule, StopFromOutside stop, java.util.function.Consumer setUp) { + return run(budget, workAhead, announcementsAllowed, NO_CAP, schedule, stop, setUp); + } + + /** The same, stopping once this many requests have been sent. */ + private RunLoop.Outcome run(Duration budget, int workAhead, int announcementsAllowed, + long mostRequests, ScheduleSettings schedule, StopFromOutside stop, + java.util.function.Consumer setUp) { try (EventStream events = new EventStream()) { setUp.accept(events); Scheduler scheduler = new Scheduler(generator(), schedule, Instant.now().plus(budget), InstantSource.system(), events::publish); return RunLoop.run(scheduler, "https://api.example", workAhead, announcementsAllowed, - PATIENT, GRACE, stop, engine, events); + mostRequests, PATIENT, GRACE, stop, engine, events); } } @@ -796,7 +834,7 @@ private RunLoop.Outcome runAgainst(String baseUrl, ScheduleSettings schedule) { try (EventStream events = new EventStream()) { Scheduler scheduler = new Scheduler(generator(), schedule, Instant.now().plusSeconds(30), InstantSource.system(), events::publish); - return RunLoop.run(scheduler, baseUrl, WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, PATIENT, + return RunLoop.run(scheduler, baseUrl, WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, events); } } diff --git a/restest-cli/src/test/java/io/restest/cli/SequencesFindWhatSingleRequestsMissTest.java b/restest-cli/src/test/java/io/restest/cli/SequencesFindWhatSingleRequestsMissTest.java index edd07b931..71d98f8a5 100644 --- a/restest-cli/src/test/java/io/restest/cli/SequencesFindWhatSingleRequestsMissTest.java +++ b/restest-cli/src/test/java/io/restest/cli/SequencesFindWhatSingleRequestsMissTest.java @@ -61,6 +61,14 @@ */ class SequencesFindWhatSingleRequestsMissTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String FIFTEEN_HUNDRED_REQUESTS = "schedule.requestsBeforeStopping=1500"; + private static final String SPECIFICATION = """ openapi: 3.0.3 info: {title: Forgetful widgets, version: "1.0"} @@ -203,7 +211,7 @@ private static String run(Path out, Path document, Path plan, String... extra) { List arguments = new ArrayList<>(List.of( "run", document.toString(), "--url", api.baseUrl(), - "--budget", "3s", + "--budget", "30s", "--set", STOP_AFTER_REQUESTS, "--set", FIFTEEN_HUNDRED_REQUESTS, "--seed", "20261005", "--campaign", plan.toString(), "--out", out.toString())); diff --git a/restest-cli/src/test/java/io/restest/cli/SettingsCommandTest.java b/restest-cli/src/test/java/io/restest/cli/SettingsCommandTest.java index f24fa8e94..5be6af2f1 100644 --- a/restest-cli/src/test/java/io/restest/cli/SettingsCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/SettingsCommandTest.java @@ -49,6 +49,14 @@ */ class SettingsCommandTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; + private static WireMockServer api; private final StringWriter screen = new StringWriter(); @@ -194,7 +202,8 @@ class Refusals { @Test @DisplayName("a setting that does not exist answers 2, naming the one it looks like") void a_misspelt_setting(@TempDir Path directory) { - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--set", "engine.maxConcurrancy=8", "--out", directory.resolve("out").toString())) .isEqualTo(ExitCode.BAD_COMMAND_LINE); @@ -207,7 +216,8 @@ void a_misspelt_setting(@TempDir Path directory) { @Test @DisplayName("a value the setting cannot take answers 2, saying what it wanted") void a_value_of_the_wrong_kind(@TempDir Path directory) { - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--set", "engine.maxConcurrency=lots", "--out", directory.resolve("out").toString())) .isEqualTo(ExitCode.BAD_COMMAND_LINE); @@ -251,7 +261,8 @@ void values_the_tool_cannot_write_down(String typed) { @Test @DisplayName("a settings file that cannot be read answers 2, and nothing is sent") void a_file_that_cannot_be_read(@TempDir Path directory) { - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--settings", directory.resolve("nowhere.yaml").toString(), "--out", directory.resolve("out").toString())) .isEqualTo(ExitCode.BAD_COMMAND_LINE); @@ -270,7 +281,8 @@ class ARunThatWasConfigured { void one_request_at_a_time(@TempDir Path directory) throws Exception { Path out = directory.resolve("out"); - assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + assertThat(run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--set", "engine.maxConcurrency=1", "--out", out.toString())) .isBetween(0, 1); @@ -290,7 +302,8 @@ void the_report_carries_the_settings(@TempDir Path directory) throws Exception { writeUpsInTotal: 7 """); - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--settings", file.toString(), "--out", out.toString()); List written = settingsIn(out.resolve("report.json")); @@ -303,7 +316,9 @@ void the_report_carries_the_settings(@TempDir Path directory) throws Exception { @DisplayName("the run says on the screen that it was configured, because an environment " + "variable is invisible in the command somebody typed") void the_run_says_it_was_configured(@TempDir Path directory) { - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + // What this is about is printed before anything is sent, so the run is given no time + // to send anything, rather than a size, which would be two more settings changed. + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1ms", "--set", "engine.maxConcurrency=8", "--out", directory.resolve("out").toString()); @@ -315,7 +330,8 @@ void the_run_says_it_was_configured(@TempDir Path directory) { @Test @DisplayName("and says nothing about settings when nobody changed any") void a_plain_run_says_nothing(@TempDir Path directory) { - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1s", + // A plain run changes no setting, a size included. + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "1ms", "--out", directory.resolve("out").toString()); assertThat(screen.toString()).doesNotContain("not what RESTest does by default"); @@ -335,9 +351,11 @@ void a_setting_the_run_obeys(@TempDir Path directory) throws Exception { .withHeader("Content-Type", "application/json") .withBody("\"not the shape the document promised\""))); try { - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "2s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "5", "--out", usual.toString()); - run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "2s", + run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "5", "--set", "report.writeUpsInTotal=0", "--out", quiet.toString()); } finally { diff --git a/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java b/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java index aefcc99b1..0b9f3efe3 100644 --- a/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java +++ b/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java @@ -66,6 +66,14 @@ */ class WholeRunTest { + /** + * The runs here are given a size rather than a length: the same requests on a slow build + * machine as on a fast one, and no second of the clock spent waiting for a budget to run out. + * The budget they are also given is only the ceiling a run that went wrong could not get past. + */ + private static final String STOP_AFTER_REQUESTS = "schedule.stopAfterRequests=true"; + private static final String SIXTY_REQUESTS = "schedule.requestsBeforeStopping=60"; + private static WireMockServer api; @BeforeAll @@ -112,7 +120,8 @@ void one_command_finds_and_reports_what_is_wrong(@TempDir Path directory) throws // --store because this is the test that proves the stored run and the report agree, and a // run is not asked to keep anything unless it is told to. int answer = run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", "1s", "--seed", "20260913", "--out", directory.toString(), "--store"); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260913", "--out", directory.toString(), "--store"); assertThat(answer).describedAs("faults were found, so the answer is 1").isEqualTo(1); @@ -164,11 +173,9 @@ void the_printed_command_reproduces_the_fault(@TempDir Path directory) throws Ex Assumptions.assumeTrue(curlIsInstalled(), "curl is not installed on this machine"); StringWriter screen = new StringWriter(); - // Two seconds rather than a fraction of one: reading the document and warming the parser - // up costs about half a second, and that comes out of the budget, so a budget of a few - // hundred milliseconds can run out before a single request has gone anywhere. run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), - "--budget", "2s", "--seed", "1", "--out", directory.toString()); + "--budget", "30s", + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.toString()); String command = screen.toString().lines() .map(String::trim) diff --git a/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java b/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java index 9bed7131f..c2122a877 100644 --- a/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java +++ b/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java @@ -33,7 +33,13 @@ * run somebody stops from outside - with Ctrl-C, say - which waits much less, because whoever * stopped it is waiting too. * - *

The last two are about how a run begins. Before anything is chosen by chance, a run can send + *

Two more say how a run stopped from outside ends: how long it may spend writing what it found + * once it has stopped waiting for its answers, and how long it then gives its directory to say what + * was and was not written, when writing did not finish in time. Two others let a run be given a + * size rather than a length: told to stop after so many requests, it ends when they have been + * answered, however fast or slow the machine it runs on. + * + *

The ones about the opening lap are about how a run begins. Before anything is chosen by chance, a run can send * every operation once, each with the request the API is most likely to accept: it first asks for * the lists of what is there, then creates, then asks for single things, then changes them, and * deletes last. Each of those steps waits for the answers to the one before, so that an identifier @@ -59,6 +65,20 @@ * most likely to be accepted, before anything is chosen by chance * @param openingLapPatience how long one step of that first round waits for the answers to the step * before it, and for what they said to be taken in, before going on anyway. Zero never waits + * @param stopAfterRequests whether a run stops once it has sent a set number of requests, before + * its budget is spent. Off, the budget alone says when a run ends. On, a run of a known size + * can be asked for, which ends as soon as the API has answered rather than when a clock says, + * and sends the same requests on a slow machine as on a fast one + * @param requestsBeforeStopping how many requests that is. The run still waits for the answers to + * the last of them, as it does at the end of its budget + * @param interruptWritingGrace how long a run stopped from outside is given to write what it + * found, once it has stopped waiting for its answers, before the program is let end without + * it. Writing takes a fraction of a second; this is how long to wait for writing that has got + * stuck before saying so, since whoever stopped the run is about to end it outright: a + * container is given ten seconds in all + * @param interruptLookingGrace how long such a run, when it ran out of time to write, gives its + * directory to say what is in it before saying that the directory did not answer. A directory + * answers at once, unless it is the reason the writing ran out of time */ public record ScheduleSettings( int workAheadFactor, @@ -66,15 +86,36 @@ public record ScheduleSettings( Duration stragglerGrace, Duration interruptGrace, boolean openingLap, - Duration openingLapPatience) { + Duration openingLapPatience, + boolean stopAfterRequests, + int requestsBeforeStopping, + Duration interruptWritingGrace, + Duration interruptLookingGrace) { private static final ScheduleSettings DEFAULTS = new ScheduleSettings(2, 1_000, - Duration.ofSeconds(10), Duration.ofSeconds(2), true, Duration.ofSeconds(2)); + Duration.ofSeconds(10), Duration.ofSeconds(2), true, Duration.ofSeconds(2), + false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); public ScheduleSettings { Objects.requireNonNull(stragglerGrace, "stragglerGrace"); Objects.requireNonNull(interruptGrace, "interruptGrace"); Objects.requireNonNull(openingLapPatience, "openingLapPatience"); + Objects.requireNonNull(interruptWritingGrace, "interruptWritingGrace"); + Objects.requireNonNull(interruptLookingGrace, "interruptLookingGrace"); + if (requestsBeforeStopping < 1) { + throw new IllegalArgumentException("requestsBeforeStopping must be at least 1, since a " + + "run that stops before its first request tests nothing: " + + requestsBeforeStopping); + } + if (interruptWritingGrace.isNegative()) { + throw new IllegalArgumentException("interruptWritingGrace cannot be negative; zero " + + "means a run stopped from outside is given no time to write what it found: " + + interruptWritingGrace); + } + if (interruptLookingGrace.isNegative()) { + throw new IllegalArgumentException("interruptLookingGrace cannot be negative; zero " + + "means the directory is never asked what it holds: " + interruptLookingGrace); + } if (workAheadFactor < 1) { throw new IllegalArgumentException("workAheadFactor must be at least 1, since a run " + "that may not work ahead of the engine at all leaves it idle between " diff --git a/restest-core/src/main/java/io/restest/core/settings/SettingKey.java b/restest-core/src/main/java/io/restest/core/settings/SettingKey.java index 83007f8ad..27942a47d 100644 --- a/restest-core/src/main/java/io/restest/core/settings/SettingKey.java +++ b/restest-core/src/main/java/io/restest/core/settings/SettingKey.java @@ -198,6 +198,16 @@ private static SettingKey key(String group, String name, SettingKind kind, Strin "whether a run starts by sending every operation once, the request likeliest to work"), key("schedule", "openingLapPatience", SettingKind.LENGTH_OF_TIME, "how long each step of that opening lap waits for the answers to the one before"), + key("schedule", "stopAfterRequests", SettingKind.YES_OR_NO, + "whether a run stops once it has sent a set number of requests, before its " + + "budget is spent"), + key("schedule", "requestsBeforeStopping", SettingKind.WHOLE_NUMBER, + "how many requests that is"), + key("schedule", "interruptWritingGrace", SettingKind.LENGTH_OF_TIME, + "how long a run stopped from outside is then given to write what it found"), + key("schedule", "interruptLookingGrace", SettingKind.LENGTH_OF_TIME, + "how long it then gives its directory to say what was written, when writing " + + "did not finish"), key("generation", "optionalNestingDepth", SettingKind.WHOLE_NUMBER, "below this depth, only what the description insists on is built"), diff --git a/restest-core/src/main/java/io/restest/core/settings/Settings.java b/restest-core/src/main/java/io/restest/core/settings/Settings.java index ef03328a5..22083962d 100644 --- a/restest-core/src/main/java/io/restest/core/settings/Settings.java +++ b/restest-core/src/main/java/io/restest/core/settings/Settings.java @@ -151,7 +151,15 @@ public static Settings from(Map given) { DEFAULTS.schedule.interruptGrace()), typed.yesOrNo("schedule.openingLap", DEFAULTS.schedule.openingLap()), typed.lengthOfTime("schedule.openingLapPatience", - DEFAULTS.schedule.openingLapPatience()))); + DEFAULTS.schedule.openingLapPatience()), + typed.yesOrNo("schedule.stopAfterRequests", + DEFAULTS.schedule.stopAfterRequests()), + typed.wholeNumber("schedule.requestsBeforeStopping", + DEFAULTS.schedule.requestsBeforeStopping()), + typed.lengthOfTime("schedule.interruptWritingGrace", + DEFAULTS.schedule.interruptWritingGrace()), + typed.lengthOfTime("schedule.interruptLookingGrace", + DEFAULTS.schedule.interruptLookingGrace()))); GenerationSettings generation = group("generation", () -> new GenerationSettings( typed.wholeNumber("generation.optionalNestingDepth", DEFAULTS.generation.optionalNestingDepth()), @@ -360,6 +368,13 @@ public String written(SettingKey key) { case "schedule.openingLap" -> String.valueOf(schedule.openingLap()); case "schedule.openingLapPatience" -> LengthOfTime.written(schedule.openingLapPatience()); + case "schedule.stopAfterRequests" -> String.valueOf(schedule.stopAfterRequests()); + case "schedule.requestsBeforeStopping" -> + String.valueOf(schedule.requestsBeforeStopping()); + case "schedule.interruptWritingGrace" -> + LengthOfTime.written(schedule.interruptWritingGrace()); + case "schedule.interruptLookingGrace" -> + LengthOfTime.written(schedule.interruptLookingGrace()); case "generation.optionalNestingDepth" -> String.valueOf(generation.optionalNestingDepth()); diff --git a/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java b/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java index 4307a065b..27b148d0a 100644 --- a/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java +++ b/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java @@ -440,13 +440,22 @@ void a_value_outside_its_range() { .hasMessageContaining("at least 1"); assertThatThrownBy(() -> new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), true, Duration.ofSeconds(-1))) + Duration.ofSeconds(2), true, Duration.ofSeconds(-1), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1))) .isInstanceOf(IllegalArgumentException.class) .hasMessageContaining("openingLapPatience cannot be negative"); assertThatThrownBy(() -> new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(-1), true, Duration.ofSeconds(2))) + Duration.ofSeconds(-1), true, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1))) .isInstanceOf(IllegalArgumentException.class) .hasMessageContaining("interruptGrace cannot be negative"); + assertThatThrownBy(() -> Settings.from(Map.of("schedule.requestsBeforeStopping", "0"))) + .isInstanceOf(SettingsException.class) + .hasMessageStartingWith("schedule: ") + .hasMessageContaining("at least 1"); + assertThatThrownBy(() -> new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), + Duration.ofSeconds(2), true, Duration.ofSeconds(2), true, 10, + Duration.ofSeconds(-1), Duration.ofSeconds(1))) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("interruptWritingGrace cannot be negative"); } /** @@ -538,7 +547,7 @@ void each_group_can_be_replaced() { assertThat(from.withEngine(from.engine().withoutConcurrency()).engine() .maxConcurrency()).isEqualTo(1); assertThat(from.withSchedule(new ScheduleSettings(3, 10, Duration.ofSeconds(1), - Duration.ofSeconds(2), true, Duration.ofSeconds(2))) + Duration.ofSeconds(2), true, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1))) .schedule().workAheadFactor()).isEqualTo(3); assertThat(from.withGeneration(GenerationSettings.defaults()).generation()) .isEqualTo(GenerationSettings.defaults()); diff --git a/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java b/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java index 4ac31c957..25c4ca0f7 100644 --- a/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java +++ b/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java @@ -86,7 +86,7 @@ void the_scheduler_without_a_first_round_sends_what_was_always_sent() throws IOE ApiModel model = petClinic(); RandomTestCaseGenerator generator = withoutSeries(model); ScheduleSettings noFirstRound = new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), false, Duration.ofSeconds(2)); + Duration.ofSeconds(2), false, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); Instant now = Instant.parse("2026-09-23T10:00:00Z"); Scheduler scheduler = new Scheduler(generator, noFirstRound, now.plusSeconds(60), InstantSource.fixed(now), announced -> { }); diff --git a/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java b/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java index f9c117607..49eeca70a 100644 --- a/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java +++ b/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java @@ -111,7 +111,7 @@ void a_wait_ends_at_the_deadline_at_the_latest() { @DisplayName("told not to wait at all, the first round goes straight from one step to the next") void no_patience_means_no_waiting() { ScheduleSettings impatient = new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), true, Duration.ZERO); + Duration.ofSeconds(2), true, Duration.ZERO, false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); Scheduler scheduler = scheduler(PETS, impatient, Duration.ofMinutes(1)); assertThat(steps(scheduler, 5)).containsExactly("send listPets (first round)", @@ -364,6 +364,6 @@ private static List steps(Scheduler scheduler, int howMany) { private static ScheduleSettings withoutAFirstRound() { return new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), false, Duration.ofSeconds(2)); + Duration.ofSeconds(2), false, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); } } diff --git a/restest-gen/src/test/java/io/restest/gen/SequencesTest.java b/restest-gen/src/test/java/io/restest/gen/SequencesTest.java index 843ab8ed3..2cf1e3649 100644 --- a/restest-gen/src/test/java/io/restest/gen/SequencesTest.java +++ b/restest-gen/src/test/java/io/restest/gen/SequencesTest.java @@ -393,7 +393,9 @@ void the_oldest_step_waiting_is_forgotten() { Settings oneAtATime = Settings.defaults() .withSchedule(new ScheduleSettings(1, schedule.announcementsAllowedToPileUp(), schedule.stragglerGrace(), schedule.interruptGrace(), schedule.openingLap(), - schedule.openingLapPatience())) + schedule.openingLapPatience(), schedule.stopAfterRequests(), + schedule.requestsBeforeStopping(), schedule.interruptWritingGrace(), + schedule.interruptLookingGrace())) .withEngine(EngineSettings.defaults().withConcurrency(1, 1, 1)) .withSequences(only(Sequences.Shape.CREATE_TWICE)); RandomTestCaseGenerator generator = new RandomTestCaseGenerator(PET_CLINIC, SEED, From 4ac093f190ab8ea5aff330a9cd8f380667217834 Mon Sep 17 00:00:00 2001 From: Sergio Segura Date: Tue, 13 Oct 2026 22:38:55 +0200 Subject: [PATCH 2/4] Run the command-line tests three at a time, score the oracles' mutations on one row, and stop watching v2 Three changes to how a build is checked, none to what is checked. The command-line module's tests run in three processes instead of one (`forkCount` in its POM). They are the command run end to end, each class against a stand-in API on a port of its own and writing under a directory of its own, and most of their time is waiting for an API to answer; the module took three of the four minutes of `./mvnw verify`, and takes a third of that now. Nothing a run keeps is shared between two processes, which the architecture rules' ban on static mutable state is there to guarantee. The mutation score of the oracles is checked on every change, on the `ubuntu-latest` / Java 25 row, as a step after the build. `docs/ci.md` said it "costs minutes" and so was left to be run by hand, which nobody did unless a pull request touched the oracles; it takes eleven seconds (176 mutations, 97% killed). A `mutation` profile in `restest-oracles/pom.xml` binds the score to that module's `verify`, so the ordinary build still does not pay for it. `ci.yml` no longer runs for `v2`, a branch no commit has reached since 3 October and that is seventy-six behind `master`. The CI page says what runs now, names the five rules its table of gates had left out, and loses four sentences from the weeks when the modules were empty. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 16 ++++++++++--- docs/ci.md | 32 ++++++++++++++++---------- restest-cli/pom.xml | 13 +++++++++++ restest-oracles/pom.xml | 49 ++++++++++++++++++++++++++++++++-------- 4 files changed, 86 insertions(+), 24 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1ec134066..9163cc180 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,4 +1,4 @@ -# Continuous integration for RESTest 2.0. +# Continuous integration for RESTest. # # Three operating systems x Java 21, 25 and 26, every row building and testing the whole reactor # (ADR-0003). Every action is pinned to a commit SHA rather than a tag: a tag can be repointed at @@ -8,9 +8,9 @@ name: CI on: push: - branches: [master, v2] + branches: [master] pull_request: - branches: [master, v2] + branches: [master] workflow_dispatch: # A new push supersedes the runs still queued for the same ref. @@ -52,6 +52,16 @@ jobs: - name: Build and test run: ./mvnw --batch-mode --no-transfer-progress verify + # The mutation score of the oracles, from one row only: the score is the same whatever the + # operating system and the Java, and it costs about ten seconds, which is the module's tests + # run once per mutation. The profile binds the score to this module's verify, and the build + # reaches the module through the ones it depends on, which is what resolves them without an + # install. Coverage is switched off because the step above has measured it already, and the + # agent would otherwise ride along into every mutant's run. + - name: Check the mutation score of the oracles + if: matrix.os == 'ubuntu-latest' && matrix.java == '25' + run: ./mvnw --batch-mode --no-transfer-progress -Pmutation verify -pl restest-oracles -am -Djacoco.skip=true + # Uploaded from one row only: nine identical reports would be noise. - name: Upload the coverage report if: matrix.os == 'ubuntu-latest' && matrix.java == '25' diff --git a/docs/ci.md b/docs/ci.md index f19902f88..bb3063551 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -5,8 +5,8 @@ behind it are in ADR-0003 (Java versions), ADR-0004 (module boundaries) and ADR- Two workflows. `ci.yml` builds and tests every change; `release.yml` makes a release from a version tag, and rehearses it on a pull request that touches how RESTest is packaged -([Releases](#releases)). `ci.yml`'s triggers are pull requests against `master` or `v2`, pushes to -those two branches, and manual dispatch. A push to a `feat/*` branch with no open pull request runs +([Releases](#releases)). `ci.yml`'s triggers are pull requests against `master`, pushes to it, +and manual dispatch. A push to a `feat/*` branch with no open pull request runs nothing, which is deliberate: a branch under active development would otherwise spend nine runners per commit. @@ -94,13 +94,13 @@ itself, in its first stage, and needs nothing the smoke run built, so the two ru |---|---|---| | Compilation and unit tests | every module | the obvious | | Java 21 bytecode | `PublishedBytecodeTest` | a class file compiled for a later release; a published jar that does not say which version of RESTest it is | -| Module boundaries | `ProductionArchitectureTest` | a dependency pointing outwards; a class in no module; `io.swagger` outside `restest-spec`; the schema validator outside `restest-oracles`; the JSON library outside `restest-core`; the database driver outside `restest-store`; process termination outside `restest-cli` (`System.exit`, `Runtime.exit`/`halt`, this JVM's `ProcessHandle.destroy`, whether called or referenced); a reassignable static field; a network dependency in `restest-core` | +| Module boundaries | `ProductionArchitectureTest` | a dependency pointing outwards; a class in no module; `io.swagger` outside `restest-spec`; the HTTP client outside `restest-exec`; the schema validator outside `restest-oracles`; the JSON library outside `restest-core`; the YAML reader outside `restest-core` and `restest-spec`; the regular-expression builder outside `restest-gen`; the command-line framework outside `restest-cli`; the database driver outside `restest-store`; process termination outside `restest-cli` (`System.exit`, `Runtime.exit`/`halt`, this JVM's `ProcessHandle.destroy`, whether called or referenced); the environment or the system properties read outside `restest-cli`; a random generator asked for by name; a reassignable static field; a network dependency in `restest-core` | | The published JSON Schemas | `PublishedSchemasTest` | a schema and RESTest's reader disagreeing about a file: a whole example in `README.md` or `docs/`, the plan RESTest carries, or a file with one thing broken; a member or a word one allows and the other does not; a keyword the engines that hold a language model to a schema do not all accept; an `$id` that does not name the version being prepared | | The rules themselves | `ArchitectureRulesSelfTest`, and for the source-tree rules the self-tests in `SourceTreeRulesTest` | a rule that no longer reports the violation it exists to report, or that reports something it should permit | | The harness's reach | `HarnessCoverageTest` | a module whose compiled classes the rules cannot see, so no rule constrains them | | Source-tree invariants | `SourceTreeRulesTest` | a module without `module-info.java`; a benchmark-platform reference in any file of this repository, or in the name of one, outside the documents the rule names; one of those no longer needing to be named there; an action pinned to a tag, a SHA with no version comment, or no pins found at all | | Coverage | JaCoCo | `restest-core` or `restest-oracles` falling below 90% of lines or 85% of branches; measured everywhere else without blocking | -| Mutation score | PIT | run on demand, not in the build: `restest-oracles` scoring below 85% | +| Mutation score | PIT | `restest-oracles` scoring below 85%, on the `ubuntu-latest` / Java 25 row (`-Pmutation`) and whenever it is run by hand | | End-to-end smoke run | `SmokeRunTest`, in the `smoke` job | the command failing against two real containerised APIs, or answering 2, 3 or 4 rather than "ran, and here is what I found". Runs only where a container runtime exists, and fails rather than skips when it was asked for | | How RESTest starts | `check-restest.sh`, in the `smoke` and `image` jobs and in `release.yml` | the image or a launcher failing to start, saying the wrong version, or answering with a number other than RESTest's own: `2` for a wrong option, `3` when nothing answers, `127` when there is no Java it can use, `143` when `docker stop` ends a run in the image; the image, or a release's folder, without the JSON Schemas of a dictionary and a plan | | Runs on a plain runtime | `StockJreRunTest`, in the `smoke` job | the tool reaching for a part of Java that a runtime carrying only the compulsory modules does not have, which stops it before its first request; a seed meaning a different run there than it does here; and a key left in `RESTEST_AUTH` failing to reach an API that refuses every request without it, or being written into any file the run leaves behind — the one place the variable is read from a real environment rather than handed to the command by a test. Pinned to Java 21, the oldest release supported and one of the two where the generators RESTest once named are optional; the gate refuses to run on an image where they are not | @@ -110,7 +110,7 @@ itself, in its first stage, and needs nothing the smoke run built, so the two ru JaCoCo's agent is attached to every module's test run, and `report` runs at `verify`. A module only produces `target/site/jacoco/` once it has tests: with no tests there is no execution data, so the report goal skips and says so. `restest-core` has both, so an HTML report is written -under its `target/site/jacoco/` on every build — the first coverage output the project has had. +under its `target/site/jacoco/` on every build. The ubuntu / 25 row uploads whatever exists as the `jacoco-report` artifact, with `if-no-files-found: warn`. Warn rather than ignore on purpose: a silent green upload of nothing @@ -139,8 +139,16 @@ in itself. ./mvnw org.pitest:pitest-maven:mutationCoverage -pl restest-oracles ``` -It is not bound to a build phase: it re-runs the tests once per mutation, so it costs minutes where -the rest of the build costs seconds. The threshold is **85%**, against 98% measured when it was set. +It re-runs the module's tests once per mutation, which takes about ten seconds, so one row of the +matrix - `ubuntu-latest` with Java 25 - runs it on every change, as a second step after the build: + +```bash +./mvnw --batch-mode --no-transfer-progress -Pmutation verify -pl restest-oracles -am -Djacoco.skip=true +``` + +The `mutation` profile, in `restest-oracles/pom.xml`, binds the score to that module's `verify`; +the ordinary `./mvnw verify` does not run it, so a laptop pays for the score only when it asks. The +threshold is **85%**, against 98% measured when it was set and 97% measured on 13 October 2026. Three survive, and they are named here rather than called equivalent, which they are not quite. All three are a `<` moved to a `<=` inside the schema oracle: how many pointers it will follow before @@ -163,7 +171,7 @@ classes, the skips are gone, and what stood in their place is now an assertion: the reactor was not built, and says so. `./mvnw verify` says: ``` -[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0 +[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0 ``` Second, `ArchitectureRulesSelfTest` proves the rules independently of whether production code @@ -182,8 +190,8 @@ that compiled five hundred and shipped one. `allowEmptyShould(true)` remains on the individual production checks, for a reason that outlasts the first weeks: the modules fill in at different times, so a rule scoped to one of them — the parser -confinement rule, now that `restest-core` holds classes and `restest-spec` does not yet — -legitimately has an empty subject set for a while. What it must not excuse is every rule being empty +confinement rule, while `restest-core` held classes and `restest-spec` did not yet — +legitimately had an empty subject set for a while. What it must not excuse is every rule being empty at once, and that cannot happen quietly: the inward-dependency and no-network rules both have subjects, and `HarnessCoverageTest` fails if a module ever leaves the import. @@ -249,8 +257,8 @@ dist/assemble.sh target/dist --archives ``` There is no shortcut worth having here: `restest-arch-tests` depends on all nine modules, so `-am` -builds the whole reactor anyway. On this project the full build takes a few seconds; just run -`./mvnw verify`. +builds the whole reactor anyway. The full build takes a couple of minutes, most of it the +command-line module's tests, which run three classes at a time; just run `./mvnw verify`. Worth knowing where the rules are actually looking. Under `verify` the reactor has reached `package`, so what the rules read is each module's **jar**, not its `target/classes` — only diff --git a/restest-cli/pom.xml b/restest-cli/pom.xml index 0e69f56fe..46cf1cb95 100644 --- a/restest-cli/pom.xml +++ b/restest-cli/pom.xml @@ -117,6 +117,19 @@ such load. Same flag, same reason, as restest-store. --> @{argLine} --enable-native-access=ALL-UNNAMED + + 3 + true diff --git a/restest-oracles/pom.xml b/restest-oracles/pom.xml index 679f064d2..2e3629032 100644 --- a/restest-oracles/pom.xml +++ b/restest-oracles/pom.xml @@ -47,18 +47,20 @@ a build pass; a rule that gets relaxed whenever it fires is not a rule. --> org.pitest @@ -109,4 +111,33 @@ + + + + + mutation + + + + org.pitest + pitest-maven + + + mutation-score + verify + + mutationCoverage + + + + + + + + From 11742fe90009861214d92b830a51a25f0ffff5d5 Mon Sep 17 00:00:00 2001 From: Sergio Segura Date: Wed, 14 Oct 2026 17:54:39 +0200 Subject: [PATCH 3/4] Keep the lines this branch added within a hundred columns Wrapping only: the arguments of the runs given a size, three constructions of the schedule settings in tests, one test's name and one paragraph of Javadoc. Nothing a test does changes. Co-Authored-By: Claude Opus 5.5 --- .../restest/cli/AKeyIsWrittenNowhereTest.java | 3 ++- .../java/io/restest/cli/AuthCommandTest.java | 27 ++++++++++++++----- .../io/restest/cli/CampaignCommandTest.java | 3 ++- .../io/restest/cli/OpeningLapCommandTest.java | 3 ++- .../test/java/io/restest/cli/RestestTest.java | 27 +++++++++++++------ .../test/java/io/restest/cli/RunLoopTest.java | 13 +++++---- .../java/io/restest/cli/WholeRunTest.java | 7 +++-- .../core/settings/ScheduleSettings.java | 16 +++++------ .../restest/core/settings/SettingsTest.java | 9 ++++--- .../java/io/restest/gen/RunOrderTest.java | 3 ++- .../java/io/restest/gen/SchedulerTest.java | 6 +++-- 11 files changed, 78 insertions(+), 39 deletions(-) diff --git a/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java b/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java index cb258e15c..5ffc94f9f 100644 --- a/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java +++ b/restest-cli/src/test/java/io/restest/cli/AKeyIsWrittenNowhereTest.java @@ -196,7 +196,8 @@ void no_key_is_written_anywhere(@TempDir Path directory) throws IOException { PrintWriter complained = new PrintWriter(problems)) { answer = Restest.run(new String[] {"run", document.toString(), "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", THREE_HUNDRED_REQUESTS, "--seed", "20260930", "--out", out.toString(), + "--set", STOP_AFTER_REQUESTS, "--set", THREE_HUNDRED_REQUESTS, "--seed", "20260930", + "--out", out.toString(), "--store", "--auth", "inAHeader=" + IN_A_HEADER, "--auth", "inTheQuery=" + IN_THE_QUERY}, printed, complained, Map.of("RESTEST_AUTH", "inACookie=" + IN_A_COOKIE)); diff --git a/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java b/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java index a4712598f..d2953ba1f 100644 --- a/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/AuthCommandTest.java @@ -175,7 +175,8 @@ void in_a_header(@TempDir Path directory) throws IOException { .willReturn(aResponse().withStatus(200))); int answer = run(Map.of(), "run", document(directory, PETSTORE), "--url", - api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", + api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", + "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString(), "--auth", KEY); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); @@ -209,7 +210,8 @@ void in_the_query_and_in_a_cookie(@TempDir Path directory) throws IOException { .atPriority(1).willReturn(aResponse().withStatus(200))); int answer = run(Map.of(), "run", document(directory, TWO_KEYS), "--url", - api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", + api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", + "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString(), "--auth", "token=" + KEY, "--auth", "session=" + cookie); @@ -233,7 +235,8 @@ void with_its_place(@TempDir Path directory) throws IOException { .atPriority(1).willReturn(aResponse().withStatus(200))); int answer = run(Map.of(), "run", document(directory, WORDS), "--url", api.baseUrl(), - "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", + "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", + "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("out").toString(), "--auth", "query:apiKey=" + KEY); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); @@ -280,7 +283,9 @@ void a_bearer_token_already_held(@TempDir Path directory) throws IOException { Path out = directory.resolve("out"); int answer = run(Map.of(), "run", document(directory, signedIn), "--url", - api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", out.toString(), + api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", + "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", + out.toString(), "--store", "--auth", "header:Authorization=Bearer " + token); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); @@ -312,7 +317,10 @@ void it_is_sent_and_a_typed_one_wins(@TempDir Path directory) throws IOException .willReturn(aResponse().withStatus(200))); int alone = run(Map.of("RESTEST_AUTH", KEY), "run", document(directory, PETSTORE), - "--url", api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", + "--url", api.baseUrl(), "--budget", "30s", "--set", + "schedule.stopAfterRequests=true", "--set", + "schedule.requestsBeforeStopping=40", + "--seed", "20260930", "--out", directory.resolve("alone").toString()); assertThat(alone).describedAs("%s%n%s", screen, problems).isBetween(0, 1); assertThat(api.findAll(getRequestedFor(urlPathEqualTo("/pets")))).isNotEmpty() @@ -321,7 +329,9 @@ void it_is_sent_and_a_typed_one_wins(@TempDir Path directory) throws IOException api.resetRequests(); int overruled = run(Map.of("RESTEST_AUTH", "api_key=the-one-left-behind"), "run", - document(directory, PETSTORE), "--url", api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", + document(directory, PETSTORE), "--url", api.baseUrl(), "--budget", "30s", + "--set", "schedule.stopAfterRequests=true", "--set", + "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", directory.resolve("overruled").toString(), "--auth", KEY); assertThat(overruled).isBetween(0, 1); @@ -336,7 +346,10 @@ void one_that_cannot_be_used_only_warns(@TempDir Path directory) throws IOExcept api.stubFor(any(anyUrl()).atPriority(1).willReturn(aResponse().withStatus(200))); int answer = run(Map.of("RESTEST_AUTH", KEY), "run", document(directory, NO_SCHEME), - "--url", api.baseUrl(), "--budget", "30s", "--set", "schedule.stopAfterRequests=true", "--set", "schedule.requestsBeforeStopping=40", "--seed", "20260930", "--out", + "--url", api.baseUrl(), "--budget", "30s", "--set", + "schedule.stopAfterRequests=true", "--set", + "schedule.requestsBeforeStopping=40", + "--seed", "20260930", "--out", directory.resolve("out").toString()); assertThat(answer).describedAs("%s%n%s", screen, problems).isBetween(0, 1); diff --git a/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java b/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java index ab1cd2264..c6d09cec5 100644 --- a/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/CampaignCommandTest.java @@ -276,7 +276,8 @@ void a_plan_that_matches_nothing_says_so_beside_what_could_not_be_read(@TempDir assertThat(run("run", "pet-shop-with-an-operation-it-cannot-read.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", plan.toString(), + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--campaign", + plan.toString(), "--out", directory.resolve("out").toString())) .isEqualTo(3); assertThat(problems.toString()) diff --git a/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java b/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java index 8407d2145..3404c2015 100644 --- a/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java +++ b/restest-cli/src/test/java/io/restest/cli/OpeningLapCommandTest.java @@ -146,7 +146,8 @@ void nothing_declared_means_anything_accepted(@TempDir Path directory) throws IO run(new StringWriter(), "run", document.toString(), "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.resolve("out").toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", + directory.resolve("out").toString()); List pinged = api.findAll(getRequestedFor(urlEqualTo("/ping"))); assertThat(pinged).isNotEmpty(); diff --git a/restest-cli/src/test/java/io/restest/cli/RestestTest.java b/restest-cli/src/test/java/io/restest/cli/RestestTest.java index 60a7372f0..96592f012 100644 --- a/restest-cli/src/test/java/io/restest/cli/RestestTest.java +++ b/restest-cli/src/test/java/io/restest/cli/RestestTest.java @@ -100,7 +100,9 @@ static void stopTheApi() { void faults_found_answer_one(@TempDir Path directory) throws Exception { int answer = run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260914", + "--out", + directory.toString()); assertThat(answer).isEqualTo(1); assertThat(screen.toString()) @@ -152,7 +154,9 @@ void the_directory_the_document_declares_is_where_the_requests_go(@TempDir Path int answer = run("run", "pet-shelter-under-a-directory.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260918", "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260918", + "--out", + directory.toString()); assertThat(answer) .describedAs("nothing here is broken - though this says only that the run finished, " @@ -210,7 +214,8 @@ void a_second_run_replaces_a_kept_run(@TempDir Path directory) { PrintWriter out = new PrintWriter(second); Restest.run(new String[] {"run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "2", "--out", directory.toString()}, + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "2", "--out", + directory.toString()}, out, new PrintWriter(problems)); out.flush(); @@ -243,7 +248,8 @@ void a_run_leaves_other_files_alone(@TempDir Path directory) throws Exception { void a_clean_run_answers_zero(@TempDir Path directory) throws Exception { int answer = run("run", "pet-shelter.yaml", "--url", api.baseUrl() + "/shelters-only", "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", + directory.toString()); // Nothing is stubbed under that prefix, so every reply is a 404 - which no oracle in this // version complains about, because a document that does not describe 404 has not been @@ -343,7 +349,8 @@ void a_get_insisting_on_a_body_is_named_rather_than_blamed_on_the_address( @TempDir Path directory) { int answer = run("run", "pet-search-with-a-body.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", + directory.toString()); assertThat(answer).isEqualTo(3); assertThat(problems.toString()) @@ -369,7 +376,9 @@ void a_get_merely_accepting_a_body_is_sent_without_it(@TempDir Path directory) { int answer = run("run", "pet-search-with-an-optional-body.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260923", + "--out", + directory.toString()); assertThat(answer).isZero(); assertThat(screen.toString()) @@ -586,7 +595,8 @@ void an_unwritable_output_directory_answers_three(@TempDir Path parent) throws E try { int answer = run("run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", + directory.toString()); assertThat(answer) .describedAs("nowhere to write the results is one of the ways a run cannot " @@ -727,7 +737,8 @@ void a_run_that_cannot_say_it_broke_still_answers_four(@TempDir Path directory) // Somewhere to write problems that breaks when it is written to stands in for that. int answer = Restest.run(new String[] {"run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", directory.toString()}, + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--out", + directory.toString()}, new PrintWriter(screen), aScreenThatBreaks(), anEnvironmentThatBreaks(() -> { throw new StackOverflowError("a shape too deep to walk"); diff --git a/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java b/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java index 9caba365a..71405456c 100644 --- a/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java +++ b/restest-cli/src/test/java/io/restest/cli/RunLoopTest.java @@ -139,7 +139,8 @@ void the_whole_budget_is_spent() { } @Test - @DisplayName("told its size, a run stops once it has sent that many requests, having waited for their answers") + @DisplayName("told its size, a run stops once it has sent that many requests, having waited " + + "for their answers") void a_run_of_a_known_size_stops_after_that_many_requests() { engine.takes(request -> Duration.ofMillis(1)); long began = System.nanoTime(); @@ -350,7 +351,8 @@ void a_lost_step_of_a_series_is_heard_and_its_series_goes_on() { Scheduler scheduler = new Scheduler(generator, WITHOUT_A_FIRST_ROUND, Instant.now().plus(BUDGET), InstantSource.system(), events::publish); outcome = RunLoop.run(scheduler, "https://api.example", WORK_AHEAD, - ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, events); + ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, new StopFromOutside(), engine, + events); } assertThat(outcome.lost()).describedAs("list reads were sent and lost: %s", outcome) @@ -742,7 +744,8 @@ void a_loop_that_fails_announces_nothing_afterwards() throws InterruptedExceptio Scheduler scheduler = new Scheduler(generator(), WITHOUT_A_FIRST_ROUND, Instant.now().plusSeconds(30), InstantSource.system(), events::publish); org.assertj.core.api.Assertions.assertThatThrownBy(() -> RunLoop.run(scheduler, - "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, GRACE, + "https://api.example", WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, + GRACE, new StopFromOutside(), engine, events)) .isInstanceOf(IllegalStateException.class); Thread.sleep(1200); @@ -834,8 +837,8 @@ private RunLoop.Outcome runAgainst(String baseUrl, ScheduleSettings schedule) { try (EventStream events = new EventStream()) { Scheduler scheduler = new Scheduler(generator(), schedule, Instant.now().plusSeconds(30), InstantSource.system(), events::publish); - return RunLoop.run(scheduler, baseUrl, WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, PATIENT, - GRACE, new StopFromOutside(), engine, events); + return RunLoop.run(scheduler, baseUrl, WORK_AHEAD, ANNOUNCEMENTS_ALLOWED, NO_CAP, + PATIENT, GRACE, new StopFromOutside(), engine, events); } } diff --git a/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java b/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java index 0b9f3efe3..d0d4e66e9 100644 --- a/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java +++ b/restest-cli/src/test/java/io/restest/cli/WholeRunTest.java @@ -121,7 +121,9 @@ void one_command_finds_and_reports_what_is_wrong(@TempDir Path directory) throws // run is not asked to keep anything unless it is told to. int answer = run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260913", "--out", directory.toString(), "--store"); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "20260913", + "--out", + directory.toString(), "--store"); assertThat(answer).describedAs("faults were found, so the answer is 1").isEqualTo(1); @@ -175,7 +177,8 @@ void the_printed_command_reproduces_the_fault(@TempDir Path directory) throws Ex run(screen, "run", "pet-shelter.yaml", "--url", api.baseUrl(), "--budget", "30s", - "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", directory.toString()); + "--set", STOP_AFTER_REQUESTS, "--set", SIXTY_REQUESTS, "--seed", "1", "--out", + directory.toString()); String command = screen.toString().lines() .map(String::trim) diff --git a/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java b/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java index c2122a877..aab01e2c5 100644 --- a/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java +++ b/restest-core/src/main/java/io/restest/core/settings/ScheduleSettings.java @@ -39,14 +39,14 @@ * size rather than a length: told to stop after so many requests, it ends when they have been * answered, however fast or slow the machine it runs on. * - *

The ones about the opening lap are about how a run begins. Before anything is chosen by chance, a run can send - * every operation once, each with the request the API is most likely to accept: it first asks for - * the lists of what is there, then creates, then asks for single things, then changes them, and - * deletes last. Each of those steps waits for the answers to the one before, so that an identifier - * one step has just been handed is there for the next. That first round is what gets an API's - * operations answered in the first seconds of a run rather than whenever chance gets round to them. - * It can be switched off, and how long a step waits is bounded, so one request the API never - * answers cannot hold the start of a run up. + *

The ones about the opening lap are about how a run begins. Before anything is chosen by + * chance, a run can send every operation once, each with the request the API is most likely to + * accept: it first asks for the lists of what is there, then creates, then asks for single things, + * then changes them, and deletes last. Each of those steps waits for the answers to the one before, + * so that an identifier one step has just been handed is there for the next. That first round is + * what gets an API's operations answered in the first seconds of a run rather than whenever chance + * gets round to them. It can be switched off, and how long a step waits is bounded, so one request + * the API never answers cannot hold the start of a run up. * * @param workAheadFactor how many requests may be waiting for an answer at once, as a multiple of * the most the engine will ever have in flight. Above one, so the engine is never left with a diff --git a/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java b/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java index 27b148d0a..bb3654437 100644 --- a/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java +++ b/restest-core/src/test/java/io/restest/core/settings/SettingsTest.java @@ -440,11 +440,13 @@ void a_value_outside_its_range() { .hasMessageContaining("at least 1"); assertThatThrownBy(() -> new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), true, Duration.ofSeconds(-1), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1))) + Duration.ofSeconds(2), true, Duration.ofSeconds(-1), false, 1_000, + Duration.ofSeconds(5), Duration.ofSeconds(1))) .isInstanceOf(IllegalArgumentException.class) .hasMessageContaining("openingLapPatience cannot be negative"); assertThatThrownBy(() -> new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(-1), true, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1))) + Duration.ofSeconds(-1), true, Duration.ofSeconds(2), false, 1_000, + Duration.ofSeconds(5), Duration.ofSeconds(1))) .isInstanceOf(IllegalArgumentException.class) .hasMessageContaining("interruptGrace cannot be negative"); assertThatThrownBy(() -> Settings.from(Map.of("schedule.requestsBeforeStopping", "0"))) @@ -547,7 +549,8 @@ void each_group_can_be_replaced() { assertThat(from.withEngine(from.engine().withoutConcurrency()).engine() .maxConcurrency()).isEqualTo(1); assertThat(from.withSchedule(new ScheduleSettings(3, 10, Duration.ofSeconds(1), - Duration.ofSeconds(2), true, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1))) + Duration.ofSeconds(2), true, Duration.ofSeconds(2), false, 1_000, + Duration.ofSeconds(5), Duration.ofSeconds(1))) .schedule().workAheadFactor()).isEqualTo(3); assertThat(from.withGeneration(GenerationSettings.defaults()).generation()) .isEqualTo(GenerationSettings.defaults()); diff --git a/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java b/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java index 25c4ca0f7..deb378e62 100644 --- a/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java +++ b/restest-gen/src/test/java/io/restest/gen/RunOrderTest.java @@ -86,7 +86,8 @@ void the_scheduler_without_a_first_round_sends_what_was_always_sent() throws IOE ApiModel model = petClinic(); RandomTestCaseGenerator generator = withoutSeries(model); ScheduleSettings noFirstRound = new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), false, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); + Duration.ofSeconds(2), false, Duration.ofSeconds(2), false, 1_000, + Duration.ofSeconds(5), Duration.ofSeconds(1)); Instant now = Instant.parse("2026-09-23T10:00:00Z"); Scheduler scheduler = new Scheduler(generator, noFirstRound, now.plusSeconds(60), InstantSource.fixed(now), announced -> { }); diff --git a/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java b/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java index 49eeca70a..6267836cb 100644 --- a/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java +++ b/restest-gen/src/test/java/io/restest/gen/SchedulerTest.java @@ -111,7 +111,8 @@ void a_wait_ends_at_the_deadline_at_the_latest() { @DisplayName("told not to wait at all, the first round goes straight from one step to the next") void no_patience_means_no_waiting() { ScheduleSettings impatient = new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), true, Duration.ZERO, false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); + Duration.ofSeconds(2), true, Duration.ZERO, false, 1_000, Duration.ofSeconds(5), + Duration.ofSeconds(1)); Scheduler scheduler = scheduler(PETS, impatient, Duration.ofMinutes(1)); assertThat(steps(scheduler, 5)).containsExactly("send listPets (first round)", @@ -364,6 +365,7 @@ private static List steps(Scheduler scheduler, int howMany) { private static ScheduleSettings withoutAFirstRound() { return new ScheduleSettings(2, 1_000, Duration.ofSeconds(10), - Duration.ofSeconds(2), false, Duration.ofSeconds(2), false, 1_000, Duration.ofSeconds(5), Duration.ofSeconds(1)); + Duration.ofSeconds(2), false, Duration.ofSeconds(2), false, 1_000, + Duration.ofSeconds(5), Duration.ofSeconds(1)); } } From 840c04da2ccb840c67773560bd4fc141bb395b69 Mon Sep 17 00:00:00 2001 From: Sergio Segura Date: Wed, 14 Oct 2026 18:19:47 +0200 Subject: [PATCH 4/4] Say in CONTRIBUTING.md that one row of CI checks the mutation threshold Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa0f5feb8..e1d3d38d0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -82,8 +82,8 @@ you disagree with is answered in "Decisions taken", not ignored. They are in [`AGENTS.md`](AGENTS.md#hard-rules), and none is negotiable in a pull request. The build enforces some of them: the module boundaries, the evaluation platform's name, the Java 21 bytecode, -and the coverage floors of `restest-core` and `restest-oracles`. The mutation threshold is run on -demand, not in the build, and review checks everything else ([`docs/ci.md`](docs/ci.md) lists each +and the coverage floors of `restest-core` and `restest-oracles`. One row of continuous integration +also holds `restest-oracles` to its mutation threshold, and review checks everything else ([`docs/ci.md`](docs/ci.md) lists each gate). If a rule is wrong, change it deliberately, in its own pull request, with the reasoning. ## Decisions