Phase 3: recording sessions, m80 emulator through VMs core, conformance harness - #34
Merged
Merged
Conversation
… it exposed Records all ten scenarios against the real service and folds the results back into the docs and the harness. Refs #6, closes #19. The recording. Every scenario is now fixture-backed, with .meta.json sidecars carrying the status and modeled error type that the body alone loses. Nineteen recorded corrections land in docs/api-surface.md: image identifiers are ARNs and VM ids are microvm-<uuid>, four VPC_EGRESS connector members are modeled optional and enforced live, SHELL_INGRESS and HTTP_INGRESS are connector types absent from the model's enum, image delete and update are asynchronous, and UpdateMicrovmImage is a full replace that mints a new version. The harness fixes. Comparing an emulator against these fixtures showed the suite failing correct behavior three ways, all suite-side: - Short AWS resource ids and the region in the per-VM endpoint hostname went unredacted, so a recorded sg-f0e6979f could never equal a generated one. The short-id rule also absorbs the UUID and ID placeholder tails, since fixtures on disk were normalized when written and a live connector already reads nc-UUID there. - The codeArtifactUri case default did not normalize to the recorded bucket, so every target failed a value that is a pure echo of the request. - errors-not-found/get-vm-missing probed the pre-recording guess mv-… for a VM id; the gateway answered the malformed path with an nginx 502 page that got recorded as service truth. The case now uses a well-formed id and the fixture is set aside as .rejected-502 pending a re-record. Two harness gaps closed alongside. An until poll kept only the settled response, discarding the transition order a live run is uniquely able to answer; the runner now records the distinct states a poll walks through, as recorded truth rather than an assertion, since an instant-settle emulator walks a shorter path and is still conformant. And -poll-timeout caps the AWS-sized case timeouts, taking an emulator run from ten minutes of almost pure waiting to sixteen seconds with identical results. Model freshness watch. conformance/cmd/modelwatch re-derives the inventory from both vendored service-2 models and cross-checks the MicroVMs half against aws-sdk-go-v2, reporting added, removed, rerouted, or re-errored operations. A scheduled workflow files or comments on a model-drift issue. First run is clean at 29 operations, which independently confirms the hand extraction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #7. Refs #6. Scaffold (#7). cmd/m80 serves all 29 operations from the start, each answering 501 until its issue lands, because the conformance runner reads 501 as unimplemented and 404 as a wrong answer — routing everything up front means the suite runs against the scaffold today and each implemented operation flips one line of the report. It currently reads 0 fail, 0 error, 10 unimplemented. /_m80/health reports coverage against the inventory. internal/clock is the seam every state machine hangs off, so a suspend-after-fifteen-minutes policy is testable in microseconds. Its test caught a real bug: Advance jumped straight to the target before running callbacks, so a transition scheduling the next hop measured from the wrong instant and every chained state landed a full window late. Time now steps to each deadline before that timer runs, which is what time.AfterFunc does. internal/store fixes two properties that are expensive to retrofit: state is region-scoped, because an emulator sharing one namespace across regions passes tests real clients fail, and every access is serialized, because the runner and KubeMicroVM's reconcilers both issue overlapping requests. Resource types belong to #8 onward; this stops at the generic container. The route table is cross-checked against conformance/inventory.json by test, and that test immediately earned itself: TerminateMicrovm is a DELETE on the VM, not a POST to a /terminate sub-resource, and connector update is PUT not PATCH. The same mistake was live in the throttle probe's teardown path, where it would have stranded billable VMs. Second recording session. Transition order is answered: resume goes SUSPENDED to RUNNING without passing back through PENDING, while the same five-second poll did catch PENDING on the initial launch — so it is a real difference between the two paths, not a sampling artifact. A suspended VM still issues auth tokens, 200 with a full token. get-vm-missing re-recorded clean as 404 ResourceNotFoundException once the case stopped probing the malformed mv-… id, and revealed resourceId and resourceType members. Throttle probe. conformance/cmd/throttleprobe fans out concurrent RunMicrovm calls to provoke the limit QuotaGuard is built around. It is a command rather than a case because it is irreducibly imperative: concurrent fan-out, ids collected from responses it may not assert on, and teardown that must run even on panic. Scoped deliberately small, and it refuses -n above 12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #6's last recordable target. Six concurrent RunMicrovm calls against a fresh account: two admitted, four rejected. The answers, none of which the model could give: - ServiceQuotaExceededException answers HTTP 402, Payment Required. Not the 429 or 400 anyone would guess. - The binding limit is allocated memory, not VM count. Two VMs at the 2048 MiB default tier put the account's base ceiling near 4096 MiB. - quotaCode, serviceCode, resourceId and resourceType are all present and all null, so a client cannot branch on which quota it hit. It also answered a different question than the one asked. No ThrottlingException and no ThrottleReason appeared, including ConcurrentSnapshotCreateLimitExceeded, because the memory ceiling fires first and masks concurrency throttling entirely. KubeMicroVM's QuotaGuard will meet 402 long before it meets a throttle on a default account. Observing the throttle path would need an account whose memory quota is raised well above its concurrency limit, which is a support ticket rather than a recording run, so m80 implements those six reasons from the model alone. Teardown behaved: both launched VMs terminated and the image deleted, leaving only the asynchronous DELETING window that is itself already a recorded fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Serves ListManagedMicrovmImages and ListManagedMicrovmImageVersions from the 2026-07-30 recorded snapshot, embedded as data. The catalog is read-only and identical in every account, so it is not state. Both conformance steps now pass fixture-backed equality against the real recorded responses — the first operations to go from unimplemented to pass, which also proves the scaffold's register-to-fixture chain end to end. Region comes from the caller's sigv4 credential scope rather than any configured value, so one instance serves every region correctly and the store's region isolation is testable without running several. An SDK pointed at m80 with an endpoint override still signs normally, so this needs nothing from the client. Two details taken from the recording rather than invented: the account segment of a managed ARN is the literal "aws", which is what distinguishes a service-owned image from a caller-owned one, and versions come back newest first, so a client taking items[0] as current gets the right answer. Version timestamps are the live values, read back specifically because the fixtures redact them and inventing them would have been indistinguishable in the suite but wrong in the data. api.WriteError puts the modeled error type in X-Amzn-Errortype and not in the body. The live service sends no __type in these bodies and the fixtures confirm it; an emulator adding one diverges on every error case. Has() is the hook CreateMicrovmImage validates baseImageArn against. That validation, and #9's rejection acceptance criterion with it, lands in #8 which owns the operation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All thirteen images-lifecycle conformance steps pass fixture-backed equality
against the live recording, along with every create-image step the VM, token,
tag and conflict scenarios open with: 26 pass, 0 fail, 13/29 operations
implemented.
The three state layers stay separate because they are not redundant. An image
tracks whether the named resource is settled, a version tracks one build
lineage, and a build tracks one attempt against one chipset generation — which
is why a version carries two builds, Graviton 4 and 3. KubeMicroVM's image
build logs feature reads the build layer, so collapsing them would break it.
Each hop is its own timer, so a poller observes PENDING, IN_PROGRESS and
SUCCESSFUL rather than a jump, and the image settles one hop after its version
so CREATED never appears before the build is usable.
Response shapes come from the fixtures, not the model, because the model does
not describe them. Create returns the full resource with every unset member
present and null; Get returns a much smaller projection; List is smaller
again. Create sends tags as null and Get sends {} on the same untagged image,
and Update omits tags entirely. None of that is derivable — it is only
knowable from a recording, and sparse bodies are the single largest source of
emulator divergence.
Also recorded rather than invented: PUT is a full replace that mints a new
version and reports all three missing required members at once, a version
PATCH during UPDATING is a 409 ConflictException while a terminal-state VM
mutation is a 400 ValidationException, delete is refused mid-build and refused
over running VMs, and an image name stays reserved through the asynchronous
delete window.
One more suite bug surfaced. versionStateTimeBucket packs a state and the UTC
hour it was reached into one string, "SUCCESSFUL#26073006". No emulator can
match a recorded one — the step passed or failed depending on whether the run
happened in the same hour of the same day, which is why one scenario's copy
passed while another's failed. Normalization now keeps the state half, which
is real signal, and flattens the clock half.
VMs are #10. The delete-over-running-VMs rule is an interface so images does
not import them and a nil checker means none exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
43 pass, 0 fail, 22/29 operations exercised, 20/29 implemented. VMs (#10). Run, get, list and terminate, with PENDING settling to RUNNING and terminate walking through TERMINATING on the clock. The recording never sampled TERMINATING at a five-second poll, but it is in the enum and a faster client can see it, so m80 goes through it rather than jumping. Recorded rather than derived: ids are microvm-<uuid>, every VM carries managed default connectors in both directions including HTTP_INGRESS which is absent from the model's enum, terminate answers 200 with an empty object rather than the VM, a cleanly terminated VM reports stateReason "Success." with the trailing period, and mutating a terminated VM is a 400 ValidationException — neither modeled conflict type. Images and vms now ask each other exactly one question through interfaces rather than importing each other: images refuses to delete while a VM runs, and vms refuses to run an image with nothing built, so a VM that could never start is a miss up front. Region templating. Cases spell a base image ARN once as ${region} and it follows whatever region the run signs for; region is a built-in param and scenario params may reference each other. Pinning us-east-1 made every case silently wrong elsewhere, since a correct implementation scopes its managed-image catalog by region. The suite now returns an identical 43/0 against us-east-1 and eu-west-1, which was impossible before. Three more fixtures set aside, and they share a cause worth naming: they recorded the account, not the service. Three not-found probes against an invented image ARN came back 403 AccessDeniedException because the recording account's IAM is resource-scoped — the bodies name user/alex and use a capital-M Message, both tells they came from the IAM layer, and the cases' own expect blocks say ResourceNotFoundException, so the recordings contradict their own intent. ListMicrovms returns every VM the account has ever run, terminated ones included, so a recorded list is a photograph of one account at one moment. conformance/README.md now documents the category. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ListMicrovms returns every VM an account has ever run, terminated ones included and apparently forever, so a recorded list can never equal a fresh target's. Dropping the step to a bare status check would have thrown away the part that matters — whether the target returns the members a client reads. expect.itemShape checks membership without checking values. exact rejects members outside the required set as well as missing ones, since sparse bodies and over-full ones are both real divergences and an emulator returning half the members of a summary is the commonest of all. minItems guards against a target that returns an empty list and satisfies every member check vacuously. vms-lifecycle/list now passes on shape where its fixture was unusable: 43 pass, 0 fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…count Three not-found probes recorded 403 AccessDeniedException and were set aside as an artifact of the recording account's permissions. That was wrong. The account holds AdministratorAccess, and re-probing the same operation against a same-account ARN returns a plain 404. The real cause was missingImageArn carrying the 123456789012 placeholder, which makes the probe cross-account. Cross-account access needs a resource-based policy no matter how much admin the caller holds, so the 403 was correct AWS behavior for the ARN actually sent — just not the behavior the case meant to test, as its own expect block said ResourceNotFoundException all along. account is now a built-in param alongside region, defaulting to m80's own 000000000000 so emulator runs need no flag. All three re-recorded against live AWS as 404 ResourceNotFoundException, and errors-not-found has no rejected fixtures left. Two fidelity gaps in m80 surfaced from the corrected fixtures. RunMicrovm against a missing image reports the missing *version* — "No active version found for MicroVM image <arn>" — not the missing image, even when the image itself is what is absent. And the version endpoints answer in terms of the version regardless of which half was really missing, so a missing image on GetMicrovmImageVersion returns the version-flavoured message rather than delegating to the image one. 43 pass, 0 fail, 22/29 operations exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mber Fidelity is not uniformly valuable, and scoring every member equally overstated floci's problems by a third. conformance/tiers.json classifies members and -tier load-bearing scores against the ones a consumer's control flow actually depends on. The cosmetic set was derived from the corpus rather than from opinion: a member null in every recorded response is a candidate, and a test re-derives that set from the fixtures so a re-recording cannot introduce an unclassified member behind a passing suite. Four candidates were promoted back by hand and the reasons written down — latestFailedImageVersion is null only because no recorded build failed, and a reconciler reads it to detect one. Every entry carries an argument; a test enforces that too. Scoring floci both ways: 34 diverging steps at tier=all, 27 at tier=load-bearing. Seven were pure decoration — always-null config knobs, the leaked versionStateTimeBucket index key, __type on error bodies, message wording. The 27 that remain are real, and they concentrate in two response builders rather than spreading across the module. A fifth normalization bug fell out of the tiered diff. UUID redaction ran after account redaction, so a generated id whose final twelve characters are all decimal had its tail eaten and no longer matched the uuid pattern — roughly one id in a couple of million, and precisely the flake nobody would ever reproduce deliberately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the 34-at-tier-all versus 27-at-load-bearing score, where the 27 concentrate, and what each cluster costs. The data already exists on the domain model and is never written to the response, so this is serialization work rather than state work — roughly 150 lines across four private response builders plus eight missing model fields. Also records why the sparse bodies are the part that matters: the operator runs its own drift detection by reading back what it wrote, so a service that does not echo the spec looks permanently drifted and the reconcile loop never converges. One correction worth keeping. The third acceptance gate was nearly written as #31's "chant MicrovmApp deploys against floci". Nothing here depends on chant — m80's go.mod is aws-sdk-go-v2 alone, the floci module carries no chant reference, and the CFN integration test builds its template inline. chant is a consumer; CloudFormation is the interface between them. Using the chant end-to-end would have added an npm install, a pinned lexicon version and a live chant-side bug to the gate for a change that only rewrites JSON. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… fix
The fixture recorded the message as
MicroVM not found: microvm-00000000-0000-0000-0000-ACCOUNT
which is the mangling the account rule produces when it runs before the
UUID rule and eats a uuid's final twelve hex characters. That ordering
was fixed in b0c92dc, but this fixture was recorded in a963201 and never
re-normalized, so it kept asserting the broken form and the case failed
against a correct emulator.
The case's id is hardcoded all-zeros, so its final group is decimal every
time rather than the roughly one generated id in a few hundred the
normalizer comment describes — which is why this was the only fixture
affected. The other 123 recorded before the fix carry no such mangling.
This file is what #33 was missing. The handoff reported 43 pass, 0 fail,
7 unimplemented, 21 skipped and 22 of 29 operations exercised; the branch
as pushed measures 40 pass, 1 fail, 6 unimplemented, 24 skipped and 21 of
29. Restoring the correction reproduces the handoff's figures exactly, so
it was measured against a working tree holding this edit and the file
never got committed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6T4MeDN1RBiWaNtud6x77
lex00
force-pushed
the
feature/6-recording
branch
from
July 31, 2026 04:06
0f243d5 to
a29b4eb
Compare
This was referenced Jul 31, 2026
main gained the mkdocs site and the strict docs gate (#27) and the floci architecture findings (#28) after this branch was cut, and the two sides had both reformatted 60-connectors.json. The conflict is a genuine union rather than a pick. main added the subset:floci tag, correcting which cases the floci subset covers. This branch added the live-recorded request members the connector create and update actually need — AssociatedComputeResourceTypes, NetworkProtocol, ClientToken and OperatorRole, with the operatorRoleArn param behind them. Both are kept; dropping either would lose a real finding. This also unblocks CI on the PR. A conflicting pull request has no mergeable ref for pull_request workflows to run against, which is why #34 showed no checks at all while #35 and #36 were green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6T4MeDN1RBiWaNtud6x77
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 through #10, plus the conformance-harness work the emulator flushed out. Ten commits, plus one restoring a fixture correction that #33 measured against but never committed — without it the branch reads 40 pass, 1 fail. The harness fixes and the emulator co-evolved, so they land together rather than as a stack.
Against a fresh m80: 43 pass, 0 fail, 7 unimplemented, 21 skipped. 22/29 operations exercised, 20/29 implemented.
go build,go vet,gofmtandgo test -raceclean.m80 is stateful and image names stay reserved through the async delete window, so a second suite run against the same instance fails on
already exists. Restart between runs.What landed
docs/api-surface.md, transition order captured, throttle probe runcmd/m80, router over all 29 operations,internal/clock,internal/store,/_m80/healthconformance/cmd/modelwatchplus a weekly workflow; first run clean at 29 operationsReview notes
Fidelity is tiered.
conformance/tiers.jsonclassifies members by whether a consumer reads them; m80 is held to-tier all, floci is gated on-tier load-bearing. A test re-derives the always-null set from the fixture corpus, so a fixture introducing a new always-null member fails the build until it is classified with a written reason.Five normalization bugs came out of pointing a real implementation at fixtures that had only ever been recorded, never compared against: short AWS resource ids, regions outside ARNs,
versionStateTimeBucketpacking a UTC hour, UUID-versus-account ordering, and case params carrying an account placeholder. All fixed and regression-tested.regionandaccountare built-in params. A pinned region breaks against a region-scoped catalog; a pinned account makes probes cross-account, which live AWS answers 403 rather than 404.Two rejected fixtures are correct as-is and documented in
conformance/README.md:get-vm-missing.*.rejected-502andvms-lifecycle/list.json.rejected-account-history.Closes #6
Closes #8
Closes #9
Closes #10
Closes #19