feat: the failure-injection levers get an HTTP surface

README.md

@@ -12,6 +12,8 @@ Then apply a `MicroVMImage` and a `MicroVM` and watch the operator drive them.
 
 One emulator, 8 MiB, next to the real operator on a real cluster. No AWS account, no credentials, no bill. The [guide](docs/kubemicrovm.md) has a worked example that goes from nothing to a running MicroVM.
 
+One flag matters for that path and is easy to miss: the operator's startup gate calls `sts:GetCallerIdentity` before it will report ready, with no endpoint override of its own, so pointing `AWS_ENDPOINT_URL_STS` at an m80 started with `-serve-sts` is what lets it boot at all. It is a shim for that one action, not an STS emulation — everything else under STS answers 501. Filed upstream as [KubeMicroVM#50](https://github.com/codriverlabs/KubeMicroVM/issues/50); when the override lands the shim can go.
+
 ## Does it actually catch anything?
 
 Running KubeMicroVM's own 63-case UAT suite against m80 surfaced three issues in the operator, none of which needed an AWS account to find.
@@ -101,6 +103,21 @@ Transitions run on an injected clock, so `-build-delay` decides how long a build
 docker run --rm -p 4290:4290 ghcr.io/intentius/m80 -build-delay 300ms
 ```
 
+Failure paths are what a consumer most needs a test target for, and they are the ones real AWS will not produce on request. `-enable-injection` exposes the levers to any client:
+
+```sh
+docker run --rm -p 4290:4290 ghcr.io/intentius/m80 -enable-injection
+
+# the next build of this image settles FAILED
+curl -X POST localhost:4290/_m80/inject -d '{"target":"build","name":"doomed"}'
+
+# the next connector of this name settles FAILED with a real reason code
+curl -X POST localhost:4290/_m80/inject \
+     -d '{"target":"connector","name":"egress","reasonCode":"SubnetOutOfIPAddresses"}'
+```
+
+Off by default: nothing under `/_m80/` is signed, so the flag is the consent. The response carries `"injected": true`, which a state m80 reached on its own never does — so a test cannot mistake an injected failure for a real one.
+
 Everything is in memory and nothing is written, so a restart is a clean account. m80 is stateful within a run — image names stay reserved through the async delete window, exactly as the real service does — so a suite that runs twice against one instance will fail the second time on `already exists`. That is fidelity, not a bug; restart between runs.
 
 m80 follows the pattern of [mudflaps](https://github.com/intentius/mudflaps) (Fly Machines) and [spritzer](https://github.com/intentius/spritzer) (Fly Sprites). A single static Go binary and distroless container that holds MicroVM images, VMs, tokens, and network connectors in memory, advances them through their lifecycle on an injected clock, and answers the real wire protocol so any SDK client works against it via endpoint override.

cmd/m80/main.go

@@ -22,6 +22,7 @@ import (
 	"github.com/intentius/m80/internal/clock"
 	"github.com/intentius/m80/internal/connectors"
 	"github.com/intentius/m80/internal/images"
+	"github.com/intentius/m80/internal/inject"
 	"github.com/intentius/m80/internal/limits"
 	"github.com/intentius/m80/internal/managedimages"
 	"github.com/intentius/m80/internal/store"
@@ -56,6 +57,11 @@ func main() {
 	throttleReason := flag.String("throttle-reason", "",
 		"ThrottleReason on throttles for the connector and tags families; the MicroVM model has no Reason member")
 	retryAfter := flag.Int("throttle-retry-after-seconds", 0, "Retry-After on throttles; 0 omits it")
+	// Failure injection is destructive by design and nothing under /_m80/ is
+	// signed, so anything that can reach the port can arm a lever. Off unless
+	// asked for, same posture as -serve-sts.
+	enableInjection := flag.Bool("enable-injection", false,
+		"expose the failure-injection levers at POST /_m80/inject, so a consumer pointed at the container can provoke a failed build or a connector failure code")
 	serveSTS := flag.Bool("serve-sts", false,
 		"answer sts:GetCallerIdentity, for consumers whose startup gate calls it before they will report ready; not an STS emulation, every other action gets 501")
 	flag.Parse()
@@ -112,6 +118,12 @@ func main() {
 	connectorSvc := connectors.NewService(clk, st, *buildDelay)
 	connectors.Register(srv, connectorSvc)
 	tags.Register(srv, imageSvc, vmSvc, connectorSvc)
+	if *enableInjection {
+		inject.Register(srv, &inject.Service{Images: imageSvc, Connectors: connectorSvc})
+		log.Warn("failure injection enabled", "path", inject.Path)
+	} else {
+		inject.RegisterDisabled(srv)
+	}
 	tokenSvc := tokens.NewService(clk)
 	tokens.Register(srv, tokenSvc, vmSvc)
 	// A VM's endpoint is a different host answered by the same process, so it

docs/api-surface.md

@@ -178,6 +178,6 @@ Two situations remain unrecorded because nothing reaches them. A `PENDING` VM an
 
 ## Health and introspection
 
-`/_m80/health` reports implemented operations against the model inventory, the mudflaps convention, plus the regions the store has been asked about. `/_m80/vm/{microvmId}/` reaches a VM's endpoint stub without forging a `Host` header.
+`/_m80/health` reports implemented operations against the model inventory, the mudflaps convention, plus the regions the store has been asked about. `/_m80/vm/{microvmId}/` reaches a VM's endpoint stub without forging a `Host` header. `POST /_m80/inject` arms a failure — a build that settles `FAILED`, or a connector that settles `FAILED` with one of the seven reason codes — and answers 404 naming the flag unless m80 was started with `-enable-injection`. See [Lifecycle](lifecycle.md#drift-levers).
 
 There is no clock endpoint. Transitions run on an injected clock and Go tests advance it directly, but nothing exposes that over HTTP, so a black-box client cannot skip a delay — it waits, or it starts m80 with a shorter `-build-delay`. An earlier draft of this page described a `/_m80/clock` hook that was never built.

docs/lifecycle.md

@@ -65,8 +65,26 @@ Transient states settle on the injected clock with short deterministic delays, t
 
 KubeMicroVM's drift detection and auto-suspend features watch for the service changing state underneath the CRs, so m80 can fail things the real service would only fail by bad luck. This is m80's version of mudflaps' failure injection, a feature the real service will never offer a test suite.
 
-Two levers exist today, and both are Go APIs rather than endpoints: `images.Service.FailNextBuild` forces the next build of a named image to `FAILED`, and `connectors.Service.FailNext` settles the next connector of a named connector into `FAILED` carrying any of the seven reason codes. Neither can be provoked against real AWS on demand — you cannot ask EC2 to run a subnet out of addresses — which is the whole reason they exist.
+Two levers: `images.Service.FailNextBuild` forces the next build of a named image to `FAILED`, and `connectors.Service.FailNext` settles the next connector of a given name into `FAILED` carrying any of the seven reason codes. Neither can be provoked against real AWS on demand — you cannot ask EC2 to run a subnet out of addresses — which is the whole reason they exist.
 
-Being Go-only bounds what they are good for. A test that imports m80 can drive them; a UAT pointed at the container cannot reach them at all.
+Both were Go-only until [#56](https://github.com/INTENTIUS/m80/issues/56). A test that imported m80 could drive them; a suite pointed at the container could not reach them at all, which is exactly backwards, since a container is what a consumer tests against.
 
-That was originally thought to block the offline drift run behind [#18](https://github.com/INTENTIUS/m80/issues/18). It did not: drift is provoked with ordinary `SuspendMicrovm` and `TerminateMicrovm` calls, which any client can make, and the [KubeMicroVM harness](kubemicrovm.md) does exactly that. The gap is real for a different reason. Failure paths are what a consumer most needs a test target for, and they are the ones m80 cannot currently be asked to take, so a test wanting to see the operator handle a failed image build has no way to cause one. Tracked as [#56](https://github.com/INTENTIUS/m80/issues/56).
+They now have an HTTP surface, off unless asked for:
+
+```sh
+docker run --rm -p 4290:4290 ghcr.io/intentius/m80 -enable-injection
+
+curl -X POST localhost:4290/_m80/inject -d '{"target":"build","name":"doomed"}'
+curl -X POST localhost:4290/_m80/inject \
+     -d '{"target":"connector","name":"egress","reasonCode":"SubnetOutOfIPAddresses"}'
+```
+
+Three things about that shape are deliberate.
+
+It is keyed by **name, not ARN**. A lever arms before the resource exists — that is what "the next build of this image fails" means — so at the moment of arming there is no ARN to name it with.
+
+The response carries **`"injected": true`**. A state m80 reached on its own never carries it, so a consumer asserting on that field cannot mistake an injected `FAILED` for one the emulator produced by its own rules.
+
+It is **off by default**. Nothing under `/_m80/` is signed, so anything that can reach the port can arm a lever; a flag is the consent, the same posture `-serve-sts` takes. Without the flag the route is still registered, and answers 404 with a message naming the flag — a bare 404 is indistinguishable from a typo in the path.
+
+The gap was originally thought to block the offline drift run behind [#18](https://github.com/INTENTIUS/m80/issues/18). It did not: drift is provoked with ordinary `SuspendMicrovm` and `TerminateMicrovm` calls, which any client can make, and the [KubeMicroVM harness](kubemicrovm.md) does exactly that. It was real for a different reason — failure paths are what a consumer most needs a test target for, and they were the ones m80 could not be asked to take.

docs/scope.md

@@ -14,6 +14,7 @@ The Lambda MicroVMs control plane and the minimal data-plane edges a client can
 | Network connectors | CRUD for VPC egress connectors, subnet and security group references accepted as opaque strings, service limits enforced |
 | `sts:GetCallerIdentity` | Off unless `-serve-sts`. A shim for consumers whose startup gate calls it, not an STS emulation: every other action answers 501. See [standing up KubeMicroVM](kubemicrovm.md) |
 | Limits | The five memory tiers, name patterns and lengths, connector subnet and security-group bounds, token expiry and port grants, the recorded account memory ceiling |
+| Failure injection | Off unless `-enable-injection`. `POST /_m80/inject` arms a build that settles `FAILED`, or a connector that settles `FAILED` with one of the seven reason codes — the failures real AWS will not produce on request. See [lifecycle](lifecycle.md#drift-levers) |
 
 ## Refused, on purpose
 

internal/api/server.go

@@ -52,6 +52,14 @@ func NewServer(c clock.Clock, s *store.Store, version string) *Server {
 	return srv
 }
 
+// Handle attaches a raw handler to a mux pattern, for the non-service surface
+// under /_m80/. Operations go through Register instead — they are routed from
+// the Routes table so an unimplemented one is a deliberate 501, and nothing
+// outside that table should be able to claim an operation's path.
+func (s *Server) Handle(pattern string, h http.HandlerFunc) {
+	s.mux.HandleFunc(pattern, h)
+}
+
 // Register attaches an implementation to an operation. Registering an unknown
 // operation panics: a typo would otherwise leave the real route on 501 while
 // the handler sat unreachable.

internal/inject/inject.go

@@ -0,0 +1,147 @@
+// Package inject exposes m80's failure-injection levers over HTTP.
+//
+// The levers themselves are older than this package and live where the state
+// they corrupt lives: images.FailNextBuild and connectors.FailNext. Both were
+// reachable only from Go, so a suite that imported m80 could drive them and a
+// suite pointed at ghcr.io/intentius/m80 could not (#56). Failure paths are
+// the ones a consumer most needs a test target for — a KubeMicroVM test that
+// wants to watch its operator handle a failed image build had no way to cause
+// one.
+//
+// Off unless asked for. Nothing under /_m80/ is signed, so anything that can
+// reach the port can reach this; a lever that arms a failure is not something
+// a container should carry by default. Same posture as -serve-sts: the flag
+// is the consent.
+package inject
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"sort"
+	"strings"
+
+	"github.com/intentius/m80/internal/api"
+	"github.com/intentius/m80/internal/connectors"
+	"github.com/intentius/m80/internal/images"
+)
+
+// Path is the one route this package serves.
+const Path = "/_m80/inject"
+
+// Targets, as they appear in a request body.
+const (
+	TargetBuild     = "build"
+	TargetConnector = "connector"
+)
+
+// Service arms the levers on behalf of an HTTP caller.
+type Service struct {
+	Images     *images.Service
+	Connectors *connectors.Service
+}
+
+type request struct {
+	Target string `json:"target"`
+	// Name of the resource the lever is keyed by.
+	//
+	// Not an ARN, which is what the issue's sketch reached for: both levers
+	// arm *before* the resource exists, so at the moment of arming there is
+	// no ARN to name it with. Keying by name is what makes "the next build of
+	// this image fails" expressible at all.
+	Name string `json:"name"`
+	// Connector only. One of connectors.ReasonCodes.
+	ReasonCode string `json:"reasonCode"`
+}
+
+type response struct {
+	// Injected is always true on success, and is the field a consumer should
+	// assert on. A state m80 reached on its own never carries it, so a test
+	// cannot mistake an injected FAILED for a real one — which is the whole
+	// reason to be careful here rather than just returning 200.
+	Injected   bool   `json:"injected"`
+	Target     string `json:"target"`
+	Name       string `json:"name"`
+	ReasonCode string `json:"reasonCode,omitempty"`
+	// Armed says what will happen and when, because a lever is a promise
+	// about a future request rather than a change to anything now.
+	Armed string `json:"armed"`
+}
+
+// Register wires the route. Call it only when injection was asked for; the
+// route is registered either way (see Disabled) so a consumer that forgot the
+// flag is told so rather than getting a bare 404 off the end of the mux.
+func Register(srv *api.Server, svc *Service) {
+	srv.Handle("POST "+Path, svc.serve)
+}
+
+// RegisterDisabled wires the route to an explanation. Without this, forgetting
+// -enable-injection produces a 404 indistinguishable from a typo in the path.
+func RegisterDisabled(srv *api.Server) {
+	srv.Handle("POST "+Path, func(w http.ResponseWriter, r *http.Request) {
+		writeError(w, http.StatusNotFound,
+			"failure injection is disabled; start m80 with -enable-injection to turn it on")
+	})
+}
+
+func (s *Service) serve(w http.ResponseWriter, r *http.Request) {
+	var req request
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "body is not JSON: "+err.Error())
+		return
+	}
+	if strings.TrimSpace(req.Name) == "" {
+		writeError(w, http.StatusBadRequest, "name is required — the levers are keyed by resource name")
+		return
+	}
+
+	switch req.Target {
+	case TargetBuild:
+		if req.ReasonCode != "" {
+			writeError(w, http.StatusBadRequest,
+				"reasonCode is a connector field; a failed build carries a message, not a code")
+			return
+		}
+		s.Images.FailNextBuild(req.Name)
+		api.WriteJSON(w, http.StatusOK, response{
+			Injected: true,
+			Target:   TargetBuild,
+			Name:     req.Name,
+			Armed:    fmt.Sprintf("the next build of image %q settles FAILED", req.Name),
+		})
+
+	case TargetConnector:
+		if !connectors.ValidReasonCode(req.ReasonCode) {
+			writeError(w, http.StatusBadRequest, fmt.Sprintf(
+				"reasonCode %q is not one of the model's seven: %s",
+				req.ReasonCode, strings.Join(sortedReasonCodes(), ", ")))
+			return
+		}
+		s.Connectors.FailNext(req.Name, req.ReasonCode)
+		api.WriteJSON(w, http.StatusOK, response{
+			Injected:   true,
+			Target:     TargetConnector,
+			Name:       req.Name,
+			ReasonCode: req.ReasonCode,
+			Armed: fmt.Sprintf("the next connector named %q settles FAILED with %s",
+				req.Name, req.ReasonCode),
+		})
+
+	default:
+		writeError(w, http.StatusBadRequest, fmt.Sprintf(
+			"target %q is not one of: %s, %s", req.Target, TargetBuild, TargetConnector))
+	}
+}
+
+func sortedReasonCodes() []string {
+	out := append([]string(nil), connectors.ReasonCodes...)
+	sort.Strings(out)
+	return out
+}
+
+// writeError keeps this surface's errors plainly non-service-shaped. Nothing
+// under /_m80/ is the MicroVMs API, so an error here must not look like one a
+// client should retry or map to a model exception.
+func writeError(w http.ResponseWriter, status int, message string) {
+	api.WriteJSON(w, status, map[string]any{"message": message})
+}