A Kubernetes operator that rolls out new model versions behind a model-quality gate. It sends a new version a small slice of live traffic, compares its predictions with the current version on the same traffic, and promotes or rolls back on its own. It checks prediction quality as well as latency and errors.
A model can return HTTP 200 in 40 ms and still be wrong. ModelGate catches flipped decisions, class-mix shifts, score-distribution changes, NaN outputs, and slow or failing servers within minutes, using live traffic and no ground-truth labels.
| Area | State |
|---|---|
ModelRollout CRD (v1alpha1), CEL validation (first step at most 50%) |
done |
| Controller: shadow → weighted steps → promote/rollback, crash-safe status, kill switch, leader election | done; unit, envtest, and live on kind |
Gateway API traffic: HTTPRoute weights + RequestMirror for shadow |
done |
| Capture sidecar (Go reverse proxy, fail-open, sampled, JSONPath extraction) | done |
| Analyzer: t-digest sketches, 9 checks, alpha-spending sequential bounds | done |
| Analyzer as Argo Rollouts web metric / Flagger webhook | done |
| Transport: NATS JetStream (replay on analyzer restart) or direct HTTP | done |
kubectl modelgate plugin (status/pause/resume/abort/promote --force) with identity |
done |
| Helm chart, Dockerfiles, CI | done; installed and exercised on kind with Envoy Gateway |
| Demo model with the PRD's six bad variants | done |
Namespace defaults (ModelGatePolicy) enforced at runtime + validating webhook |
done; envtest and live on kind |
| PostgreSQL audit log (verdicts, disagreement samples, every decision and actor) | done; unit, real Postgres, and live on kind |
| Grafana dashboard (Helm ConfigMap for the Grafana sidecar) | done |
Delayed-label accuracy (labelAccuracy check, /v1/outcomes API) |
done; unit, benchmark, live on kind |
| Opt-in, time-limited payload capture with redaction in the sidecar | done; unit and controller tests |
| Replay of recorded requests for low-traffic models | done; unit, envtest, and kind e2e with no live traffic |
| KServe v2 / Triton: REST and gRPC tensors, GRPCRoute | done; unit, real gRPC client and server, local smoke, kind e2e |
| Bad-model benchmark (50 runs per variant), sidecar load test | done |
| kind e2e in CI (NATS, Postgres, Envoy Gateway), Chaos Mesh run | done; see hack/e2e/run.sh |
| Release: multi-arch images, SBOMs, cosign signatures, vulnerability scans | done; release workflow, not yet run on a tag |
┌──────────── HTTPRoute (weights + RequestMirror) ────────────┐
client ──► Gateway │
├──► <name>-stable Service ──► [capture sidecar → model vN] │
└──► <name>-canary Service ──► [capture sidecar → model vN+1] │
│ prediction events (no raw payloads)
▼
NATS JetStream / HTTP ──► analyzer (sketches, checks)
▲ verdict
controller ────────┘ └─► HTTPRoute, Deployments, status, events
- Changing
spec.templatecreates revision<name>-<hash>next to the stable one. - Shadow: the gateway mirrors a share of requests to the canary. Users only see stable responses, and the analyzer pairs both responses by
X-Request-Id. The gateway sets that ID and copies it onto the mirrored request; Envoy Gateway replaces client-supplied IDs with its own. - Steps: the canary serves 5% → 25% → 50% of live traffic. Each step compares canary and stable over the same time window.
- When every stage passes, the canary becomes stable and the old revision is deleted. A failure moves all traffic back to stable, deletes the canary, and records why.
- No statistical check is judged until
minSamplesis reached for both versions. Missing data holds a rollout and never promotes it. - Hard checks (
validity,errorRate) roll back immediately, even beforeminSamples, as soon as the evidence is conclusive. - A check fails only when its confidence bound crosses the threshold, not just the point estimate.
- Every bound spends a false-alarm budget
alphaacross looks: look k in a stage getsalpha/(k(k+1)). That sums toalphahowever often the controller polls. This answers the PRD's open question: the union bound gives a one-line false-alarm proof, and the Chernoff (binomial KL) bounds stay tight for rates near 0 or 1. - A step passes when its pause has elapsed and the latest verdict passes. The rollout fails after
failureLimitconsecutive failed verdicts. - If the analyzer is unreachable, the kill switch is on, or
spec.pausedis set, the rollout holds at its current weight. - Looks are spaced by
analysis.interval, so extra reconciles from watch events never count as additional failures. - Each stage's window starts at the first sketch bucket after the stage began, so no traffic from the previous stage leaks in.
p99Ratiois skipped in shadow unless itsphaseis set toshadow. Mirrored traffic is fire-and-forget (open-loop) while user traffic is closed-loop, so near capacity the canary queues more and looks slower even when the code is identical. On kind this produced a 2x p99 gap between identical models.- Capture sampling (
capture.sampleRate) hashes the request ID, so both sidecars record the same requests and shadow pairs survive sampling.
| Type | Output | Fails when |
|---|---|---|
agreementRate (shadow) |
classification | upper bound of paired agreement < min |
meanAbsDiff (shadow) |
regression, score | lower bound of mean |canary − stable| > max |
psi |
all (classes, or score deciles) | PSI > max and chi-square rejects equality |
chiSquare |
all | chi-square rejects and Cramér's V > max |
ks |
score, regression | KS − both DKW radii > max |
wasserstein |
score, regression | W₁ > max and KS rejects equality |
validity (hard) |
all | any invalid output when max: 0, else lower bound > max |
errorRate (hard) |
all | lower bound of canary 5xx rate > max |
p99Ratio |
all | canary p99 (lower band) / stable p99 (upper band) > max |
labelAccuracy |
classification | lower bound of (stable accuracy − canary accuracy) on delayed ground-truth labels > max (default 0.02) |
When true outcomes arrive later (a chargeback, a confirmed diagnosis), post them to the analyzer and add a labelAccuracy check:
curl -X POST http://modelgate-analyzer.modelgate-system.svc/v1/outcomes -d \
'[{"rollout": "demo/demo-model", "requestId": "4f1c...", "label": "benign"}]'- { name: accuracy, type: labelAccuracy, max: 0.02, minSamples: 200 }- Labels are joined to the captured predictions by request ID, so the client's
X-Request-Idmust be the ID it later reports the outcome for. Predictions wait up to an hour for their label. Gateways often replace that header: Envoy Gateway does unless aClientTrafficPolicysetsheaders.requestID: Preserve, as config/samples/demo.yaml does. - Each label counts in the time bucket its prediction was served in. The check pools labels over the whole rollout, because the labels for one stage usually arrive after that stage has ended.
minSampleson the check counts labeled predictions per version, 200 by default. Until there are enough, the check is inconclusive and the rollout holds at its current weight rather than promoting. A namespace policy'sminSamplesfloor does not apply to label checks.
pip install -r demo/model/requirements.txt
hack/quickstart.sh # cluster, images, ModelGate, audit DB, Prometheus, Grafana, demo traffic
hack/demo.sh bad swapped # ship a broken model and watch it roll back
hack/demo.sh good # ship a retrained model and watch it promote
hack/demo.sh audit # the audit trail
hack/quickstart.sh down # stop the background traffic; "delete" removes the clusterThe script builds and installs everything from source and starts 50 req/s of demo traffic, plus ground-truth labels 20 s after each request. It then prints the Grafana URL (http://localhost:3000/d/modelgate) and the next commands. The demo rollout uses short stages (shadow 60 s, then 20% and 50% for 45 s each), so a good model promotes in about three minutes. kubectl modelgate status demo-model -n demo -w follows a rollout live.
Runs the real analyzer and sidecar binaries in front of two demo model servers, with mirroring simulated:
pip install -r demo/model/requirements.txt
make smoke # or: go build -o bin/ ./cmd/... && python hack/local_smoke.py --variant swappedVariants: good identical swapped scaling stale accuracy-drop nan slow.
make kind-up # kind + Envoy Gateway + GatewayClass "eg"
make kind-load # build and load images
make deploy # helm install into modelgate-system
make demo # Gateway, HTTPRoute, ModelRollout in namespace demo
# start a rollout with a broken model:
kubectl -n demo patch modelrollout demo-model --type=json \
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/env/0/value","value":"swapped"}]'
kubectl modelgate status demo-model -n demoHelm does not upgrade CRDs in
crds/. After changing the CRD runkubectl apply --server-side -f config/crd/; it is too large for client-side apply. Shadow sampling uses the Gateway APIRequestMirror.percentfield, which is in the experimental channel. With standard-channel CRDs, mirroring falls back to 100%.
Platform teams set a floor once per namespace with a ModelGatePolicy named default (config/samples/policy.yaml). Teams may tighten it, never loosen it.
- The controller is the guard. Every reconcile merges the policy into the rollout in memory, taking the stricter threshold for each check type and phase. Required checks the rollout doesn't declare are added as
policy-<name>. The policy also sets a floor onminSamplesand capsfailureLimit,alphaand the first step's weight. The stored spec is never rewritten. ThePolicyAppliedcondition says what was raised. If the policy can't be read, the rollout doesn't proceed. - The webhook gives feedback. It rejects a create or spec update that is looser than the policy and names every offending field. Metadata-only updates always pass, so
kubectl modelgate abortkeeps working on rollouts that predate a stricter policy. It fails open (failurePolicy: Ignore), because runtime enforcement covers safety. The Helm chart generates its TLS certificate at install time and reuses it on upgrade; cert-manager is not needed.
kubectl modelgate pause demo-model -n demo
kubectl modelgate resume demo-model -n demo
kubectl modelgate abort demo-model -n demo
kubectl modelgate promote demo-model -n demo --force
kubectl modelgate payloads demo-model -n demo # see Payload captureThe plugin looks up the caller's identity with a SelfSubjectReview, and the controller writes it into the rollout's events. The cluster-wide kill switch is freeze: "true" in ConfigMap modelgate-system/modelgate-config.
Model servers that speak the Open Inference Protocol (KServe v2), such as Triton, KServe, MLServer and TorchServe, need no JSONPath. Name the output tensors instead:
spec:
routeRef: { name: fraud, kind: GRPCRoute } # HTTPRoute (default) for REST
output:
type: classification
protocol: kserveV2gRPC # or kserveV2 for REST, json (default) for anything else
labelTensor: { name: probabilities, reduce: argmax } # label is the class index
scoreTensor: { name: probabilities, reduce: max } # score is the top probability- Reading tensors. Only the first row (the first entry along the batch dimension) is read.
reduce: first(default) takes its first element,argmaxthe index of its largest element, andmaxthe largest element. A label tensor can also beBYTES, like Triton's classification extension. Over gRPC, both typedcontentsandraw_output_contentsare decoded, for every datatype including FP16 and BF16. NaN values count as invalid outputs. - Which calls count. With
kserveV2, only POSTs to.../inferare recorded, and withkserveV2gRPC, only unaryModelInfercalls. Health, metadata and other calls pass through untouched. A gRPC error status is recorded as its HTTP equivalent, soINVALID_ARGUMENTis a client error andINTERNALorUNAVAILABLEcounts towarderrorRate. - gRPC traffic. The sidecar accepts HTTP/2 without TLS (h2c) and speaks it to the model server. The Services get
appProtocol: kubernetes.io/h2c, and the controller steers aGRPCRoutethe same way it steers an HTTPRoute: backend weights for steps,RequestMirrorfor shadow. Shadow mode needs a gateway that supportsRequestMirroron GRPCRoutes. Envoy Gateway only resolves a GRPCRoute mirror target that the route also lists as a backend, so the canary Service is always listed, at weight 0 when it gets no live traffic. config/samples/demo-grpc.yaml runs the demo model as a Triton-style gRPC server. - Limits. Streaming gRPC calls pass through but are not analyzed. Payload capture keeps only JSON bodies, so it records no gRPC bodies. Replay sends JSON over HTTP, so it works with
kserveV2REST but not withkserveV2gRPC; the CRD rejects that combination.
By default the sidecar sends only the extracted prediction, latency and status, never request or response bodies. When a disagreement needs a closer look, turn on payload capture for a limited time:
spec:
capture:
payloads:
until: "2026-11-07T18:00:00Z" # at most 24 hours ahead
sampleRate: 0.01 # share of requests, default 0.01
maxBytes: 8192 # largest body kept, default 8192
redact: ["$.instances[*].ssn", "$.patient.*", "$['card number']"]kubectl modelgate payloads demo-model -n demo # the 10 newest
kubectl modelgate payloads demo-model -n demo --request-id 4f1c # stable and canary for one request- Redaction happens in the sidecar, before anything leaves the pod: each path's values become
"[REDACTED]". Paths support.key,['key'],[n],[*]and.*. Only JSON bodies are captured, because only a parsed body can be redacted reliably. Bodies that are not JSON, are truncated, or are larger thanmaxBytesare left out and the record says why. Headers are never captured. - It ends by itself. The sidecar stops at
until, so capture ends on time even if the controller is down. The webhook rejects anuntilmore than 24 hours ahead; if the webhook is bypassed, the controller keeps capture off and says so in thePayloadCapturecondition. Expiry does not restart pods. Turning capture on or off, or changing its settings, rolls the model pods, like any other capture setting. - Stable and canary keep the same requests. Payload sampling uses the request ID, like event sampling, so you can compare both responses to one request. Each sidecar keeps at most 20 payloads per second.
- Where it goes: payloads travel with the events to the analyzer, which keeps the last 500 per rollout for an hour in memory and serves them at
GET /v1/payloads?rollout=ns/name[&requestId=...]. With NATS, events, including payloads, are also stored in the JetStream stream for an hour. The plugin reads them through the API server's service proxy, so it needsservices/proxyonmodelgate-analyzerinmodelgate-system. The analyzer API itself has no authentication; restrict it with a NetworkPolicy if other workloads share the cluster. - Namespace policy:
spec.payloadCapture.disabled: truein theModelGatePolicyforbids capture in the namespace, andspec.payloadCapture.redactadds paths every rollout must redact.
A model that serves a few requests a minute may never reach minSamples on live traffic, so its rollouts hold forever. Replay sends recorded requests to both versions while a canary is under analysis:
spec:
strategy:
shadow: { duration: 10m }
replay:
configMapRef: { name: fraud-replay, key: requests.jsonl } # key defaults to requests.jsonl
path: /predict # where requests are POSTed, default /
ratePerSecond: 20 # recorded requests per second, default 5, at most 200The ConfigMap holds one JSON request body per line. To record real traffic, turn on payload capture for a while and export the captured requests:
kubectl modelgate payloads fraud -n ml --limit 500 --requests > requests.jsonl
kubectl -n ml create configmap fraud-replay --from-file=requests.jsonl- Same request, both versions. Each recorded request goes to the stable and canary Services with the same new request ID, so the analyzer pairs the two responses like mirrored traffic, and
agreementRateandmeanAbsDiffwork with no live traffic at all. IDs are picked so the sidecars'capture.sampleRatekeeps every replayed request. - Users never see replayed responses. The controller sends them straight to the Services, not through the HTTPRoute. It cycles through the file for as long as the canary is under analysis, and stops when the rollout is paused, held by the kill switch or a missing analyzer, promoted or rolled back.
- It adds load to stable. Every replayed request also runs on the production model; size
ratePerSecondfor that. Exported requests are redacted, so a field your model needs must not be in the redact list, or the replayed predictions will differ from production. - Status: the
Replaycondition says how many requests were loaded, or why replay is off (ConfigMapNotFound,KeyNotFound,InvalidRequests).modelgate_replay_requests_total{target,outcome}on the controller counts what was sent. The controller needs network access to the model Services; allow it if a NetworkPolicy restricts them.
Set audit.existingSecret in the Helm chart to a Secret holding a PostgreSQL DSN (key dsn), and the controller records every decision in table modelgate_audit, which it creates on first use. A demo database for kind is in config/samples/postgres.yaml.
- What is recorded: first deploys, rollout starts, every verdict with its checks and up to five disagreement samples (
verdictjsonb), step advances, promotions, rollbacks, holds, pauses, resumes, aborts and forced promotions. Each row carries the actor:system:modelgate-controllerfor automatic decisions, the caller's Kubernetes identity for plugin actions. - Ordering: entries are written before the status change they describe, so nothing takes effect without a record. Writes are idempotent, so a retried reconcile does not record a decision twice.
- When the database is down: step advances, promotions, new rollouts and forced promotions wait, and the rollout shows
Pausedwithaudit log unavailable. Rollbacks and aborts still happen. Their entries are dropped and anAuditWriteFailedevent is emitted, because an audit outage must never keep a bad canary serving traffic.
SELECT recorded_at, action, actor, canary_revision, canary_weight, reason
FROM modelgate_audit WHERE namespace = 'demo' AND rollout = 'demo-model'
ORDER BY recorded_at DESC LIMIT 20;Without a DSN, decisions are still recorded as Kubernetes events and in status.
The chart installs ConfigMap modelgate-dashboard with label grafana_dashboard: "1", which the Grafana sidecar in kube-prometheus-stack loads. It shows each rollout's phase, canary weight, verdicts, check values, promotions, rollbacks, reconcile errors, analyzer ingest, and sidecar capture outcomes. The JSON is in charts/modelgate/dashboards/modelgate.json for manual import; turn the ConfigMap off with grafana.dashboard.enabled=false.
Pushing a tag vX.Y.Z runs .github/workflows/release.yaml. It publishes multi-arch (amd64, arm64) images for the controller, analyzer and sidecar to GHCR, scans them with Trivy, and attaches an SPDX SBOM to each one as a signed attestation. Images and the OCI Helm chart are signed with keyless cosign. Plugin binaries, CRDs and checksums go to the GitHub release, with the checksums signed. To verify an image:
cosign verify ghcr.io/<owner>/modelgate-controller:<version> \
--certificate-identity-regexp 'https://github.com/<owner>/<repo>/.github/workflows/release.yaml@refs/tags/v.*' \
--certificate-oidc-issuer https://token.actions.githubusercontent.comChanges are listed in CHANGELOG.md.
- Argo Rollouts: see config/samples/argo-analysistemplate.yaml.
GET /v1/argo/verdictreturns{"decision": "pass|fail|inconclusive", ...}, and an inconclusive verdict pauses instead of promoting. - Flagger: point a webhook at
POST /v1/flagger/check. It takes the same keys inmetadata(stable,canary,window,minSamples,checks) and returns 200 for pass, 412 for fail, and 425 for inconclusive. - Compact check syntax:
name:type[:min=x|max=y][:phase=shadow|canary], comma-separated.
Model pods still need the capture sidecar, with MODELGATE_REVISION set to the pod-template hash.
api/v1alpha1/ ModelRollout types (deepcopy + CRD generated by controller-gen)
internal/stats/ t-digest, KS/Wasserstein, Chernoff/DKW/quantile bounds, chi-square, PSI
internal/analysis/ time-bucketed per-revision sketches, shadow pairing, check engine
internal/analyzer/ HTTP API, Argo/Flagger adapters, NATS consumer, metrics
internal/capture/ sidecar proxy, JSONPath subset, batching publisher (HTTP/NATS)
internal/controller/ reconciler, Deployment/Service/HTTPRoute rendering, metrics
internal/policy/ ModelGatePolicy merge (teams may only tighten)
internal/webhook/ validating webhook for ModelRollouts
internal/audit/ audit log interface and PostgreSQL store
cmd/ manager, analyzer, sidecar, kubectl-modelgate
charts/modelgate/ Helm chart, including the Grafana dashboard
demo/model/ scikit-learn demo model over JSON, KServe v2 REST and gRPC; bad variants, load generator
hack/ kind setup, e2e and chaos suite, local smoke test, benchmark, sidecar load test
make test # vet + unit tests
make envtest # controller against a real kube-apiserver + etcd
make smoke # real analyzer + sidecars + demo models, no cluster
make bench # bad-model suite, 50 runs per variant
make loadtest # sidecar overhead at 1,000 req/s
make e2e # kind: NATS, Postgres, Envoy Gateway, real rollouts under load
make e2e-chaos # the same while Chaos Mesh kills the controller, analyzer and NATS
make e2e-replay # no live traffic: replayed requests alone must promote and roll back
make vulncheck # govulncheck on the module- Unit: sketch accuracy and bound coverage under repeated looks. 50 simulated identical-model rollouts stay under 2% false rollbacks, and each bad variant must fail on the expected check. Controller logic runs against a fake client.
- envtest: the webhook behind a real API server. A loose rollout is rejected with a reason, a compliant one is accepted, annotation-only updates pass after the policy tightens, and a second policy name is refused.
- envtest: the generated CRD schema, CEL rules, defaults, pod-template metadata, the status subresource, and the experimental mirror field are all enforced by a real API server. It also drives a full rollout: deploy, shadow, three steps, promote, then a bad canary rolled back.
- Local smoke: all 8 demo variants through the real binaries.
goodandidenticalpass; the 6 bad variants fail.--protocol v2and--protocol grpcrun the same over KServe v2 REST and gRPC; CI runs both. - Audit: controller tests check the recorded sequence for a full rollout, the actors for pause, resume and abort, and fail-safe behavior with the database down (holds, never promotes, still rolls back, a forced promote waits and is recorded once).
internal/auditruns against a real PostgreSQL whenMODELGATE_TEST_POSTGRES_DSNis set; CI runs it with a Postgres service container. - kind e2e (CI):
hack/e2e/run.shinstalls everything with NATS JetStream and the audit log, sends load through Envoy Gateway, and runs rollouts of demo variants. Good variants must promote and bad ones must roll back; a bad variant ever becoming stable fails the run at once. It then checks the audit log against the outcomes. The CI chaos job runs the same suite while Chaos Mesh kills a controller replica every 40 s, the analyzer every 90 s, and NATS every 150 s.CHAOS=kubectldoes the same kills without Chaos Mesh. The CI replay job sends no live traffic at all, soidenticalmust promote andswappedmust roll back on replayed requests alone; it reads the model through KServe v2 REST tensors. The CI grpc job runs the demo model as a KServe v2 gRPC server behind a GRPCRoute. - kind + Envoy Gateway: tested by hand.
- A good rollout promotes through every stage.
swappedandnanroll back in shadow with zero user exposure;slowrolls back at 5%.- Analyzer down for 80 s: the rollout holds, then resumes.
- Leader controller killed mid-step: the standby takes over and finishes.
- Namespace policy: the webhook rejects a looser spec, the runtime condition lists what was raised, and a 3% first-step cap reaches the HTTPRoute (97/3).
- The kill switch freezes rollouts; the plugin's pause, resume, abort and promote guard all work, and an abort records
kubernetes-admin.
make bench: real scikit-learn variant predictions, the real analysis engine, PRD rollout config (shadow 10m at 20%, then 5%/10m, 25%/15m, 50%/15m, minSamples 2000). Simulated time, 50 req/s, 50 runs each.
| Variant | Expected | Result | Median time to decision | Median user exposure | Caught by |
|---|---|---|---|---|---|
| identical | promote | 0/50 false rollbacks | 53.5 min (full rollout) | n/a | - |
| swapped labels | rollback | 50/50 | 4.0 min | 0% (shadow) | agreement |
| scaling bug | rollback | 50/50 | 4.0 min | 0% (shadow) | agreement |
| stale data | rollback | 50/50 | 4.0 min | 0% (shadow) | agreement |
| accuracy drop (0.959 to 0.901) | rollback | 50/50 | 4.0 min | 0% (shadow) | agreement |
| NaN outputs | rollback | 50/50 | 0.5 min | 0% (shadow) | validity |
| 2x latency | rollback | 50/50 | 24.0 min | 2.9% | latency at the 5% step |
The latency regression misses the 15-minute target at low traffic. It can only be judged on split traffic, and the 5% step needs 2000 canary samples: 17.5 min at 100 req/s, 14.0 min at 200 req/s. The accuracy-drop variant is caught because it disagrees with the good model on about 7% of requests. A regression that keeps decisions this close to the good model's would still need delayed labels. The identical variant has exactly equal predictions, so its false-alarm rate tests sampling noise in split traffic; the unit tests also cover noisy near-identical models.
The hardest case in the PRD: what if the label-free checks were blind to the accuracy drop? go run ./hack/benchmark -variants identical,accuracy-drop -label-delay <d> -label-rate 0.5 -labels-only removes every label-free quality check (only validity, latency and errors remain) and adds labelAccuracy (max drop 0.02). Half of the requests get their true diagnosis after the delay. The rollout config and 50 runs per variant are as above.
| Label delay | accuracy-drop caught | Median time to rollback | Median exposure | identical: false rollbacks | identical: median time to promote |
|---|---|---|---|---|---|
| 1 min | 50/50 | 6.5 min | 0% (shadow) | 0/50 | 53.5 min |
| 10 min | 50/50 | 15.0 min | 1.3% | 0/50 | 54.5 min |
| 30 min | 50/50 | 35.0 min | 0.56% | 0/50 | 74.5 min |
Ground truth alone catches the 5.8-point drop every time, roughly the label delay plus five minutes later. The cost is time: with 30-minute labels, a good model waits about 20 extra minutes in its last stage for enough labels before it is promoted.
make loadtest: open-loop at 1,000 req/s for 30 s, 5 ms model, capture on every request. The sidecar adds 0.13 ms at p99 (5.85 to 5.98 ms) and drops no events. Measured on one Windows dev machine, client and server on the same host, so treat it as indicative, not per-pod.
- The canary runs the same replica count as stable.
- Only the current template can be rendered. If someone deletes the old stable Deployment mid-rollout, the rollout stalls instead of recreating it.
- The analyzer keeps sketches in memory. Run it with NATS so a restart can replay the last hour.
- A model whose decisions stay very close to the good model's, but are worse, gets past the label-free checks. The
accuracy-dropvariant is caught without labels only because it disagrees on about 7% of requests. For closer regressions, uselabelAccuracy; how long it takes depends on how quickly labels arrive (see the benchmark). - Delayed labels are supported for classification outputs only.
Licensed under Apache 2.0.