feat(control-plane): Expose node isolation in CubeOps/WebUI and converge multi-replica CubeMaster state near-real-time via Redis Stream - #1309
Conversation
… Stream
- Add controlevents package: publish node isolation events to a shared Redis Stream so every CubeMaster replica converges in-memory views within seconds after a DB write. MySQL remains the source of truth; nodemeta.loopReload covers missed events.
- Wire CubeMaster main.go: controlevents.Init is a fatal dependency in coreInit (Redis unreachable => CubeMaster refuses to start).
- Extend nodemeta/isolation.go: append a best-effort event publish at the tail of the DB-write function; add a symmetric ApplySchedulingDisabledLocal for consumer callbacks (mutually exclusive via lockNodeLabels).
- Register cube:v1:shared:master:control:events (Stream, shared scope, XADD MAXLEN ~ 10000) in rediskey + redis-key-spec.md.
- CubeOps: add PUT/DELETE /api/v1/nodes/{nodeID}/isolation (thin pass-through to CubeMaster REST, no request body, idempotent); expose schedulingDisabled on every nodeView; extend CubeMasterClient interface and cubemaster.Client with IsolateNode/UnisolateNode (+ PUT helper).
- Web: introduce IsolateConfirmDialog (Portal), add Isolate / Unisolate actions on Nodes list (Radix dropdown menu) and NodeDetail header, show an Isolated badge, format condition badges via new utils.ts helpers. Bilingual locales extended.
- Docs: extend node-isolation.md (4 entry points, DB+Stream mechanism, multi-replica scope, 60s wait window clarified as create-RPC pipeline window).
Signed-off-by: jinlong <jinlong@tencent.com>
![cubesandboxbot[bot]](https://avatars.githubusercontent.com/in/3653584?s=60&v=4){kind=small}
| // controlevents fans out control-plane mutations (e.g. node isolation) to | ||
| // every Cubemaster replica via Redis Stream. Redis is a hard dependency: | ||
| // without it multi-replica cordon convergence is not guaranteed. | ||
| if err := controlevents.Init(ctx, nodemeta.ApplySchedulingDisabledLocal); err != nil { |
There was a problem hiding this comment.
This makes Redis a fatal startup dependency for CubeMaster. Previously a Redis outage at boot was explicitly non-fatal (lifecycle.Init warns and continues — see the comment just above). Now a PING/TYPE failure aborts coreInit → stdlog.Fatalf, so a transient Redis blip during a CubeMaster restart bricks the whole control plane.
For a single-replica deployment the stream provides no convergence value (nothing to fan out to), yet the hard dependency still applies. Consider either:
- making
controlevents.Initnon-fatal (log a warning and degrade to DB-reload-only convergence, matching lifecycle's pattern), or - gating the fatal requirement on an explicitly multi-replica config.
At minimum this operational regression should be called out in the docs, since deploy/kubernetes/chart/files/cube-master/conf.yaml ships with Redis but a restart during a Redis maintenance window would now fail to boot.
| } | ||
| // Confirm the Redis server accepts stream commands. TYPE on a missing key | ||
| // returns "none"; any transport/command error is fatal at startup. | ||
| if _, err := pool.Do("TYPE", EventStreamKey); err != nil { |
There was a problem hiding this comment.
TYPE on a missing key returns "none" with no error, so this probe passes even when cube:v1:shared:master:control:events already exists as a different key type (e.g. a leftover String/Hash from a bad deploy or a name collision). In that case XADD/XREAD fail with WRONGTYPE at runtime, and because the publisher swallows XADD errors (publisher.go best-effort), the failure is completely silent — multi-replica convergence silently degrades to DB reload while operators believe the stream is healthy.
Suggest asserting the returned type is "none" or "stream" and failing fast otherwise, e.g.:
kind, err := redis.String(pool.Do("TYPE", EventStreamKey))
if err != nil {
return fmt.Errorf("controlevents: redis TYPE %s failed: %w", EventStreamKey, err)
}
if kind != "none" && kind != "stream" {
return fmt.Errorf("controlevents: key %s exists as type %q, want stream", EventStreamKey, kind)
}
| if err != nil { | ||
| return nil, "", err | ||
| } | ||
| _ = ctx |
There was a problem hiding this comment.
ctx is never actually used — the blocking XREAD BLOCK 5000 cannot be interrupted on shutdown, so the consumer goroutine lingers up to ~5s after ctx is cancelled and holds a pooled Redis connection blocked for the full 5s on every iteration.
With the default max_active: 32 this is fine, but on a deployment with a small pool this permanently consumes one connection (and one MaxActive slot) per replica, and shutdown is delayed by up to 5s + the 1s error-retry sleep. Consider XREAD BLOCK in a shorter loop that checks ctx.Done() between reads (e.g. block 1s and re-check), or pass a context-derived deadline.
| nodeID, disabled, changed, out.SchedulingDisabled) | ||
|
|
||
| // Fan out to all Cubemaster replicas via Redis Stream (best-effort). | ||
| controlevents.PublishNodeIsolationDefault(ctx, nodeID, disabled) |
There was a problem hiding this comment.
The event is published even when changed == false (idempotent re-PUT/DELETE on a node already in the target state). Every replica then receives and re-applies a no-op, generating redundant stream traffic and log noise. Consider publishing only when changed:
if changed {
controlevents.PublishNodeIsolationDefault(ctx, nodeID, disabled)
}Minor, but this also keeps the stream free of duplicate entries that the 10k MAXLEN cap would otherwise have to trim.
Review: PR #1309 — Node isolation in CubeOps/WebUI + Redis Stream multi-replica convergenceAI-generated review — no human approval implied. OverallA well-structured feature. The No clear correctness blockers. The findings below are robustness/design concerns, roughly ordered by severity. Findings1. Redis becomes a fatal startup dependency (Medium)
2. Startup
|
3 participants
feat(control-plane): Expose node isolation in CubeOps/WebUI and converge multi-replica CubeMaster state near-real-time via Redis Stream
TL;DR
cubemastercli+ CubeMaster HTTP (port 8089)PUT/DELETE /api/v1/nodes/{id}/isolation· CubeMaster HTTP ·cubemasterclinodeViewcarries an extraschedulingDisabled: booleannodemetaperiodic DB reload (typical 60s wait window)cube:v1:shared:master:control:eventsbroadcasts events → every replica's in-memory view converges within seconds; DB reload is demoted to a safety netDELETEalso supportedType: 🚀 Feature (new functionality)
Components touched: CubeMaster control plane · CubeOps backend · Web frontend · bilingual docs
🎯 Background & Motivation
Node isolation (scheduling cordon / uncordon) is a routine operations action:
Previously this action was only available via
cubemasterclior CubeMaster HTTP, which made the ops mental load heavier. Worse, in multi-CubeMaster-replica deployments, the scheduling view between a new replica or different replicas was kept in sync only bynodemeta.loopReload(a periodic refresh, by default every tens of seconds) — meaning operators had to wait a whole window before another replica "saw" an isolation take effect.This PR closes both gaps in one go:
🏗️ Architecture at a Glance
flowchart TB subgraph Clients["Client Entry Points"] WebUI["WebUI (Nodes / NodeDetail)"] CubeOps["CubeOps REST"] CMHTTP["CubeMaster HTTP :8089"] CLI["cubemastercli"] end subgraph CM["CubeMaster Replica (any peer)"] direction TB OpsH["CubeOps handler writeIsolation"] Client["cubemaster.Client PUT/DELETE /internal/meta/nodes/{id}/isolation"] MetaSvc["meta.writeIsolation"] SetNode["nodemeta.SetNodeSchedulingDisabled"] LocalCache["local snapshot + syncLocalcache"] Pub["controlevents.Publisher.PublishNodeIsolation (best-effort)"] subgraph Peers["Other CubeMaster Replicas"] direction TB ReplA["consumer.Run — Replica A"] ReplB["consumer.Run — Replica B"] ReplC["consumer.Run — Replica C"] end Apply["nodemeta.ApplySchedulingDisabledLocal"] end DB[("MySQL node_meta (source of truth)")] Stream[("Redis Stream cube:v1:shared:master:control:events")] WebUI --> OpsH CubeOps --> OpsH CMHTTP --> MetaSvc CLI --> MetaSvc OpsH --> Client Client --> MetaSvc MetaSvc --> SetNode SetNode --> DB SetNode --> LocalCache SetNode -.best-effort.-> Pub DB -.loopReload safety net.-> SetNode Pub -->|XADD MAXLEN ~10000| Stream Stream -.XREAD $ BLOCK 5s broadcast.-> ReplA Stream -.XREAD $ BLOCK 5s broadcast.-> ReplB Stream -.XREAD $ BLOCK 5s broadcast.-> ReplC ReplA --> Apply ReplB --> Apply ReplC --> Apply Apply --> LocalCache classDef db fill:#fde,stroke:#c33,color:#000 classDef stream fill:#def,stroke:#37c,color:#000 classDef peer fill:#efe,stroke:#393,color:#000 classDef critical fill:#fee,stroke:#e33,color:#000 class DB db class Stream stream class ReplA,ReplB,ReplC peer class Pub critical📦 Changes
A. CubeMaster control plane
A new
controleventspackage (publisher / consumer / schema / apply / init + tests) is responsible for broadcasting "node isolation" events to every CubeMaster replica via a Redis Stream: once the local replica finishes its DB write it publishes best-effort; other replicas consume viaXREAD $ BLOCKand callback intonodemetato mutate their local view and localcache; missed events are covered by DB reload. At startup the package issues PING / TYPE against Redis — any failure is fatal.nodemeta/isolation.goappends a best-effort event publish at the tail of the existing DB-write function; a symmetric "local apply" function (no DB write) is added for consumer callbacks; the two paths are mutually exclusive vialockNodeLabels.main.goinsertscontrolevents.Initinto thecoreInitorder, afternodemetaand beforescheduler, as a fatal dependency — Redis unavailable ⇒ CubeMaster refuses to start.rediskeygains aMasterControlEvents()constructor, yieldingcube:v1:shared:master:control:events(Stream,sharedscope,XADD MAXLEN ~ 10000).B. CubeOps backend
Two new REST endpoints,
PUT/DELETE /api/v1/nodes/{nodeID}/isolation, are added. The handler is a thin pass-through to CubeMaster's existing isolate / unisolate REST; the response is the updated fullnodeView, now including aschedulingDisabledfield. ThenodeViewstruct, theCubeMasterClientinterface, andcubemaster.Clientall gain the corresponding methods (including a PUT helper), andCubeOps/README.mdgets two extra lines. Both endpoints have no request body and are idempotent.On the test side,
cluster_test.gohas its list assertion rewritten and three new cases (isolate / unisolate / 404) added;fake_cm_*get their stubs. CubeOps has no direct coupling to the newcontroleventspackage — it merely wraps the existing CubeMaster REST into two client methods; stream broadcasting happens entirely inside CubeMaster.C. Web frontend
A generic
IsolateConfirmDialog(Portal, reused by the list page and the detail page) is introduced, exposingopen/onClose/onConfirm/pending/errorprops and an error-formatting helper.The list page
Nodes.tsxadds a ⋯ menu at the top-right of every node card: when not isolated, the menu shows a destructive "Isolate" item (opens the global dialog); when isolated, it shows an "Unisolate" item (submits directly); the card title displays an "Isolated" badge driven byschedulingDisabled; the bottom conditions use the unified readable text and colour scheme.The detail page
NodeDetail.tsxadds an isolate / unisolate button to the Header before the heartbeat time. The icon switches with state (ShieldOff/ShieldCheck); isolate goes through the confirm dialog, unisolate submits directly; the title carries the same "Isolated" badge; on success, a toast fires and both the current-node and the list caches are invalidated; errors are rendered as a red banner at the top of the page.The API client gains
isolate/unisolate; the schema gains theschedulingDisabledfield;utils.tsgains formatting and colour helpers for condition badges (reused by the list and the detail-pageConditionRow). Bilingual locales are extended with isolation-related copy (12 keys in the detail page + 1 key in the list page).D. Docs
The bilingual
node-isolation.mdextends the entry-point list from 2 to 4 (WebUI · CubeOps · CubeMaster HTTP ·cubemastercli), adds a "DB + Stream" mechanism section and a "Multi-replica CubeMaster" scope note, and clarifies the 60s wait window as the in-flight create-RPC pipeline window rather than a cache delay.The bilingual
redis-key-spec.mdregisters the new Stream key, its field contract, and theMAXLEN ~ 10000TTL policy. EN and ZH are fully aligned — no single-side updates.