feat(warehouse): component warehouse in CubeOps with on-demand node download

[fslongjin]

Background

Since CubeSandbox introduced component multi-versioning, sandbox create/restore requests pin the exact component versions from the template. Previously, if a node did not have a required version locally (typical cases: a new node joining the cluster, or an existing node missing an older version), creation failed immediately and operators had to SSH into the node to copy directories or reinstall by hand.

This PR automates that away: CubeOps acts as the cluster-level component warehouse. Operators import once; nodes download on demand when a version is missing; creation proceeds once the download succeeds, and only fails if it cannot. The overall architecture is described below, with the notable implementation trade-offs listed under "Key Implementation Decisions".

Architecture

Each component keeps a clear responsibility. CubeMaster's boundary is unchanged — it remains unaware of per-node inventory and never handles component files itself:

flowchart TB
  subgraph control [Control plane]
    UI[CubeOps console]
    Ops[CubeOps component warehouse<br/>version catalog + artifact storage]
    UI -->|import / browse / pre-install| Ops
  end

  subgraph sources [Import sources]
    Up[Admin upload]
    GH[GitHub Release]
    CNB[CNB]
    Up --> Ops
    GH --> Ops
    CNB --> Ops
  end

  subgraph node [Compute node]
    Cubelet[Cubelet] --> Inv[Local multi-version inventory]
    Cubelet -->|on-demand download when missing| Ops
  end

Loading

CubeOps: the warehouse and its import entrance

• What gets archived: the one-click installation package is unpacked into four component kinds — guest image, Agent, Shim, and kernel. Kernels follow the existing convention of a content-derived short name rather than the release tag, so the same kernel gets the same name in the warehouse and on every node.
• Three import sources: upload from the console, GitHub Release, and CNB. Remote downloads are restricted to admin-configured whitelisted repositories and domains, and every redirect hop is re-validated to prevent access to unintended internal addresses.
• Imports are asynchronous jobs: installation packages are large and parsing takes a while, so an import request just creates a job and returns; unpacking and archiving happen in a background worker. Multiple CubeOps replicas are supported — each job is claimed by exactly one replica, and a job whose owner has been unresponsive for over 30 minutes can be taken over by another replica. The console has a dedicated jobs page showing progress.
• Two API families:
  • Admin APIs sit behind login authentication and cover catalog listing, version detail, version deletion, the "in warehouse but not on node" coverage view, and pre-install to selected nodes.
  • Node-facing APIs serve Cubelet. They are separate from the admin APIs and carry no cluster credential (matching how Cubelet already calls CubeMaster's internal endpoints — security relies on network isolation). They provide artifact download, pre-install job claim/ack, and local inventory reporting.
• Artifact storage: files live in a shared directory (default /data/cubeops/warehouse), mounted as a shared persistent volume under Kubernetes. Content is written to a temporary directory first and moved into place only when complete, so a partial directory is never mistaken for an installable version; on shared storage that does not support atomic rename (e.g. NFS), it falls back to copying while preserving executable permissions.
• Re-import never overwrites: importing the same component + version again is skipped, protecting existing content; forced overwrite is deferred to a later phase.
• Unpacking safety: path-traversal protection and size caps (8 GiB per file, 16 GiB total).

Cubelet: on-demand download when a version is missing

• Trigger point: while resolving local component paths during create/restore, a missing version is first requested from CubeOps; only after it is downloaded into the local multi-version inventory does the flow continue. Without a warehouse address configured, behavior is identical to today (fail with "version missing").
• Concurrent de-duplication: when several concurrent creations on the same node miss the same version, they merge into a single download instead of racing on the same directory.
• Atomic installation: the blob is unpacked into a temporary directory, required files are validated, and only then moved into its final inventory location — an install counts only when fully written; an interrupted download never does.
• Distinguishable failures: errors clearly separate "version not in warehouse" from "download or validation failed". In neither case does Cubelet silently substitute another version.
• Inventory reporting and pre-install: the background loop periodically reports a snapshot of the local inventory to CubeOps (skipped when unchanged) and claims pending pre-install jobs for this node every 30 seconds.

CubeMaster: scheduling unchanged, compatibility evaluation corrected

Master keeps dispatching create/restore exactly as before, but the compatibility-matrix semantics had to be corrected in step: since nodes now self-heal missing versions from the warehouse, "node has not reported versions yet" or "node lacks a version" no longer implies template incompatibility. The new rule: a template with all four component versions pinned is immediately compatible; only legacy records with incomplete version information evaluate to unknown. The old per-dimension comparison logic was removed — otherwise multi-version templates would be wrongly marked stale and trigger rebuilds. Additionally, the create timeout is relaxed from 300s to 600s to accommodate the download wait.

Console

Three new pages: warehouse overview (per-component version counts, sizes, missing-node counts, active jobs), component detail (per-version source and checksum, installed/missing node lists, with pre-install from selected missing nodes and version deletion), and a jobs center (import and pre-install jobs, paginated). The import form supports all three sources and multiple architectures. Full English and Chinese UI translations are included.

Key implementation decisions

A few points discussed during design review landed as follows:

• Imports run as asynchronous jobs (job table + background workers + multi-replica claiming) so that large packages never block the request.
• Timeouts are tiered along the whole chain: node download defaults to 10 minutes (configurable), a single pre-install job to 12 minutes, CubeOps remote fetch and large-file writes to 30 minutes each, and the reverse proxy to 30 minutes.
• Error classes converge from three to two: externally we distinguish only "version not in warehouse" and "download unsuccessful"; the specific download-vs-validation failure reason is carried in the error message.
• Capabilities added along the way: deleting a warehouse version cascades to cancel its pending pre-install jobs; paginated job listings; deployment fails fast when multiple replicas are configured without shared storage; an in-place upgrade path for existing nodes (warehouse address written automatically).

Explicitly out of scope for this phase: Master rescheduling based on node inventory, nodes downloading directly from GitHub/CNB, falling back to a different version when one is missing, and fleet-wide forced pre-install.

Code size

97 files, +9,083 / −329. Breakdown by component (lines counted from the staged diff):

Component	Feature code	Test code	Deploy/config	Docs & i18n	Total
CubeOps (warehouse server)	2,761	1,503	36	63	4,363
Cubelet (on-demand download)	702	540	2	0	1,244
CubeMaster (compatibility fix)	24	63	2	0	89
web (console)	1,672	760	0	212	2,644
deploy (Kubernetes chart + one-click)	0	0	338	0	338
CubeDB (migrations)	0	0	168	0	168
docs (EN/ZH user docs)	0	0	0	236	236
Total	5,159	2,866	547	511	9,083

Notes: "test code" for web is mock data (for offline development of the three new pages); backend tests cover importing, unpacking, disk layout, catalog aggregation, handlers and the store layer (including database integration tests), plus Cubelet download, installation, concurrency merging, and inventory reporting.

Deployment & upgrade notes

• Kubernetes: CubeOps gains a 50 GiB shared persistent volume; multi-replica deployments require shared storage or deployment fails fast (intentional, with a corresponding guard test). Compute nodes pick up the warehouse address automatically via an environment variable that is written into the Cubelet config.
• One-click installs: existing nodes get the warehouse address written in place by the upgrade script — no reinstall needed.
• Database migrations add 4 tables; no destructive changes.

[cubesandboxbot]

AI-Generated Review — PR #1353

⚠️ This review was produced by an automated AI reviewer and does not represent a human approval. Please treat it as a structured second pass, not a sign-off. All findings below were checked against the PR diff and the base-branch code; severity is my judgment, not a guarantee.

Overview

This is a large, well-structured feature: CubeOps becomes a cluster-level component warehouse for the four versioned components (cube-shim, cube-image, cube-agent, cube-kernel-scf), with on-demand download by Cubelet when a pinned version is missing during create/restore. Highlights:

• Version catalog in Postgres/MySQL (dual-dialect store, olderThanDurationSQL using DB clock) + on-disk artifact layout under /data/cubeops/warehouse, plus 4 migrations.
• Async import jobs (upload / GitHub Release / CNB Release) with multi-replica claiming (stale running jobs reclaimable after 30 min).
• Careful defensive engineering in the download/install path: singleflight dedup with context.WithoutCancel, atomic temp-dir + rename install, per-component path-traversal validation, size caps on extract (8 GiB/file, 16 GiB total), content-derived kernel version names (sha256-<12>).
• Helm + one-click wiring, CUBE_OPS_ADDR injection, nginx client_max_body_size 8g, 50 Gi PVC.
• Three new WebUI pages and a clean i18n pass.

The test surface is genuinely good (store tests across both dialects, handler tests, importer tests, cubelet install/fetch tests).

Findings

1. (Medium) Compat matrix and WebUI are blind to cube-shim drift

web/src/pages/Templates.tsx:946 — replicaLiveDiffers (Templates.tsx and the identical helper in TemplateDetail.tsx) compares only guest-image/cube-agent/kernel. Under the new semantics a replica is only OK when all four components — including cube-shim — are pinned (evaluateCompat in CubeMaster/pkg/templatecenter/store.go), and shim is the most runtime-sensitive of the four. But TemplateNodeCompat has no shim bound/current fields and node-side GetNodeComponentVersions still reports only the other three. A shim-only node upgrade therefore produces no drift note and no way to see it in the UI, even though create will use the pinned shim.

2. (Medium) STALE status is now unreachable; dead code left in server and WebUI

CubeMaster/pkg/templatecenter/store.go:1174 — evaluateCompat returns only OK/UNKNOWN; CompatStatusStale can no longer be produced. GetCompatMatrix still counts stale_templates/stale_replicas/affected_nodes, and the WebUI still renders STALE KPI cards (gated on summary.staleTemplates > 0, which can never be true going forward), the compat.status.STALE/compat.kpi.staleTemplates locale keys, and a compatTone STALE case. This is dead code after the semantic change. It also deserves a release-note callout: legacy pre-multi-version replicas (guest+agent pinned but no kernel/shim) move from OK/STALE to UNKNOWN and now surface the rebuild warning even though they were previously treated as compatible.

3. (Medium) Node warehouse endpoints are unauthenticated behind a spoofable header

CubeOps/internal/handler/warehouse.go:501 — /internal/warehouse/* (blob, jobs, ack, inventory) is gated only by X-Cube-Node-ID (or node_id query) with no shared secret. Anything that can reach CubeOps :3010 — any pod via the Helm ClusterIP service, or anything on the one-click host network — can impersonate any node: overwrite another node's install-coverage records (PUT /inventory), ack another node's preinstall jobs (POST /jobs/:id/ack), and download all artifacts (GET /blob). This matches CubeMaster's existing /internal/meta header-only model, so it is a documented cluster-trust boundary rather than a regression; the nginx edge correctly does not proxy /internal/. Recommend a per-node registration secret or network policy for the mutating routes, especially in multi-tenant clusters.

4. (Medium) Upload endpoint has no server-side size cap

CubeOps/internal/handler/warehouse.go:177 — Upload does io.Copy with no byte limit. The only caps live at the edge (client_max_body_size 8g in nginx) and MaxMultipartMemory (spool threshold only). A direct request to CubeOps :3010 (no nginx in front — the normal case for one-click localhost and for cluster pods) can write an arbitrarily large file into _uploads/, and the 2-hour sweep TTL means a burst can fill the warehouse volume. Enforce the same 8 GiB cap server-side.

5. (Low) warehouseApi.upload bypasses the 401 refresh flow

web/src/api/client.ts:366 — unlike ops()/api(), upload uses a raw fetch with the token read straight from localStorage and no refresh/retry. After the access token expires (~15–30 min, with refresh rotation), uploads fail with 401 until the user reloads. Route it through ops() or add the shared refresh-on-401 logic; the manual JSON.parse also throws on non-JSON error bodies where safeJson is used elsewhere.

6. (Low) Cubelet 10-min download timeout vs CubeOps 30-min fetch budget

Cubelet/pkg/cubelet/cubelet.go:59 — default cubeops_timeout (10 min) is shorter than CubeOps warehouse.fetch_timeout (30 min) and the importer claim-stale threshold (30 min). On a cold node whose component is not yet in the warehouse, CubeOps may need up to 30 min to fetch from GitHub before streaming to the node; the cubelet client gives up at 10 min. The singleflight continues server-side, so a retry succeeds — but the first create can fail spuriously at the 10-minute mark. Consider raising the cubelet default or tying it to the 600 s create timeout.

Additional observations (not blocking)

• GetBlob mid-stream errors (CubeOps/internal/handler/warehouse.go:397): status 200 is written before WriteTarGz; a failure is only slog.Error-logged, so the client receives a truncated gzip. The cubelet's gzip/tar reader detects truncation and can retry, so impact is limited to observability — consider a HEAD/Content-Length option or an erroring trailer.
• Re-import of an already-imported tag silently keeps the old artifact: for shim/image/agent, the version key is the tag; installExtracted short-circuits when the destination dir already exists. If v0.6.0 is re-published with different content, re-importing it will not update the warehouse and there is no checksum comparison or warning. By-design immutability for kernels (content-addressed) but a footgun for tag-addressed components.
• Import claim race at the 30-min boundary: a job whose fetch saturates the full 30-min budget has no heartbeat during execute, so ListImportWork can surface it as stale and a second replica will claim it → duplicate GitHub download (both eventually mark succeeded). Low impact, but a periodic updated_at bump during long fetches would close it.
• DB-clock staleness: olderThanDurationSQL compares against NOW(); if the app and DB clocks skew, stale-claim timing drifts accordingly.
• The compat.banner/staleDesc/staleTitle locale strings were correctly updated to match the new UNKNOWN semantics — no i18n mismatch found.

Overall

A comprehensive, well-tested feature with strong defensive engineering in the artifact install path (singleflight, atomic rename, traversal guards, size caps) and thorough dual-dialect coverage. The issues above are mostly observability/robustness/security-boundary refinements rather than functional blockers. The two I'd prioritize are #1 (shim drift is invisible to operators even though shim is the component most likely to need attention) and #3 (confirm network isolation for the internal routes in every deployment mode).

[fslongjin]

Replied on each inline thread. Summary:

Fixed: crash-window catalog repair (R1), upload GC + 2h TTL (R2), cancel running preinstall on delete (R4), inventory PUT no longer fails create (R5/R8), independent ack context (R7), preinstall stale check on the DB clock (R9).

False positives / not changing: unauthenticated /internal/warehouse matching CubeMaster (R3), global 30-minute WriteTimeout (R6).

[fslongjin]

Replied on each second-round inline thread.

Fixed: skip hashing on duplicate import when dest dir and catalog row already exist; CreatePreinstallJobs now inserts the whole batch in one transaction (partial failure rolls back).

Not changing: unauthenticated /internal/warehouse (same as CubeMaster /internal/meta); compat matrix OK for fully pinned replicas; GetBlob streams after 200; lexical version order (kernel keys are not semver).

[cubesandboxbot]

Drift detection misses cube-shim. Under the new multi-version semantics a replica is OK only when guest-image, cube-agent, kernel, and cube-shim are all pinned (see evaluateCompat in CubeMaster/pkg/templatecenter/store.go), and this replicaLiveDiffers helper drives the new drift note / driftReplicas KPI. It compares only guest/agent/kernel — TemplateNodeCompat has no shim bound/current fields and node-side GetNodeComponentVersions still reports only guest-image/cube-agent/kernel. A node whose shim was upgraded (the most runtime-sensitive of the four, it must match the containerd runtime) will show no drift indicator even though create uses the pinned shim. Consider adding a shim dimension to TemplateNodeCompat and to the node-side version report. The identical pattern exists in TemplateDetail.tsx replicaLiveDiffers.

[cubesandboxbot]

evaluateCompat can now return only OK or UNKNOWN, so CompatStatusStale becomes unreachable: GetCompatMatrix still counts stale_templates/stale_replicas/affected_nodes by stored status, but no newly-written row can be STALE (historical rows get recomputed by ScanNodeCompat). The STALE branches that remain — the web KPI cards gated on matrix.summary.staleTemplates > 0, compat.status.STALE / compat.kpi.staleTemplates locale keys, the compatTone STALE case — are effectively dead code. Worth a cleanup pass, and a release-note callout that legacy pre-multi-version replicas (guest+agent pinned, no kernel/shim) move from OK/STALE to UNKNOWN and will now surface the rebuild warning even though they were previously considered compatible.

[cubesandboxbot]

Node identity for every /internal/warehouse/* endpoint is a caller-supplied header (X-Cube-Node-ID, or a node_id query param) with no shared secret. Any process that can reach CubeOps :3010 — in Helm, any pod via the ClusterIP service; in one-click, anything on the host network — can impersonate any node: PUT /internal/warehouse/inventory overwrites another node's install-coverage records, POST /internal/warehouse/jobs/:id/ack can mark another node's preinstall jobs succeeded/failed, and GET /internal/warehouse/blob can pull every artifact. This matches CubeMaster's existing /internal/meta trust model (also header-only), so it's a documented cluster-trust boundary rather than a regression — but the README note ("do not publish this prefix") only protects the nginx edge; the ClusterIP service and one-click host exposure still leave it reachable from anything on the cluster/host. Consider a per-node registration secret or network policy, at least for the mutating inventory/ack routes.

[cubesandboxbot]

io.Copy streams the uploaded .tar.gz with no byte limit. The only caps are edge-level: nginx client_max_body_size 8g (chart + one-click) and MaxMultipartMemory (which only controls when the multipart body spools to disk, not the final size). A request that reaches CubeOps :3010 directly — no nginx in front, which is how one-click deploys talk to it on localhost and how any cluster pod can reach it — can write an arbitrarily large file into _uploads/, and the 2-hour sweep TTL means a burst of large uploads can fill the warehouse volume before cleanup. Worth enforcing the same 8 GiB cap in Upload (and/or a total-quota check against the layout root) so the limit does not depend on the proxy in front of the server.

[cubesandboxbot]

Unlike every other WebUI API call, warehouseApi.upload uses a raw fetch with the access token read straight from localStorage and no 401 refresh/retry. After the access token expires (~15–30 min lifetime, with refresh-token rotation), the upload fails with 401 and the user must reload the page; ops()/api() in lib/api.ts share a single-flight refreshAccessToken() that this path bypasses. Consider routing the upload through ops() (passing the FormData as body), or adding the same refresh-and-retry on 401 here. The manual JSON.parse(text) also throws on non-JSON error bodies, whereas the shared helpers use safeJson.

[cubesandboxbot]

Default cubeops_timeout (10 min) is shorter than CubeOps's own fetch budget: warehouse.fetch_timeout defaults to 30 min and the importer claim-stale threshold is 30 min. On a cold node whose component is not yet in the warehouse, CubeOps may need to download from GitHub (up to 30 min) before it can stream to the node; the cubelet's client times out at 10 min. The server-side singleflight fetch continues (context.WithoutCancel), so a retry succeeds once the artifact lands — but the first sandbox create on a cold node can fail spuriously at exactly the 10-minute mark. Consider raising the cubelet default (or tying it to the 600 s create timeout) so the "up to 10 minutes" WebUI copy matches the server's actual budget.