Skip to content

multi-node-deploy.md

Latest commit

 

History

History
234 lines (169 loc) · 10.8 KB

multi-node-deploy.md

File metadata and controls

234 lines (169 loc) · 10.8 KB
Raw

Multi-Node Cluster Deployment

This guide explains how to expand a single-node Cube Sandbox deployment into a multi-node cluster by adding compute nodes. Compute nodes run only the sandbox runtime components (Cubelet with the embedded network runtime, CubeShim) and register themselves to the control plane on the first machine.

:::: warning Production Use If you plan to use Cube Sandbox in a production environment, please refer to the Network Hardening guide to secure your deployment before exposing services to untrusted networks. ::::

:::: tip Prerequisite You must have a working control node deployed via the Self-Build Deployment Guide before adding compute nodes. ::::

Architecture Overview

┌─────────────────────────────────────────┐
│           Control Node                  │
│  CubeMaster, CubeOps, cube-api,         │
│  CubeProxy, CoreDNS, MySQL, Redis,      │
│  Cubelet (network runtime)              │
└──────────────────┬──────────────────────┘
                   │  /internal/v1/node-agent API
       ┌───────────┼───────────┐
       ▼           ▼           ▼
┌────────────┐┌────────────┐┌────────────┐
│ Compute #1 ││ Compute #2 ││ Compute #N │
│ Cubelet    ││ Cubelet    ││ Cubelet    │
│ net runtime││ net runtime││ net runtime│
└────────────┘└────────────┘└────────────┘
  • The control node runs the full stack: orchestration (CubeMaster), node management (CubeOps), API gateway (cube-api), proxy (CubeProxy + CoreDNS), databases (MySQL + Redis), the bundled MinIO S3 volume store, and also acts as a compute node itself.
  • Each compute node runs only Cubelet with its embedded network runtime. It registers to the control-plane CubeOps and receives sandbox scheduling requests from CubeMaster.

Prerequisites

Each compute node must meet the same hardware and software requirements as the control node:

  • Physical machine or bare-metal server (nested virtualization is not supported)
  • x86_64 or aarch64 (ARM64) architecture with KVM enabled (ls /dev/kvm)
  • Docker installed and running
  • Network connectivity to the control node (specifically to CubeOps on port 3010 for node registration, and to the S3 endpoint — port 9000 with the bundled MinIO)

For the full requirements list, see Self-Build Deployment — Prerequisites.

Step 1: Prepare the Release Bundle

Use the same release bundle that was built for the control node. Copy it to the compute node and extract:

tar -xzf cube-sandbox-one-click-<version>.tar.gz
cd cube-sandbox-one-click-<version>

Step 2: Configure Environment Variables

cp env.example .env

Edit .env and set the following variables:

ONE_CLICK_DEPLOY_ROLE=compute
CUBE_SANDBOX_NODE_IP=<current-node-ip>
ONE_CLICK_CONTROL_PLANE_IP=<control-plane-ip>

# CUBE_S3_* is required — the volume plugin hard-depends on S3. Compute nodes
# never deploy MinIO, so copy these values from the control node's .one-click.env
# (they must match exactly so the plugin resolves the same store). With the
# bundled MinIO the block looks like:
CUBE_S3_ENDPOINT=http://<control-plane-ip>:9000
CUBE_S3_ACCESS_KEY_ID=<from control .one-click.env>
CUBE_S3_SECRET_ACCESS_KEY=<from control .one-click.env>
CUBE_S3_BUCKET=cube-volumes
CUBE_S3_S3FS_EXTRA_OPTS=-ouse_path_request_style
Variable Description
ONE_CLICK_DEPLOY_ROLE Must be set to compute for compute-only nodes
CUBE_SANDBOX_NODE_IP This node's primary network interface IP
ONE_CLICK_CONTROL_PLANE_IP The control node's IP; automatically expanded to <ip>:3010 for CubeOps node registration
CUBE_S3_* Required. The volume plugin hard-depends on S3. Compute nodes never deploy MinIO, so copy these from the control node's .one-click.env (or point them at your own S3-compatible store). With the bundled MinIO, also allow TCP 9000 from compute to the control node.

::: tip Enforced at install time install-compute.sh validates CUBE_S3_ENDPOINT before touching any local configuration and aborts if it is missing, instructing you to copy CUBE_S3_* from the control node. Set these variables before running the installer. :::

You can also specify the CubeOps endpoint explicitly if it uses a non-default port:

ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR=<control-plane-ip>:3010

ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR takes precedence over ONE_CLICK_CONTROL_PLANE_IP when both are set.

Step 3: Install

sudo ./install-compute.sh

The compute-node install script will:

  1. Install only Cubelet with the embedded network runtime, cube-shim, cube-image, cube-kernel-scf, and the runtime scripts
  2. Start only the host process cubelet
  3. Automatically point Cubelet's meta_server_endpoint to the control-plane CubeOps
  4. Register the node and report status through the control plane /internal/v1/node-agent API to CubeOps

Verifying the Deployment

Health Check

sudo ./smoke.sh

In compute-node mode, quickcheck.sh verifies:

  • Local Cubelet and embedded network runtime health
  • Reachability of the control-plane CubeOps
  • That the current node appears under /internal/v1/nodes/{node_id} on the control plane

Verify from the Control Node

On the control node, you can confirm the compute node has registered:

curl http://127.0.0.1:3010/internal/v1/nodes

The response should include the compute node's IP and a healthy status.

Configure CubeMaster Scheduler Scoring

For multi-node deployments, configure CubeMaster's scheduler.score on the control node. If scoring is omitted, CubeMaster filters eligible nodes and then selects from the filtered node order, which can concentrate new sandboxes on the first eligible node until resource filters push traffic elsewhere.

Merge the following scheduler fields into the existing scheduler section of cubemaster.yaml. Keep your existing filter, timeout, overcommit, and other scheduler settings.

scheduler:
  # Keep your existing filter, timeout, overcommit, and other scheduler settings.
  priority_select_num: 3
  score:
    enable_scorers:
      - real_time_weighted_average
    resource_weights:
      mvm_num: 2
      local_create_num: 3
      quota_cpu_usage: 1
      quota_mem_usage: 1
    plugin_conf:
      real_time_weighted_average:
        weight: 1.0
        enable_weight_factors:
          - mvm_num
          - local_create_num
          - quota_cpu_usage
          - quota_mem_usage

For multi-node clusters, set scheduler.priority_select_num to a value greater than 1 so CubeMaster randomly selects from the top scored nodes. The shipped default config uses priority_select_num: 1, which means scoring only determines which single node receives the next sandbox. Use 3 as a starting point for small clusters and tune it based on your node count. scheduler.least_select_name defaults to random, so it usually does not need to be set explicitly.

For the complete CubeMaster scheduler reference, including Cubelet node reports, quota / label / concurrency effects, and template redo after adding compute nodes, see CubeMaster Scheduler Configuration.

After updating cubemaster.yaml, restart CubeMaster with your normal deployment procedure so the scheduler loads the new scoring configuration.

Common Operations

Stop Compute Node Services

sudo ./down.sh

In compute-node mode, this only stops cubelet. It does not affect the control plane or other compute nodes.

Reinstall

To reinstall a compute node, simply run install-compute.sh again. The script automatically stops the existing deployment before installing.

View Logs

Component Log Path
Cubelet /data/log/Cubelet/
CubeShim /data/log/CubeShim/
Hypervisor (VMM) /data/log/CubeVmm/
Runtime PID files /var/run/cube-sandbox-one-click/
Process stdout/stderr /var/log/cube-sandbox-one-click/

For control-node log paths, see Self-Build Deployment — View Logs.

Configuration Reference

Compute nodes use the same .env file format. The following variables are specific to or particularly relevant for compute-node deployments:

Variable Default Description
ONE_CLICK_DEPLOY_ROLE control Must be set to compute
ONE_CLICK_CONTROL_PLANE_IP empty Control-plane host IP; expanded to <ip>:3010 by default
ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR empty Explicit CubeOps address; takes precedence over ONE_CLICK_CONTROL_PLANE_IP
CUBE_SANDBOX_NODE_IP 10.0.0.10 Required. This node's primary network interface IP
CUBE_SANDBOX_NETWORK_CIDR 192.168.0.0/18 (from config.toml) cubevs local network CIDR. Should match the control-plane value. IPv4 CIDR format (e.g., 10.100.0.0/18), mask range /16–/24. Auto-detected for host network conflicts at install time.
CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK 0 Set to 1 to skip CIDR conflict detection (not recommended).
ONE_CLICK_RUN_QUICKCHECK 1 Run health check after installation
CUBE_S3_* empty / filled by control MinIO Required. Volume plugin is an S3 hard dependency. Copy from the control node's .one-click.env; ENDPOINT / ACCESS_KEY_ID / SECRET_ACCESS_KEY / BUCKET have no usable default.

For the full configuration reference (build-time options, database, proxy, etc.), see Self-Build Deployment — Configuration Reference.

Troubleshooting

Compute Node Cannot Reach CubeOps

Verify network connectivity:

curl http://<control-plane-ip>:3010/internal/v1/nodes

If this fails, check:

  • Firewall rules on the control node (port 3010 must be accessible)
  • The ONE_CLICK_CONTROL_PLANE_IP or ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR value in .env

Node Not Appearing in Control Plane

If smoke.sh passes locally but the node does not appear on the control plane:

  1. Check Cubelet logs: /data/log/Cubelet/
  2. Verify meta_server_endpoint in the Cubelet config points to the correct CubeOps address
  3. Ensure CUBE_SANDBOX_NODE_IP is correctly set to a routable IP (not 127.0.0.1)

For general troubleshooting (Docker, KVM, DNS, etc.), see Self-Build Deployment — Troubleshooting.