infra: Phase B — node capacity migration CAX11→CAX21 (40→80 GB) + C.1 kubelet tuning

[matthias476]

Split out of the hardening tracker #131 (architecture plan Phase B + Task C.1). These are the last two items of the post-outage hardening plan; everything else is done. Phase B is held pending a deliberate go-ahead (it costs money + the control-plane step has a short downtime window). C.1 is sequenced after Phase B because it only becomes safe on the bigger nodes.

Phase B — why

• RAM headroom is tight on the 4 GB CAX11s (workers 73–83% used; CP ~70% of physical). Runs today, but no headroom for node-failure rescheduling or growth (squeeze is headroom, not current load — no OOMs).
• Disk: 40 GB roots recurringly fill (containerd images + MicroOS snapshots) — the original outage trigger. 80 GB roots give real margin.
• Target: all 4 nodes CAX21 (8 GB RAM, 80 GB disk). Cost: +€16.68/mo total (cax11 €5.34 → cax21 €9.51 each ×4).

Mechanism (pre-verified)

server_type is not ForceNew in the hcloud provider → changing it is an in-place resize (power off → resize → power on). The node keeps its identity, its attached Cloud Volumes (Postgres PVs), and its Terraform address — no destroy/recreate. Migrate one node at a time via the pool's nodes map (preserves resource addresses; splitting the count=3 pool into three pools would churn addresses — do NOT). Hazards: node is down during its resize (drain first), and Hetzner only grows the disk when resized without "keep disk" → verify the disk actually grew to ~76 GB.

Phase B steps

B.1 — Pre-flight

• Throwaway terraform plan after preparing the nodes map shows ~ update in-place (or hcloud_server modify), no hcloud_volume changes, no destroy/replace. If it shows replacement → stop, fix the nodes-map keys.
• Take an etcd snapshot (rollback for B.3): k3s etcd-snapshot save --name pre-cax21-<ts> on the CP; record the name.

B.2 — Migrate the 3 workers (one at a time, own PR each)

• Convert the worker-nbg1 pool to an explicit nodes = { "0"={} "1"={} "2"={} } map (no server_type change yet) → PR alone first → plan must be no-op (proves the map maps onto the existing 3 nodes).
• Per node "0"/"1"/"2": kubectl drain → set nodes."<k>".server_type="cax21" → PR → read plan: exactly ONE in-place hcloud_server update, no volume change, no replace → merge → verify node rejoins + df -h / shows ~76 GB (not ~38) → kubectl uncordon. CNPG stays 2/2 throughout.

B.3 — Migrate the control plane (downtime window)

• Decision recorded: Option 1 (accept ~5–10 min API/ArgoCD/kubectl downtime). Workloads keep running (kubelet caches state); only the API server is down while the single CP resizes. etcd snapshot from B.1 is the rollback. (Option 2 = go HA first; not chosen.)
• During the window: set the CP server_type="cax21" → PR → verify in-place plan (no etcd volume change / no replace) → merge → watch resize.
• After CP returns: kubectl get --raw=/readyz = ok; CP Ready with ~76 GB; both CNPG clusters healthy. (If broken: k3s server --cluster-reset --cluster-reset-restore-path=<snapshot>.)

Phase B done when: all 4 nodes CAX21 with ~76 GB disks (growth verified, not assumed); CNPG stayed healthy; etcd snapshot retained.

---

C.1 — Kubelet eviction + reservation tuning (do AFTER Phase B)

Why after B: reservations carve RAM away from pod-allocatable. On 4 GB nodes that risks OOMKills/unschedulable pods; on 8 GB (post-B) there's headroom.

Why it matters: the outage trigger was a full disk → kubelet abruptly mass-evicting pods → CSI unregister → cascade. C.1 makes the node degrade gracefully: reserve headroom for system/containerd, and evict a few low-priority pods early & softly (with the disk alert firing) before the hard wall triggers the cascade.

File: infrastructure/cluster/main.tf — add to the module "kube-hetzner" block (per Codex finding #3, use k3s_global_kubelet_args, which covers CP and agents; NOT k3s_exec_agent_args):

k3s_global_kubelet_args = [
  "kube-reserved=cpu=100m,memory=300Mi,ephemeral-storage=2Gi",
  "system-reserved=cpu=200m,memory=350Mi,ephemeral-storage=4Gi",
  "eviction-hard=memory.available<150Mi,nodefs.available<10%",
  "eviction-soft=nodefs.available<15%",
  "eviction-soft-grace-period=nodefs.available=2m",
  "eviction-minimum-reclaim=nodefs.available=2Gi",
]

• ⚠️ Last-wins gotcha: kubelet applies repeated --kubelet-arg last-wins, and global args append after the module's per-nodepool default kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi. A partial global value (e.g. ephemeral-only) would silently drop the cpu/memory reservation → always specify the full cpu+memory+ephemeral triple on both reserved flags.
• PR → read plan → confirm it reconfigures kubelet with no node recreation → merge.
• Verify on a node: both kube-reserved (with memory) and system-reserved args present; kubectl get node ... -o jsonpath='{.status.allocatable.memory}' reflects the reservation (not full 8 GiB).
• Watch ~24 h for unexpected OOMKills (reservation reduces pod-allocatable RAM).

C.1 done when: kubelet args applied on all nodes (verified), no node recreation, no OOM regressions.

---

Refs: tracker #131 · architecture plan docs/superpowers/plans/2026-05-28-cluster-architecture.md (Phase B = B.1–B.3, Task C.1).