Skip to content

CKS: upgradeKubernetesCluster fails on control node when binaries ISO ships headlamp.yaml instead of dashboard.yaml #14244

Description

@mw-0

problem

Since 4.23, create-kubernetes-binaries-iso.sh packages the Headlamp dashboard as headlamp.yaml
instead of dashboard.yaml. The cluster create path (k8s-control-node.yml) handles both files,
but upgrade-kubernetes.sh still hardcodes:

/opt/bin/kubectl apply -f ${BINARIES_DIR}/dashboard.yaml

The script runs with bash -e, so on the first control node it exits non-zero after kubeadm
upgrade apply and the kubelet swap have already succeeded. The worker retries
(cloud.kubernetes.cluster.upgrade.retries) and then fails the job, leaving the control node
on the new version but cordoned, the workers on the old version, the cluster in Alert,
and the DB still recording the old version.

Management server log:
SSH execution of command sudo ./upgrade-kubernetes.sh.sh 1.36.0 true false false true
has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
Failed to upgrade Kubernetes cluster : single, unable to upgrade Kubernetes node on VM : single-control-...

Running the script by hand on the control node shows:
error: the path "/mnt/k8sdisk//dashboard.yaml" does not exist

ISO contents:
10-kubeadm.conf autoscaler.yaml cni cri-tools docker headlamp.yaml k8s kubelet.service
manifest.yaml network.yaml provider.yaml snapshot-crds.yaml

versions

ACS 4.23.0.0 (also present on main at 0a5bf30)
Kubernetes upgrade 1.35.8 -> 1.36.0, ISO built with the 4.23 create-kubernetes-binaries-iso.sh
Hypervisor: kvm Network: isolated..

The steps to reproduce the bug

  1. Build and register a Kubernetes version ISO with the 4.23 create-kubernetes-binaries-iso.sh (contains headlamp.yaml)
  2. Deploy a CKS cluster on the previous minor version
  3. Call upgradeKubernetesCluster to the new version
  4. Job fails on the first control node; kubectl get nodes shows control on new version, SchedulingDisabled

What to do about it?

Make upgrade-kubernetes.sh use the same fallback as k8s-control-node.yml: apply headlamp.yaml
if present, else dashboard.yaml, else warn. I'll open a PR. Workaround: patch
script/upgrade-kubernetes.sh inside the management server jar and restart.
I already have tested a fix and will put in a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions