Skip to content

CKS: preserve node offerings during nodeids scale-in - #14459

Open
andrijapanicsb wants to merge 2 commits into
mainfrom
cks-nodeids-scale-in-preserve-offerings
Open

andrijapanicsb wants to merge 2 commits into
mainfrom
cks-nodeids-scale-in-preserve-offerings

Conversation

@andrijapanicsb

@andrijapanicsb andrijapanicsb commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #14458. Fixes #14552. Found during QA for #14223; this is a separate PR based on main.

Fix

A nodeids-only scale-in could fall back to the old global offering and try to resize unrelated nodes before removing the selected worker.

Omitted offering parameters now preserve existing offerings. Count-only accounting uses the node-type offering, and autoscaling-only requests still work without resizing VMs. The normal drain/remove workflow is unchanged.

Testing

138 CKS unit tests passed, including regression tests for selected-worker removal, mixed/legacy offerings and autoscaling-only requests.

Live verification used the #14223 baseline with this fix applied. A request with only the cluster ID and nodeids removed the exact additional worker:

Node Before After
Original control 3 CPU / 3072 MiB Unchanged, same offering
Original worker 3 CPU / 4096 MiB Unchanged, same offering
Selected additional worker 3 CPU / 4096 MiB Removed from CloudStack and Kubernetes

Retained nodes stayed Ready with unchanged guest/Kubernetes capacity, allocatable resources, boot IDs and kubelet start times. All 17 resident Pod UIDs/restart counts were unchanged; HTTP/DNS passed. Repeating the removed ID was rejected without changes.

Final cluster: 6 CPU / 7168 MiB. Global RAM returned to 9/~14 GiB after a 13-GiB test peak. Original node placement was restored.

Update for #14552

A scale request that names no offering (size=N, enabling autoscaling, cluster-autoscaler scale-up) no longer resizes every node to the old cluster-wide offering. New workers use the current worker offering.

A request that changes autoscaling and the worker offering together now keeps the autoscaling settings. Before, they were saved and then overwritten with the old values.

Live-tested on packages from 9614df02ac: 20 checks through the API, the web UI and the real cluster-autoscaler, all passed. Results are in this comment.

Not covered by this PR: the cluster-wide serviceofferingid still shows the old value after a per-group scale. That is tracked in #14702.

Do not infer a global resize when offering parameters are omitted. Keep node-type offerings in count-only accounting and continue processing explicit autoscaling changes without a resize. Add regression coverage for selected-worker removal, mixed and legacy offerings, and autoscaling-only requests.

Fixes #14458
@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 54.54545% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.80%. Comparing base (345db51) to head (9614df0).
⚠️ Report is 438 commits behind head on main.

Files with missing lines Patch % Lines
...er/actionworkers/KubernetesClusterScaleWorker.java 42.85% 1 Missing and 3 partials ⚠️
...bernetes/cluster/KubernetesClusterManagerImpl.java 75.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14459      +/-   ##
============================================
+ Coverage     18.07%   20.80%   +2.73%     
- Complexity    16357    20890    +4533     
============================================
  Files          5937     6053     +116     
  Lines        537029   548232   +11203     
  Branches      65833    73239    +7406     
============================================
+ Hits          97043   114039   +16996     
+ Misses       428901   420812    -8089     
- Partials      11085    13381    +2296     
Flag Coverage Δ
uitests 3.86% <ø> (-0.18%) ⬇️
unittests 22.23% <54.54%> (+3.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andrijapanicsb
andrijapanicsb marked this pull request as ready for review October 21, 2026 20:36
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

🔴 Test Coverage Grade: D — Marginal

Metric Value
Line coverage 25.73%
Branch coverage 20.91%

Grade Scale

Grade Line Coverage Meaning
🟢 A ≥ 80% Excellent - this code sleeps well at night 😴
🟡 B 60-79% Good - almost there, don't stop now 😉
🟠 C 40-59% Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D 20-39% Marginal - boldly shipping where no test has gone before 🖖
⛔ F < 20% Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

@andrijapanicsb andrijapanicsb added this to the 24.0 milestone Oct 21, 2026
@andrijapanicsb
andrijapanicsb requested review from Pearl1594 and nvazquez and removed request for Pearl1594 October 21, 2026 23:39
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19661

@modupei

modupei commented Oct 26, 2026 •

Copy link
Copy Markdown

@andrijapanicsb
We’re seeing a similar issue on CloudStack 4.23 with different control and worker plans. When autoscaling tries to add a worker, CloudStack tries to resize the existing worker and fails with error 530. This blocks our QA.

We haven’t tested this PR. Does it also cover this scale-up issue, and is a fix planned for 4.23?

@DaanHoogland

Copy link
Copy Markdown
Contributor

@andrijapanicsb We’re seeing a similar issue on CloudStack 4.23 with different control and worker plans. When autoscaling tries to add a worker, CloudStack tries to resize the existing worker and fails with error 530. This blocks our QA.

We haven’t tested this PR. Does it also cover this scale-up issue, and is a fix planned for 4.23?

@modupei 4.23 is not an LTS version. The next path to upgrade will be 24.0. This is aimed at that, but it is not sure yet if it will make it in.

@modupei

modupei commented Oct 29, 2026 •

Copy link
Copy Markdown

@andrijapanicsb We’re seeing a similar issue on CloudStack 4.23 with different control and worker plans. When autoscaling tries to add a worker, CloudStack tries to resize the existing worker and fails with error 530. This blocks our QA.
We haven’t tested this PR. Does it also cover this scale-up issue, and is a fix planned for 4.23?

@modupei 4.23 is not an LTS version. The next path to upgrade will be 24.0. This is aimed at that, but it is not sure yet if it will make it in.

@andrijapanicsb
Thanks for clarifying. We’ll follow this PR for updates on 24.0.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@andrijapanicsb We’re seeing a similar issue on CloudStack 4.23 with different control and worker plans. When autoscaling tries to add a worker, CloudStack tries to resize the existing worker and fails with error 530. This blocks our QA.

We haven’t tested this PR. Does it also cover this scale-up issue, and is a fix planned for 4.23?

#14223

I think these are all for 24.0 :)

A scale request that changed autoscaling and the worker offering together
saved the autoscaling settings and then wrote the old values back.
Reload the cluster after autoscaling so the offering update keeps them.
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package G

@ACSHomeBot

ACSHomeBot commented Nov 15, 2026 •

Copy link
Copy Markdown

Accepted: package kvm G for 9614df02ac5b.

Planned: RPM + DEB, KVM SystemVM.

Status: Completed. Packaging results.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@sonarqubecloud

Copy link
Copy Markdown

@ACSHomeBot

Copy link
Copy Markdown

Packaging results:

Result Artifact Platform Build Time
PASS RPM EL (EL8/9/10) 30 min
PASS DEB Ubuntu, Debian 22 min

Test packages are available at:

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@modupei find the packages above - 24.0.0-snapshot (i.e. built from this PR)

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20063

@andrijapanicsb

andrijapanicsb commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Live test results

In short: this PR was tested on a real CloudStack with real Kubernetes clusters, through the API, the web UI, and with the real cluster-autoscaler. A scale request that names no compute offering now leaves the existing nodes alone, and ordinary clusters with one offering keep working as before. All 20 checks passed.

Tested on packages built from 9614df02ac (24.0.0-SNAPSHOT, EL9), two management servers, two KVM hosts, Kubernetes 1.28.4.

Cluster with separate control and worker offerings (API)

"Out of date" means the workers were already on a bigger offering (3 GB) while the cluster-wide value still said 2 GB. This is the situation from #14552.

Situation What happened Outcome
Scale out by size only One worker added, control node kept its own offering PASS
Remove one named worker (#14458) Only that worker removed PASS
Autoscaling on, then off Only the settings changed PASS
Autoscaling on plus new worker offering in one request Workers resized, autoscaling stayed on with min and max kept PASS
Out of date: autoscaling off No node resized PASS
Out of date: scale out by size only New worker on 3 GB, others untouched PASS
Out of date: remove one named worker Only that worker removed PASS
Out of date: real cluster-autoscaler adds a node under load New worker on 3 GB, others untouched, waiting pods started PASS
Out of date: real cluster-autoscaler removes an idle node Empty worker removed, others untouched PASS
Explicit cluster-wide offering (behaviour that must stay) Every node resized to it PASS

Plain cluster, one offering for all nodes (API)

Situation What happened Outcome
Scale out by size only New worker on the cluster's one offering PASS
Scale in by size only One worker removed, nothing resized PASS
Remove one named worker Only that worker removed PASS
Autoscaling on, then off Settings saved and cleared, nothing resized PASS
Explicit cluster-wide offering change Both nodes resized, cluster-wide value updated PASS

Web UI

Action What happened Outcome
Scale dialog: autoscaling off No node resized PASS
Scale dialog: size 2 to 3 One worker added, others untouched PASS
Scale dialog: autoscaling on plus worker offering change, one submit Workers resized, control untouched, autoscaling kept PASS
Scale dialog: autoscaling off again No node resized PASS
Instances tab: Delete Node Only that worker removed PASS

The UI run also showed which callers were exposed. The Scale dialog always sends the current control and worker offerings, so it never hit #14552. The Delete Node button sends only the node ID, which is the #14458 case. API clients and the cluster-autoscaler send no offering at all.

Not covered

Noticed in existing code, not caused by this PR

Full report with screenshots: attached PDF.

pr14459-live-test-summary.pdf

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

5 participants