OCPBUGS-62270: UPSTREAM: <carry>: Skip CPU resource status for workload-pinned pods #2490

harche · 2025-10-16T16:56:16Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes CPU double-counting for workload-partitioned (managed) pods when
InPlacePodVerticalScaling is enabled (default in Kubernetes 1.33+).

When workload partitioning is enabled, pods have CPU resources specified as
management.workload.openshift.io/cores in spec.requests. However, the kubelet also populates cpu
in status.containerStatuses[].resources.requests by reading from cgroups. This causes the scheduler
to count CPU twice:

From management.workload.openshift.io/cores in pod spec
From cpu in pod status

This leads to scheduler seeing artificially inflated CPU usage, resulting in false "Insufficient
cpu" errors when scheduling new pods.

Which issue(s) this PR fixes:

Fixes # https://issues.redhat.com/browse/OCPBUGS-62270

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fixed CPU double-counting issue for workload-partitioned pods that caused false "Insufficient cpu"
  scheduling errors when InPlacePodVerticalScaling is enabled.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

openshift-ci-robot · 2025-10-16T16:56:26Z

@harche: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

edb8bd5|Skip CPU resource status for workload-pinned pods: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

haircommander · 2025-10-16T17:01:35Z

pkg/kubelet/kuberuntime/kuberuntime_container.go

 		return nil, nil, err
 	}

+	isManagedPod := utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodVerticalScaling) && managed.IsManagedPodFromRuntimeService(ctx, m.runtimeService, activePodSandboxID)


I don't think we need to check for feature gate here. cStatus.Resources != nil will only be true if the feature is enabled

Addressed, thanks.

haircommander · 2025-10-16T17:03:05Z

pkg/kubelet/managed/managed.go


+// IsPodSandboxManagedPod checks if a pod sandbox belongs to a managed pod
+// by looking for workload pinning annotations.
+func IsPodSandboxManagedPod(sandboxAnnotations map[string]string) bool {


nit: can this be defined below where it's called in IsManagedPodFromRuntimeService

Addressed, thanks.

still not addressed it seems?

just a nit tho, I'm fine leaving as is

openshift-ci-robot · 2025-10-16T17:34:43Z

@harche: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

3fc7c95|Skip CPU resource status for workload-pinned pods: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2025-10-20T17:46:14Z

@harche: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

ff0c855|Skip CPU resource status for workload-pinned pods: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2025-10-20T17:47:52Z

@harche: This pull request references Jira Issue OCPBUGS-62270, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-10-20T17:49:51Z

@harche: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

9382bbe|UPSTREAM: : Skip CPU resource status for workload-pinned pods: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

openshift-ci-robot · 2025-10-20T17:53:10Z

@harche: This pull request references Jira Issue OCPBUGS-62270, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes CPU double-counting for workload-partitioned (managed) pods when
InPlacePodVerticalScaling is enabled (default in Kubernetes 1.33+).

When workload partitioning is enabled, pods have CPU resources specified as
management.workload.openshift.io/cores in spec.requests. However, the kubelet also populates cpu
in status.containerStatuses[].resources.requests by reading from cgroups. This causes the scheduler
to count CPU twice:

From management.workload.openshift.io/cores in pod spec

From cpu in pod status

This leads to scheduler seeing artificially inflated CPU usage, resulting in false "Insufficient
cpu" errors when scheduling new pods.

Which issue(s) this PR fixes:

Fixes # https://issues.redhat.com/browse/OCPBUGS-62270

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Fixed CPU double-counting issue for workload-partitioned pods that caused false "Insufficient cpu"
 scheduling errors when InPlacePodVerticalScaling is enabled.
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

harche · 2025-10-20T17:54:45Z

/jira refresh

openshift-ci-robot · 2025-10-20T17:54:54Z

@harche: This pull request references Jira Issue OCPBUGS-62270, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

harche · 2025-10-20T17:59:06Z

/hold for another round of testing with real cluster after resolving merge conflicts during rebasing.

harche · 2025-10-20T19:01:13Z

/retest-required

haircommander · 2025-10-21T18:26:32Z

/lgtm

harche · 2025-10-21T18:53:50Z

/test integration

harche · 2025-10-22T00:50:31Z

/test e2e-aws-ovn-serial

sdodson · 2025-10-22T12:51:15Z

/test verify-commits

openshift-ci · 2025-10-22T13:05:07Z

@harche: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-techpreview-serial	`9e371ef`	link	false	`/test e2e-aws-ovn-techpreview-serial`
ci/prow/okd-scos-e2e-aws-ovn	`9e371ef`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

harche · 2025-10-23T14:28:45Z

/test e2e-aws-ovn-techpreview

rphillips · 2025-10-27T15:06:38Z

/hold cancel
/approve

rphillips · 2025-10-27T15:08:11Z

/verified
/label remove backports/unvalidated-commits

openshift-ci-robot · 2025-10-27T15:08:14Z

@rphillips: /verified bypass and /verified remove do not support arguments. See https://docs.ci.openshift.org/docs/architecture/jira/#premerge-verification for more information.

In response to this:

/verified
/label remove backports/unvalidated-commits

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-10-27T15:09:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, harche, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/DOWNSTREAM_OWNERS~~ [rphillips]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rphillips · 2025-10-27T15:12:28Z

/verified by @harche

openshift-ci-robot · 2025-10-27T15:12:41Z

@rphillips: This PR has been marked as verified by @harche.

In response to this:

/verified by @harche

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rphillips · 2025-10-27T15:14:56Z

/label remove backports/unvalidated-commits

rphillips · 2025-10-27T15:16:10Z

/remove-label backports/unvalidated-commits

openshift-ci · 2025-10-27T15:16:23Z

@rphillips: Can not set label backports/unvalidated-commits: Must be member in one of these teams: [openshift-staff-engineers]

In response to this:

/remove-label backports/unvalidated-commits

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

sdodson · 2025-10-27T15:23:24Z

/remove-label backports/unvalidated-commits

openshift-ci-robot · 2025-10-27T15:27:51Z

/retest-required

Remaining retests: 0 against base HEAD 96593f3 and 2 for PR HEAD 9e371ef in total

openshift-ci-robot · 2025-10-27T15:39:15Z

@harche: Jira Issue Verification Checks: Jira Issue OCPBUGS-62270
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-62270 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

In response to this:

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes CPU double-counting for workload-partitioned (managed) pods when
InPlacePodVerticalScaling is enabled (default in Kubernetes 1.33+).

When workload partitioning is enabled, pods have CPU resources specified as
management.workload.openshift.io/cores in spec.requests. However, the kubelet also populates cpu
in status.containerStatuses[].resources.requests by reading from cgroups. This causes the scheduler
to count CPU twice:

From management.workload.openshift.io/cores in pod spec

From cpu in pod status

This leads to scheduler seeing artificially inflated CPU usage, resulting in false "Insufficient
cpu" errors when scheduling new pods.

Which issue(s) this PR fixes:

Fixes # https://issues.redhat.com/browse/OCPBUGS-62270

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Fixed CPU double-counting issue for workload-partitioned pods that caused false "Insufficient cpu"
 scheduling errors when InPlacePodVerticalScaling is enabled.
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

haircommander · 2025-10-27T16:09:34Z

/cherry-pick release-4.20

openshift-cherrypick-robot · 2025-10-27T16:12:17Z

@haircommander: #2490 failed to apply on top of branch "release-4.20":

Applying: UPSTREAM: <carry>: Skip CPU resource status for workload-pinned pods
Using index info to reconstruct a base tree...
M	pkg/kubelet/kuberuntime/kuberuntime_container.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/kubelet/kuberuntime/kuberuntime_container.go
CONFLICT (content): Merge conflict in pkg/kubelet/kuberuntime/kuberuntime_container.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 UPSTREAM: <carry>: Skip CPU resource status for workload-pinned pods

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-merge-robot · 2025-10-29T14:09:27Z

Fix included in accepted release 4.20.0-0.nightly-2025-10-29-051631

openshift-ci-robot added the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Oct 16, 2025

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 16, 2025

openshift-ci bot requested review from mrunalp and rphillips October 16, 2025 16:57

haircommander reviewed Oct 16, 2025

View reviewed changes

harche force-pushed the OCPBUGS-61220 branch from edb8bd5 to 3fc7c95 Compare October 16, 2025 17:34

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 19, 2025

harche force-pushed the OCPBUGS-61220 branch from 3fc7c95 to ff0c855 Compare October 20, 2025 17:46

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 20, 2025

harche changed the title ~~[WIP] Skip CPU resource status for workload-pinned pods~~ OCPBUGS-62270: Skip CPU resource status for workload-pinned pods Oct 20, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 20, 2025

openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Oct 20, 2025

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Oct 20, 2025

harche changed the title ~~OCPBUGS-62270: Skip CPU resource status for workload-pinned pods~~ OCPBUGS-62270: UPSTREAM: <carry>: Skip CPU resource status for workload-pinned pods Oct 20, 2025

harche force-pushed the OCPBUGS-61220 branch from ff0c855 to 9382bbe Compare October 20, 2025 17:49

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 20, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2025

openshift-ci bot assigned haircommander Oct 21, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 21, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 27, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 27, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 27, 2025

openshift-ci bot removed the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Oct 27, 2025

openshift-merge-bot bot merged commit ce4a3ce into openshift:master Oct 27, 2025
20 of 22 checks passed

harche mentioned this pull request Oct 27, 2025

[release-4.20] OCPBUGS-63601: Skip CPU resource status for workload-partitioning #2499

Closed

sdodson mentioned this pull request Oct 28, 2025

OCPBUGS-63601: UPSTREAM: <carry>: Skip CPU resource status for workload-partitioning #2504

Merged

OCPBUGS-62270: UPSTREAM: <carry>: Skip CPU resource status for workload-pinned pods #2490

OCPBUGS-62270: UPSTREAM: <carry>: Skip CPU resource status for workload-pinned pods #2490

Uh oh!

Conversation

harche commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

openshift-ci-robot commented Oct 16, 2025

Uh oh!

haircommander Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

harche Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

haircommander Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

harche Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

haircommander Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

haircommander Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Oct 16, 2025

Uh oh!

openshift-ci-robot commented Oct 20, 2025

Uh oh!

openshift-ci-robot commented Oct 20, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

openshift-ci-robot commented Oct 20, 2025

Uh oh!

openshift-ci-robot commented Oct 20, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

harche commented Oct 20, 2025

Uh oh!

openshift-ci-robot commented Oct 20, 2025

Uh oh!

harche commented Oct 20, 2025

Uh oh!

harche commented Oct 20, 2025

Uh oh!

haircommander commented Oct 21, 2025

Uh oh!

harche commented Oct 21, 2025

Uh oh!

harche commented Oct 22, 2025

Uh oh!

sdodson commented Oct 22, 2025

Uh oh!

openshift-ci bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harche commented Oct 23, 2025

Uh oh!

rphillips commented Oct 27, 2025

Uh oh!

rphillips commented Oct 27, 2025

Uh oh!

harche commented Oct 16, 2025 •

edited

Loading

openshift-ci bot commented Oct 22, 2025 •

edited

Loading