[release-4.20] OCPBUGS-63542: Change the capacity struct from int to ptrOfInt #185
Conversation
Today, in CNCC we store the capacity values as
integers:
type capacity struct {
IPv4 int `json:"ipv4,omitempty"`
IPv6 int `json:"ipv6,omitempty"`
IP int `json:"ip,omitempty"`
}
When capacity is full, CNCC sets the value to 0.
Also, depending on the platform it also ignores
setting fields it doesn't care about (example AWS
doesn't use IP, gcp and azure don't use IPv4 and IPv6).
However given we have omitempty set, this was omitting
the zero value in the annotation. When OVN-Kubernetes
reads this annotation it was then setting the capacity
to unlimited:
nodeEgressIPConfig := []nodeEgressIPConfiguration{
{
Capacity: Capacity{
IP: UnlimitedNodeCapacity,
IPv4: UnlimitedNodeCapacity, --> we set this to maxint32
IPv6: UnlimitedNodeCapacity,
},
},
}
which is causing all EgressIPs to be
assigned to this node leading to:
status:
conditions:
- lastTransitionTime: "2025-10-06T19:24:24Z"
message: "Error processing cloud assignment request, err: PrivateIpAddressLimitExceeded:
Number of private addresses will exceed limit.\n\tstatus code: 400, request
id: 457f4332-e9c4-44c9-bfcf-deeb5e7e43ce"
In this fix, what we really want is to remove omitempty
so that the zero capacity gets reflected correctly, however
doing so also means fields that are unset will also be zero
which can lead to confusion. Basically we are not able to
distinguish between unset field and 0 value fields.
Hence we are changing the capacity struct to be pointer type
values so that null/nil means unset and 0 means full capacity.
We still keep the omitempty since we don't need to do anything
with unset fields - there is no behaviour change there and
OVN-Kubernetes will continue to treat that as unlimited
capacity.
Upgrades: CNCC upon reboot seems to call:
func (n *NodeController) SyncHandler(key string) error {
....
// Filter out cloudPrivateIPConfigs assigned to node (key) and write the entry
// into same slice starting from index 0, finally chop off unwanted entries
// when passing it into GetNodeEgressIPConfiguration.
index := 0
for _, cloudPrivateIPConfig := range cloudPrivateIPConfigs {
if isAssignedCloudPrivateIPConfigOnNode(cloudPrivateIPConfig, key) {
cloudPrivateIPConfigs[index] = cloudPrivateIPConfig
index++
}
}
nodeEgressIPConfigs, err := n.cloudProviderClient.GetNodeEgressIPConfiguration(node, cloudPrivateIPConfigs[:index])
if err != nil {
return fmt.Errorf("error retrieving the private IP configuration for node: %s, err: %v", node.Name, err)
}
return n.SetNodeEgressIPConfigAnnotation(node, nodeEgressIPConfigs)
}
// SetCloudPrivateIPConfigAnnotationOnNode annotates the corev1.Node with the cloud subnet information and capacity
func (n *NodeController) SetNodeEgressIPConfigAnnotation(node *corev1.Node, nodeEgressIPConfigs []*cloudprovider.NodeEgressIPConfiguration) error {
annotation, err := n.generateAnnotation(nodeEgressIPConfigs)
if err != nil {
return err
}
klog.Infof("Setting annotation: '%s: %s' on node: %s", nodeEgressIPConfigAnnotationKey, annotation, node.Name)
return retry.RetryOnConflict(retry.DefaultRetry, func() error {
ctx, cancel := context.WithTimeout(n.ctx, controller.ClientTimeout)
defer cancel()
// See: updateCloudPrivateIPConfigStatus
nodeLatest, err := n.kubeClient.CoreV1().Nodes().Get(ctx, node.Name, metav1.GetOptions{})
if err != nil {
return err
}
existingAnnotations := nodeLatest.Annotations
existingAnnotations[nodeEgressIPConfigAnnotationKey] = annotation
nodeLatest.SetAnnotations(existingAnnotations)
_, err = n.kubeClient.CoreV1().Nodes().Update(ctx, nodeLatest, metav1.UpdateOptions{})
return err
})
}
and we seem to be overwriting the annotation - so we should be good on upgrades
in changing from older annotations to new annotations - where 0 valued fields
will appear for full capacity nodes.
Once that happens, OVN-Kubernetes should overrite the UnlimitedValue to value 0
tat indicates 0 capacity and we should enter:
if eNode.egressIPConfig.Capacity.IP < util.UnlimitedNodeCapacity {
if eNode.egressIPConfig.Capacity.IP-len(eNode.allocations) <= 0 {
klog.V(5).Infof("Additional allocation on Node: %s exhausts it's IP capacity, trying another node", eNode.name)
continue
}
}
if eNode.egressIPConfig.Capacity.IPv4 < util.UnlimitedNodeCapacity && utilnet.IsIPv4(eIP) {
if eNode.egressIPConfig.Capacity.IPv4-getIPFamilyAllocationCount(eNode.allocations, false) <= 0 {
klog.V(5).Infof("Additional allocation on Node: %s exhausts it's IPv4 capacity, trying another node", eNode.name)
continue
}
}
if eNode.egressIPConfig.Capacity.IPv6 < util.UnlimitedNodeCapacity && utilnet.IsIPv6(eIP) {
if eNode.egressIPConfig.Capacity.IPv6-getIPFamilyAllocationCount(eNode.allocations, true) <= 0 {
klog.V(5).Infof("Additional allocation on Node: %s exhausts it's IPv6 capacity, trying another node", eNode.name)
continue
}
}
these desired conditions correctly.
Signed-off-by: Surya Seetharaman <[email protected]>
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-60806 has been cloned as Jira Issue OCPBUGS-63542. Will retitle bug to link to clone. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-63542, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@zshi-redhat: This pull request references Jira Issue OCPBUGS-63542, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
/retest |
openshift-ci-robot
added
jira/valid-bug
|
@zshi-redhat: This pull request references Jira Issue OCPBUGS-63542, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@openshift-cherrypick-robot: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/verified by pre-merge test. 1. create aws cluster with PR build 2. label a node as egress node, ipv4 capacity is 14 3. assign 14 external IPs on node via aws console, then restart CNCC 4. check ipv4 capacity on node, it shows "ipv4":0 as expected 5. create one more EIP, the EIP will not be assigned to the node due to no capacity as expected 6. check cloudprivateipconfigs, there is no entry for the new created EIP 7. scale up another node in the same subnet, and label it with k8s.ovn.org/egress-assignable when it is Ready 8. check the new added EIP will be assigned to the new added node, cloudprivateipconfig status is CloudResponseSuccess pre-merge tested for upgrade on aws cluster: 1. create aws cluster with 4.19 nightly build 2. label a node as egress node, ipv4 capacity is 14 3. assign 14 external IPs on node via aws console, then restart CNCC 4. check ipv4 capacity on node, it disappeared 5. create one more EIP, got cloudprivateipconfig with CloudResponseError 6. upgrade the cluster to PR build 7. check the ip capacity on the node, it displays "ipv4":0 as expected |
openshift-ci-robot
added
the
verified
|
@qiowang721: This PR has been marked as verified by In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: openshift-cherrypick-robot, ricky-rav The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/label backport-risk-assessed |
openshift-ci
bot
added
the
backport-risk-assessed
openshift-ci
bot
assigned
asood-rh,
barbora137,
huiran0826,
jechen0648,
Meina-rh,
mffiedler and
qiowang721
openshift-merge-bot
bot
merged commit a365cf9
into
openshift:release-4.20
|
@openshift-cherrypick-robot: Jira Issue Verification Checks: Jira Issue OCPBUGS-63542 Jira Issue OCPBUGS-63542 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cherry-pick release-4.19 |
|
@tssurya: new pull request created: #186 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
14 participants
This is an automated cherry-pick of #183
/assign tssurya