PVC stuck in Pending state if pod uses nodeName to pin it on a particular worker node

[nagyclaudiu83]

What steps did you take and what happened:
A PVC created from a StorageClass with WaitForFirstConsumer would not be provisioned if the pod that wants to use it is pinned to a node by using nodeName. It will work if the pod is pinned to a node by using nodeAffinity.

What did you expect to happen:
PVC gets provisioned

Anything else you would like to add:
This is a simplified example and set-up.
I have a SC

$ kubectl describe sc openebs-lvmpv
Name:            openebs-lvmpv
IsDefaultClass:  No
Provisioner:           local.csi.openebs.io
Parameters:            fstype=xfs,storage=lvm,volgroup=data_vg
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
AllowedTopologies:
  Term 0:              kubernetes.io/hostname in [example-node1, example-node2, example-node3]
Events:                <none>

I create 2 PVCs aaa2 and aaa3.

$ kubectl get pvc
NAME   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
aaa2   Pending                                      openebs-lvmpv   <unset>                 6s
aaa3   Pending                                      openebs-lvmpv   <unset>                 3s

I have 2 deployments that use aaa2 and aaa3 respectively. Only difference between them is the way that the pod is pinned to the node example-node1. Deployment aaa2 is pinned by using "nodeName: example-node1" in the pod spec and deployment aaa3 is pinned using a full affinity block.

$ kubectl apply -f pvc-test-deployment2.yaml -f pvc-test-deployment3.yaml
deployment.apps/aaa2 created
deployment.apps/aaa3 created
$ kubectl get pvc
NAME   STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
aaa2   Pending                                                                        openebs-lvmpv   <unset>                 5m26s
aaa3   Bound     pvc-03271706-5bbd-4270-8f11-507b73f86033   12Gi       RWO            openebs-lvmpv   <unset>                 5m23s

As it can be seen, aaa3 PVC and deployment work, but for the aaa2 PVC and deployment the provisioning doesn't work. Relevant events:

Pod event:

Unable to attach or mount volumes: unmounted volumes=[aaa2], unattached volumes=[], failed to process volumes=[aaa2]: error processing PVC default/aaa2: PVC is not bound

PVC event:

waiting for first consumer to be created before binding

In log of pod openebs-lvm-localpv-controller container there are no messages related to

provision "default/aaa2" class "openebs-lvmpv": started

Environment:

• LVM Driver version
• Kubernetes version (use kubectl version): 1.31
• Kubernetes installer & version: k3s v1.31.5+k3s1
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu 24.04

[tiagolobocastro]

This is expected behavior.

Here's a quote from there:

Note:

If you choose to use WaitForFirstConsumer, do not use nodeName in the Pod spec to specify node affinity. If nodeName is used in this case, the scheduler will be bypassed and PVC will remain in pending state.

Instead, you can use node selector for kubernetes.io/hostname

[nagyclaudiu83]

Thank you Tiago.

Good to learn something every day.

It is clear from the kubernetes documentation that it is the expected behaviout. I can only recommend the quote above is added to the lvm-localpv just in case other fail to find the info in kubernetes.io .

[tiagolobocastro]

@balaharish7 could you please add this note to the troubleshooting perhaps?