Prior workloads are still able to consume GPU even after gpu-operator operands are disabled

[anoopsinghnegi]

1. Quick Debug Information

• OS/Version: RHEL8.9
• Kernel Version: 4.18.0-513.11.1.el8_9
• Container Runtime Type/Version: Containerd
• K8s Flavor/Version: K8s
• GPU Operator Version: 23.9.1

2. Issue or feature description

When we disabled the GPU operator by applying the label nvidia.com/gpu.deploy.operands=false on the GPU node, the prior workloads(pods) are still able to consume GPU, and new GPU workloads go to the pending state.
GPU node describe logs show available GPU count '0' nvidia.com/gpu: 0
and when we enabled the GPU operator again setting nvidia.com/gpu.deploy.operands=true, the exiting pods which were consuming GPUs got terminated.

Is this the expected behaviour?
or, is there any configuration to terminate GPU workloads if the GPU operator gets disabled?

3. Steps to reproduce the issue

1. Installed the GPU Operator, all of its operands should be up and ready (driver, plugin, etc.)
2. Execute GPU workloads which consume GPU, and verify the nvidia-smi command to confirm process consuming GPU.
3. Apply nvidia.com/gpu.deploy.operands=false label on GPU nodes, the GPU operator will terminate the driver and other pods of GPU-Operators running on GPU nodes.
4. Previous running GPU workloads continue to use GPU and on the GPU node "nvidia-smi" will show the processes using GPU.

4. Information to [attach]

Before disabling the GPU operator operands

# kubectl get pods -A | grep nvidia

gpu-operator      nvidia-container-toolkit-daemonset-xfgtx                           1/1     Running             0              9h
gpu-operator      nvidia-cuda-validator-nlqpp                                        0/1     Completed           0              9h
gpu-operator      nvidia-dcgm-exporter-f6vj2                                         1/1     Running             0              9h
gpu-operator      nvidia-device-plugin-daemonset-9hbhb                               1/1     Running             0              9h
gpu-operator      nvidia-driver-daemonset-j4brr                                      1/1     Running             0              9h
gpu-operator      nvidia-mig-manager-kd5xr                                           1/1     Running             0              9h
gpu-operator      nvidia-operator-validator-g62x2                                    1/1     Running             0              9h

After disabling the GPU operator operands

# kubectl get pods -A | grep nvidia
gpu-operator      nvidia-cuda-validator-nlqpp                                        0/1     Completed           0              9h

Also, gpu-operator pod's logs are empty

# kubectl logs gpu-operator-231asx212 -n gpu-operator
#

We are not sure if is it possible to terminate the GPU workloads when the GPU-operator set to disabled or if it is a design restriction.

[cdesiniotis]

@anoopsinghnegi this is expected behavior and we don't currently automate this. Would it be sufficient for you to filter for pods which request a nvidia.com/gpu on the node and delete them, e.g. using a script?