How to upgrade driver version?

[adityapatadia]

We are using this guide to install drivers: https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers

Now, the drivers for COS are locked and it always installs 450.119.04. We want to upgrade driver to version 460.32.03 because https://github.com/FFmpeg/nv-codec-headers needs driver version 455.28 or newer.

How can we upgrade driver version?

[garyo]

Same issue -- need at least 455.x for recent Optix.

[kewu1992]

The latest COS-89 version (cos-89-16108-470-1) supports 460.73.01 although it is not the default GPU version. It seems GKE hasn't adopted the COS version so you may need to wait for some time. Once the COS version is available on GKE, a custom daemonset can be used to specify the GPU version to install:

 initContainers:
      - image: "cos-nvidia-installer:fixed"
        imagePullPolicy: Never
        name: nvidia-driver-installer
        resources:
          requests:
            cpu: "0.15"
        securityContext:
          privileged: true
        command:
        - install
        - "-version=460.73.01"
        env:
# the following is the same as the original yaml file

Note that this is not the official interface and we may stop supporting the specific 460.73.01 in future COS releases. If you need have a specific [GKE version, GPU driver version] combination that you'd like us to support, please file a feature request against GKE product.

[kewu1992]

@pradvenkat

[adityapatadia]

How do we know when GKE has updated version available?

[adityapatadia]

Currently, GKE supports cos-89-16108-403-46 so how do we know which driver this supports?

[pradvenkat]

How do we know when GKE has updated version available?

GKE release notes should tell you about these types of changes: https://cloud.google.com/kubernetes-engine/docs/release-notes. I expect cos-89-16108-470-1 should be available in rapid channel this week.