crictl info does not match containerd.toml

[santurini]

I am experiencing an issue with the NVIDIA device plugin on my k3s cluster after enabling MPS mode. In fact switching from the GPU-Operator to the standalone nvidia-device-plugin, every time a systemctl daemon-reload is triggered, my pods lose access to the GPUS (NVIDIA/nvidia-container-toolkit#48).

As the official issue suggest, a workaround should be changing the /etc/containerd/config.toml in order to set SystemdCgroup = false.

Even if I do this, when I run crictl info this is the configuration I have for the nvidia runtime:

  "config":
      ...
      "runtimes": {
        "nvidia": {
          ...
          "options": {
            "BinaryName": "/usr/local/nvidia/toolkit/nvidia-container-runtime",
            "SystemdCgroup": true
          },
        },

While this is my containerd configuration:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
  BinaryName = "/usr/bin/nvidia-container-runtime"
  CriuImagePath = ""
  CriuPath = ""
  CriuWorkPath = ""
  IoGid = 0
  IoUid = 0
  NoNewKeyring = false
  NoPivotRoot = false
  Root = ""
  ShimCgroup = ""
  SystemdCgroup = false

How can I change the configuration I get when running crictl info to change the SystemdCgroup ?

[brandond]

Please read the docs: https://docs.k3s.io/advanced#configuring-containerd

K3s does not use /etc/containerd/config.toml.