container-toolkit does not modify the containerd config correctly when there are multiple instances of the `containerd` binary

[tariq1890]

NOTE: This issue is specific to container-toolkit when run as an OCI container via gpu-operator

To enable the nvidia-specific container runtime handlers, the toolkit must overlay config changes on the existing containerd configuration. For the toolkit to do this, it must retrieve the existing containerd config. Currently, it executes the following steps to achieve this:

i) Run containerd config dump (chroot into the host system)
ii) If i) fails, fall back to retrieving the config TOML from the file specified in CONTAINERD_CONFIG

This algorithm falls short in the scenario of multiple containerd instances running on the same host.

Consider the example of a k0s-based node which runs off a containerd embedded within the k0s system.

The same system also has a vanilla containerd installed. So we have containerd binaries in two locations

1. /usr/bin/containerd
2. /var/lib/k0s/bin/containerd

In this case, we expect the toolkit to modify the config of the k0s-embedded containerd. What ends up happening is - Step i) of the algorithm is run, which executes the containerd binary located in /usr/bin (this binary path is chosen since it is resolved via the PATH env var).

In this case, we would have wanted the toolkit to fall back to step ii., as it would then retrieve the desired config from the k0s-managed containerd; but the fallback is never triggered as step i. is successful.

Reproduction

To reproduce this issue

i. Install a vanilla containerd package on the host and ensure that it's running
ii. Install k0s and setup a k0s cluster.
iii. Install gpu-operator (which includes toolkit) with the necessary config overrides to point to k0s

helm install gpu-operator -n gpu-operator --create-namespace \
  nvidia/gpu-operator $HELM_OPTIONS \
    --version=v24.9.2 \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/etc/k0s/containerd.d/nvidia.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/run/k0s/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia

[diamonwiggins]

Closed a duplicate issue I had created in the gpu operator repo - NVIDIA/gpu-operator#1323. Some additional context and a temporary workaround below for others running into this.

Additional Context

GPU Operator v24.9.x switched to fetching container runtime configuration via CLI, causing failures specifically in Kubernetes distributions like k0s that statically compile containerd binaries. Although issue #777 added fallback support if containerd CLI doesn't exist, the current problem persists because there is no explicit way to always enforce using the configuration file instead of the containerd CLI(to my knowledge).

References:
Original related issue (#1109)
PR to implement fallback to file base retreival(#777)

Temporary Workaround

Either:

• Downgrade GPU Operator to v24.6.2 and override the driver version to 550.127.05, or
• Use GPU Operator v24.9.2 and downgrade NVIDIA Container Toolkit to version 1.16.2.

helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v24.9.2 \
    --set toolkit.version=v1.16.2-ubuntu20.04 \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/etc/k0s/containerd.d/nvidia.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/run/k0s/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia

[elezar]

@diamonwiggins since you have an environment that reproduces this information would you be able to verify that:

• that a socket can be passed to the containerd config dump command
• that the containerd config dump command returns the "correct" config when the socket is specified

[elezar]

@tariq1890 is the following really the desired functionality:

In this case, we would have wanted the toolkit to fall back to step ii., as it would then retrieve the desired config from the k0s-managed containerd; but the fallback is never triggered as step i. is successful.

Would retrieving the config from the k0s containerd binary not also be sufficient? Note that it seems that we're generating an NVIDIA-specific config and not a full config in the containerd.d folder seeming to idicate that the behaviour may need to be more complex in this case.

[diamonwiggins]

since you have an environment that reproduces this information would you be able to verify that:

• that a socket can be passed to the containerd config dump command
• that the containerd config dump command returns the "correct" config when the socket is specified

@elezar The following two commands produce the same config for me:

containerd --address=/run/k0s/containerd.sock config dump

containerd --address=/run/containerd/containerd.sock config dump

No matter what address i pass in, the grpc address in the config dumped seems to never change. It's always:

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

If i uncomment and modify the grpc address via /etc/containerd/config.toml then the correct address is shown in the config dump, but it still doesn't look like the "correct" config when /run/k0s/containerd.sock is set.

Running containerd -c /etc/k0s/containerd.toml config dump generates a config that has Nvidia specific information, so feels like the CLI gives more precedence to the config vs. the selected socket.

[elezar]

Thanks for checking the behaviour @diamonwiggins. This means that we can't "simply" specify the socket when running an arbitrary containerd binary and expect the config to be consistent. One option we would have is to allow the path to the contianerd binary to be specificed as an argument to the toolkit container instead of looking for this in the path. (This is what I was refering to in #982 (comment) but I did not express it clearly).

Could you confirm that running the config dump command k0s containerd binary renders the expected config?

[niklasfrick]

Any news on this?

[max-augmodo]

I ran into a variant of this issue with the following setup:

1. Create an Ubuntu 24.04 machine.

2. Install Docker via docker-ce 5:28.2.2-1~ubuntu.24.04~noble package which

3. also installs containerd via containerd.io 1.7.27-1 package as an APT dependency.

4. Later, install k3s v1.32.5+k3s1 which embeds containerd v2.0.5-k3s1.32 component with a command like curl -sfL https://get.k3s.io | sh -

5. Finally, install gpu-operator v25.3.0 installed with the following k3s HelmChart:

   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: gpu-operator
   ---
   apiVersion: helm.cattle.io/v1
   kind: HelmChart
   metadata:
     name: gpu-operator
     namespace: gpu-operator
   spec:
     repo: https://helm.ngc.nvidia.com/nvidia
     chart: gpu-operator
     version: v25.3.0
     targetNamespace: gpu-operator
     valuesContent: |-
       toolkit:
         env:
         - name: CONTAINERD_SOCKET
           value: /run/k3s/containerd/containerd.sock
   

This creates a confusing ticking timebomb which is suspect works like the following:

1. Intially, docker.service and containerd.service systemd units continue to run and all your Docker containers will continue to run like normal.
2. gpu-operator will happily install NVIDIA drivers and trigger the correct k3s containerd to restart (via the socket). Then, you can freely run GPU workloads with the runtimeClassName: nvidia option. Guess: I think this works because K3s automatically configures the /var/lib/rancher/k3s/agent/etc/containerd/config.toml with enough settings to work, see https://docs.k3s.io/advanced#alternative-container-runtime-support page.
3. BUT! gpu-operator also changes the /etc/containerd/config.toml file which creates a ticking timebomb because the host system doesn't have enough NVIDIA stuff (or maybe just doesn't have a compatible containerd to read the updated file).
4. Things can go on like this; but when you restart containerd with sudo systemctl restart containerd (or reboot your server), then containerd will fail with a containerd: failed to load TOML from /etc/containerd/config.toml: invalid disabled plugin URI "cri" expect io.containerd.x.vx error which prevents docker from restarting. K3s will continue to run.
5. To recover, you can move the /etc/containerd/config.toml out of the way and then restart containerd.

For reference, see following containerd config files:

1. /etc/containerd/config.toml.orig which I think containerd.io installs:

   #   Copyright 2018-2022 Docker Inc.
   
   #   Licensed under the Apache License, Version 2.0 (the "License");
   #   you may not use this file except in compliance with the License.
   #   You may obtain a copy of the License at
   
   #       http://www.apache.org/licenses/LICENSE-2.0
   
   #   Unless required by applicable law or agreed to in writing, software
   #   distributed under the License is distributed on an "AS IS" BASIS,
   #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   #   See the License for the specific language governing permissions and
   #   limitations under the License.
   
   disabled_plugins = ["cri"]
   
   #root = "/var/lib/containerd"
   #state = "/run/containerd"
   #subreaper = true
   #oom_score = 0
   
   #[grpc]
   #  address = "/run/containerd/containerd.sock"
   #  uid = 0
   #  gid = 0
   
   #[debug]
   #  address = "/run/containerd/debug.sock"
   #  uid = 0
   #  gid = 0
   #  level = "info"
   

2. /etc/containerd/config.toml which I suspect gpu-operator is writing to the host system:

   disabled_plugins = ["cri"]
   imports = ["/etc/containerd/config.toml"]
   oom_score = 0
   plugin_dir = ""
   required_plugins = []
   root = "/var/lib/containerd"
   state = "/run/containerd"
   temp = ""
   version = 2
   
   [cgroup]
     path = ""
   
   [debug]
     address = ""
     format = ""
     gid = 0
     level = ""
     uid = 0
   
   [grpc]
     address = "/run/containerd/containerd.sock"
     gid = 0
     max_recv_message_size = 16777216
     max_send_message_size = 16777216
     tcp_address = ""
     tcp_tls_ca = ""
     tcp_tls_cert = ""
     tcp_tls_key = ""
     uid = 0
   
   [metrics]
     address = ""
     grpc_histogram = false
   
   [plugins]
   
     [plugins."io.containerd.gc.v1.scheduler"]
       deletion_threshold = 0
       mutation_threshold = 100
       pause_threshold = 0.02
       schedule_delay = "0s"
       startup_delay = "100ms"
   
     [plugins."io.containerd.internal.v1.opt"]
       path = "/opt/containerd"
   
     [plugins."io.containerd.internal.v1.restart"]
       interval = "10s"
   
     [plugins."io.containerd.internal.v1.tracing"]
   
     [plugins."io.containerd.metadata.v1.bolt"]
       content_sharing_policy = "shared"
   
     [plugins."io.containerd.monitor.v1.cgroups"]
       no_prometheus = false
   
     [plugins."io.containerd.nri.v1.nri"]
       disable = true
       disable_connections = false
       plugin_config_path = "/etc/nri/conf.d"
       plugin_path = "/opt/nri/plugins"
       plugin_registration_timeout = "5s"
       plugin_request_timeout = "2s"
       socket_path = "/var/run/nri/nri.sock"
   
     [plugins."io.containerd.runtime.v1.linux"]
       no_shim = false
       runtime = "runc"
       runtime_root = ""
       shim = "containerd-shim"
       shim_debug = false
   
     [plugins."io.containerd.runtime.v2.task"]
       platforms = ["linux/amd64"]
       sched_core = false
   
     [plugins."io.containerd.service.v1.diff-service"]
       default = ["walking"]
       sync_fs = false
   
     [plugins."io.containerd.service.v1.tasks-service"]
       blockio_config_file = ""
       rdt_config_file = ""
   
     [plugins."io.containerd.snapshotter.v1.aufs"]
       root_path = ""
   
     [plugins."io.containerd.snapshotter.v1.blockfile"]
       fs_type = ""
       mount_options = []
       root_path = ""
       scratch_file = ""
   
     [plugins."io.containerd.snapshotter.v1.btrfs"]
       root_path = ""
   
     [plugins."io.containerd.snapshotter.v1.devmapper"]
       async_remove = false
       base_image_size = ""
       discard_blocks = false
       fs_options = ""
       fs_type = ""
       pool_name = ""
       root_path = ""
   
     [plugins."io.containerd.snapshotter.v1.native"]
       root_path = ""
   
     [plugins."io.containerd.snapshotter.v1.overlayfs"]
       mount_options = []
       root_path = ""
       sync_remove = false
       upperdir_label = false
   
     [plugins."io.containerd.snapshotter.v1.zfs"]
       root_path = ""
   
     [plugins."io.containerd.tracing.processor.v1.otlp"]
   
     [plugins."io.containerd.transfer.v1.local"]
       config_path = ""
       max_concurrent_downloads = 3
       max_concurrent_uploaded_layers = 3
   
       [[plugins."io.containerd.transfer.v1.local".unpack_config]]
         differ = ""
         platform = "linux/amd64"
         snapshotter = "overlayfs"
   
   [proxy_plugins]
   
   [stream_processors]
   
     [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
       accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
       args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
       env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
       path = "ctd-decoder"
       returns = "application/vnd.oci.image.layer.v1.tar"
   
     [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
       accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
       args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
       env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
       path = "ctd-decoder"
       returns = "application/vnd.oci.image.layer.v1.tar+gzip"
   
   [timeouts]
     "io.containerd.timeout.bolt.open" = "0s"
     "io.containerd.timeout.metrics.shimstats" = "2s"
     "io.containerd.timeout.shim.cleanup" = "5s"
     "io.containerd.timeout.shim.load" = "5s"
     "io.containerd.timeout.shim.shutdown" = "3s"
     "io.containerd.timeout.task.state" = "2s"
   
   [ttrpc]
     address = ""
     gid = 0
     uid = 0
   

3. /var/lib/rancher/k3s/agent/etc/containerd/config.toml which K3s automatically generates:

   # File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
   version = 3
   root = "/var/lib/rancher/k3s/agent/containerd"
   state = "/run/k3s/containerd"
   
   [grpc]
     address = "/run/k3s/containerd/containerd.sock"
   
   [plugins.'io.containerd.internal.v1.opt']
     path = "/var/lib/rancher/k3s/agent/containerd"
   
   [plugins.'io.containerd.grpc.v1.cri']
     stream_server_address = "127.0.0.1"
     stream_server_port = "10010"
   
   [plugins.'io.containerd.cri.v1.runtime']
     enable_selinux = false
     enable_unprivileged_ports = true
     enable_unprivileged_icmp = true
     device_ownership_from_security_context = false
   
   [plugins.'io.containerd.cri.v1.images']
     snapshotter = "overlayfs"
     disable_snapshot_annotations = true
   
   [plugins.'io.containerd.cri.v1.images'.pinned_images]
     sandbox = "rancher/mirrored-pause:3.6"
   
   [plugins.'io.containerd.cri.v1.runtime'.cni]
     bin_dir = "/var/lib/rancher/k3s/data/cni"
     conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
     runtime_type = "io.containerd.runc.v2"
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
     SystemdCgroup = true
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
     runtime_type = "io.containerd.runhcs.v1"
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
     runtime_type = "io.containerd.runc.v2"
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
     BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
     SystemdCgroup = true
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia-cdi']
     runtime_type = "io.containerd.runc.v2"
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia-cdi'.options]
     BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
     SystemdCgroup = true
   
   [plugins.'io.containerd.cri.v1.images'.registry]
     config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"
   
   [plugins.'io.containerd.cri.v1.images'.registry.configs.'us-central1-docker.pkg.dev'.auth]
     username = "_json_key"
     password = "...
   

For a path forward with docker+containerd on host and containerd+gpu-operator on k3s (on host), I wonder what's best. Because K3s automatically handles the nvidia runtime, maybe there's a way to just avoid writing to /etc/containerd/config.toml and let K3s handle it. If gpu-operator does need to configure containerd, then K3s projects suggestes an approach involving a config.toml.tmpl template https://docs.k3s.io/advanced#configuring-containerd. Please let me know if I should open a separate issue since even though this variant is about multiple containerd instances, there's probably some different details between k3s and k0s.

For an overly simple workaround, I sent the gpu-operator containerd configs to /dev/null like this:

---
apiVersion: v1
kind: Namespace
metadata:
  name: gpu-operator
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: gpu-operator
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v25.3.0
  targetNamespace: gpu-operator
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_CONFIG
        value: /dev/null
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock

which seems to work for k3s; but long term probably isn't correct since then I'm missing the containerd configs that gpu-operator decides to add.

[diamonwiggins]

Could you confirm that running the config dump command k0s containerd binary renders the expected config?

@elezar my apologies, as I missed your follow up for additional information a while back. Is there any information that I can provide to help push this along?