Add CUDA forward compatibility hook #906

elezar · 2025-02-06T16:58:40Z

With #877 the default behaviour of the NVIDIA Container Runtime / NVIDIA Container Runtime Hook was changed to not mount compat libraries from the container into the container. This removed "automatic" support for CUDA Forward compatibility.

This change attempts to address this by adding a createContainerHook that will create a file in /etc/ld.so.conf.d/ in the container to ensure that the /usr/local/cuda/compat libraries are added to the ldcache over the libraries mounted from the host. The provided host diver version is compared to the version of the compat libraries in the container and the config update is only performed if the compat libraries are newer than the host drivers.

Note that the hook only creates a file in the container's file system and does not perform any mount operations. This means that this mechanism is not present the same vulnerabilities causing CVE-2024-0132 and CVE-2025-23359.

In the case of the legacy runtime, this behaviour is only triggered if the allow-cuda-compat-libs-from-container feature flag is not enabled. The CDI spec generation has also been extended to include this hook.

cmd/nvidia-cdi-hook/compat-libs/compat-libs.go

internal/modifier/gated.go

elezar · 2025-02-10T13:44:11Z

internal/modifier/gated.go

+		compatLibHookDiscoverer := discover.NewCUDACompatHookDiscoverer(logger, cfg.NVIDIACTKConfig.Path, driver)
+		discoverers = append(discoverers, compatLibHookDiscoverer)
+		if cfg.NVIDIAContainerRuntimeConfig.Mode == "legacy" {
+			ldcacheIpdateHookDiscoverer, err := discover.NewLDCacheUpdateHook(


Note: Since the nvidia-container-runtime-hook is invoked as a prestart hook, this is done BEFORE the createContainer hook that we insert above. This means that we need to once again run the update ldcache hook.

@klueska one option would be to create the file in /etc/ld.so.conf.d when invoking the nvidia-container-runtime-hook instead -- before calling out to the nvidia-container-cli. We don't have as ready access to the driver version, but we could extract it there.

If we create the file in nvidia-container-runtime-hook, then wouldn't that remove the need to add the cuda-compat createContainer hook altogether? And as a result, the forward compatibility support in the legacy stack would not require users to use NVIDIA container Runtime, correct?

I did try it, and my initial assumption was incorrect. Because we invoke ldconfig as:

argv = (char * []){cnt->cfg.ldconfig, "-f", "/etc/ld.so.conf", "-C", "/etc/ld.so.cache", cnt->cfg.libs_dir, cnt->cfg.libs32_dir, NULL};

in libnvidia-container, this means that the cnt->cfg.libs_dir and cnt->cfg.libs32_dir folders take precedence over the files in /etc/ld.so.conf.d and the CUDA libraries present there are used.

We could rework the libnvidia-container implementation further, but the intent of this change is also to provide the functionality for #910 so that we can remove the legacy code path by default.

Got it. So for legacy mode, we are adding two createContainer hooks ( the cuda-compat hook and the update-ldcache) to ensure the compat libs are used, the caveat being that we execute ldconfig twice.

Yes, that is correct. Note that since #910 disables the legacy mode by default, the intent is that this particular configuration becomes more uncommon.

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go

cmd/nvidia-cdi-hook/cudacompat/cudacompat_test.go

internal/modifier/gated.go

ArangoGutierrez

Not blocking comments

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go

rajatchopra · 2025-02-25T23:51:12Z

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go

+
+	driverMajor := strings.SplitN(hostDriverVersion, ".", 2)[0]
+
+	if driverMajor >= compatMajor {


The string comparison shouldn't be done. There should be an Atoi before this.
Example: driverMajor = "11", compatMajor = "101"
The current code will show that driverMajor is greater than compatMajor, which is wrong. Can add a testcase too.

Yes, that's true. This relies on the current state of our driver versions where the branch (major) version is always some three digit number. This is also why string comparison is used in libnvidia-container at present, but is not future proof and will break when we release a 1xxx driver branch. I will update to use Atoi instead.

rajatchopra · 2025-02-25T23:56:47Z

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go

+
+	// Create the 'cuda-compat' command
+	c := cli.Command{
+		Name:  "cuda-compat",


Grammar review for fun.
If its a command, then it should be a verb. 'enable-cuda-compat'?

This change adds an nvidia-cdi-hook enable-cuda-compat hook that checks the container for cuda compat libs and updates /etc/ld.so.conf.d to include their parent folder if their driver major version is sufficient. This allows CUDA Forward Compatibility to be used when this is not available through the libnvidia-container. Signed-off-by: Evan Lezar <[email protected]>

This change adds the enable-cuda-compat hook to the incomming OCI runtime spec if the allow-cuda-compat-libs-from-container feature flag is not enabled. An update-ldcache hook is also injected to ensure that the required folders are processed. Signed-off-by: Evan Lezar <[email protected]>

Signed-off-by: Evan Lezar <[email protected]>

ssst0n3 · 2025-03-18T03:22:23Z

internal/config/features.go

+	//
+	// Note: Since this mechanism replaces the logic in the `nvidia-container-cli`,
+	// toggling this feature has no effect if `allow-cuda-compat-libs-from-container` is enabled.
+	DisableCUDACompatLibHook *feature `toml:"disable-cuda-compat-lib-hook,omitempty"`


This feature flag seems not used in the cdi mode. Is this expected?

ssst0n3 · 2025-03-18T03:25:37Z

pkg/nvcdi/driver-nvml.go

@@ -97,6 +97,8 @@ func NewDriverLibraryDiscoverer(logger logger.Interface, driver *root.Driver, nv
 		libraryPaths,
 	)

+	// TODO: The following should use the version directly.
+	cudaCompatLibHookDiscoverer := discover.NewCUDACompatHookDiscoverer(logger, nvidiaCDIHookPath, driver)


Should the disable-cuda-compat-lib-hook feature flag be used here?

AO696 approved these changes Feb 6, 2025

View reviewed changes

elezar force-pushed the add-compat-lib-hook branch from f21f7a6 to d617fb5 Compare February 6, 2025 22:24

elezar commented Feb 6, 2025

View reviewed changes

cmd/nvidia-cdi-hook/compat-libs/compat-libs.go Outdated Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch 3 times, most recently from 606aed5 to ba30397 Compare February 6, 2025 23:08

elezar requested review from klueska, tariq1890 and cdesiniotis February 7, 2025 08:41

elezar changed the title ~~Add compat lib hook~~ Add CUDA forward compatibility hook Feb 7, 2025

elezar force-pushed the add-compat-lib-hook branch 3 times, most recently from 0c4dd3d to 0b50458 Compare February 7, 2025 13:07

elezar marked this pull request as ready for review February 7, 2025 13:10

elezar added the must-backport The changes in PR need to be backported to at least one stable release branch. label Feb 7, 2025

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from d1e97a0 to eeef750 Compare February 7, 2025 21:08

cdesiniotis reviewed Feb 8, 2025

View reviewed changes

internal/modifier/gated.go Outdated Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from 88841fa to 8e0c69b Compare February 10, 2025 13:38

elezar commented Feb 10, 2025

View reviewed changes

elezar force-pushed the add-compat-lib-hook branch 3 times, most recently from 77bfb3c to 1140fcf Compare February 10, 2025 15:08

tariq1890 reviewed Feb 10, 2025

View reviewed changes

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go Outdated Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch from 1140fcf to cb6b4f6 Compare February 10, 2025 18:18

cdesiniotis reviewed Feb 10, 2025

View reviewed changes

internal/modifier/gated.go Outdated Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from f39a194 to aaf5ed8 Compare February 11, 2025 14:23

ArangoGutierrez reviewed Feb 25, 2025

View reviewed changes

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go Show resolved Hide resolved

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go Outdated Show resolved Hide resolved

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go Show resolved Hide resolved

ArangoGutierrez mentioned this pull request Feb 25, 2025

Create .so symlinks for driver libraries in container #326

Open

rajatchopra reviewed Feb 25, 2025

View reviewed changes

wqlparallel mentioned this pull request Feb 26, 2025

Does CDI mode support CUDA Forward Compatibility? #942

Open

elezar force-pushed the add-compat-lib-hook branch from 7ba4ea6 to e4b2ba1 Compare February 26, 2025 12:43

elezar modified the milestones: Disable legacy code path by default, v1.17.5 Feb 27, 2025

elezar self-assigned this Feb 27, 2025

klueska approved these changes Feb 27, 2025

View reviewed changes

ArangoGutierrez approved these changes Feb 27, 2025

View reviewed changes

elezar added 8 commits February 27, 2025 15:58

Add ldconfig hook in legacy mode

b7fbd56

Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Add basic CDI generate test

c9422f1

Signed-off-by: Evan Lezar <[email protected]>

Add enable-cuda-compat hook to CDI spec generation

70b1f5a

Signed-off-by: Evan Lezar <[email protected]>

Ensure that mode hook is executed last

2adef99

Signed-off-by: Evan Lezar <[email protected]>

Add basic integration tests for forward compat

011fb72

Signed-off-by: Evan Lezar <[email protected]>

Add disable-cuda-compat-lib-hook feature flag

aff9301

Signed-off-by: Evan Lezar <[email protected]>

elezar force-pushed the add-compat-lib-hook branch from 5ee65f9 to aff9301 Compare February 27, 2025 13:58

elezar merged commit 968e2cc into NVIDIA:main Feb 27, 2025
16 checks passed

elezar mentioned this pull request Feb 27, 2025

Add CUDA forward compatibility hook #948

Merged

ssst0n3 reviewed Mar 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CUDA forward compatibility hook #906

Add CUDA forward compatibility hook #906

elezar commented Feb 6, 2025 •

edited

Loading

elezar Feb 10, 2025

elezar Feb 10, 2025

cdesiniotis Feb 10, 2025 •

edited

Loading

elezar Feb 11, 2025 •

edited

Loading

cdesiniotis Feb 14, 2025

elezar Feb 17, 2025 •

edited

Loading

ArangoGutierrez left a comment

rajatchopra Feb 25, 2025

elezar Feb 26, 2025

rajatchopra Feb 25, 2025

ssst0n3 Mar 18, 2025

ssst0n3 Mar 18, 2025


		driverMajor := strings.SplitN(hostDriverVersion, ".", 2)[0]

		if driverMajor >= compatMajor {

Add CUDA forward compatibility hook #906

Add CUDA forward compatibility hook #906

Conversation

elezar commented Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdesiniotis Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

elezar Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elezar Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

ArangoGutierrez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elezar commented Feb 6, 2025 •

edited

Loading

cdesiniotis Feb 10, 2025 •

edited

Loading

elezar Feb 11, 2025 •

edited

Loading

elezar Feb 17, 2025 •

edited

Loading