[Java] ONNXRuntime crashes in Karaf context due to unexpected GPU enumeration via nvdlistx.dll despite CPU-only configuration

[ChristianMueller2]

Problem:
When initializing an ONNX model using the Java API inside an Apache Karaf context, the JVM crashes with a FAIL_FAST_INVALID_ARG_c0000409 exception. The crash trace points to nvdlistx.dll, indicating that GPU enumeration via DirectX (D3D12) is being triggered — even though the session is explicitly configured for CPU-only execution using:

        ortEnvironment = OrtEnvironment.getEnvironment(OrtLoggingLevel.ORT_LOGGING_LEVEL_VERBOSE);
	OrtSession.SessionOptions sessionOptions = new OrtSession.SessionOptions(); // corrected instantiation
	sessionOptions.setOptimizationLevel(OrtSession.SessionOptions.OptLevel.ALL_OPT);
	sessionOptions.addCPU(true);
	ortSession = ortEnvironment.createSession(modelPath, sessionOptions);

This issue does not occur when running the same code in a standalone unit test outside of Karaf.

Is there a way to fully disable GPU probing at runtime from the Java API?

PROCESS_NAME: java.exe

EXCEPTION_CODE_STR: c0000409

EXCEPTION_PARAMETER1: 0000000000000005

STACK_TEXT:
000000b5150f56e0 00007ffa1448642f : 0000000000000000 0000000000000000 0000000000000000 000000000000000c : nvdlistx!nvCoprocThunk+0x11db0
000000b5150f5710 00007ffa1448631a : 0000000000000022 000000b5150f57c0 0000000000000104 00007ffa14478f0b : nvdlistx!nvCoprocThunk+0x11d77
000000b5150f5750 00007ffa14486449 : 0000000000000022 00007ffa14486575 0000000000000000 0000000000000104 : nvdlistx!nvCoprocThunk+0x11c62
000000b5150f57d0 00007ffa14483dcb : 000001df0000001c 000001df325fec98 000001df00000000 000000b500000053 : nvdlistx!nvCoprocThunk+0x11d91
000000b5150f5810 00007ffa1447507a : 0000000000000000 000000b5150f5880 0000000010f9dc80 0000000000000000 : nvdlistx!nvCoprocThunk+0xf713
000000b5150f5840 00007ffa144751b8 : 000001df325fec98 00007ffa144a2d98 00007ffa00000200 0000000000000090 : nvdlistx!nvCoprocThunk+0x9c2
000000b5150f58a0 00007ffa14474bad : 000001df4ddac398 000001df4ddac370 0000000000000000 000001df4ddac398 : nvdlistx!nvCoprocThunk+0xb00
000000b5150f58d0 00007ffa14474432 : 000001df4ddac398 000001df4ddac370 0000000000000000 000000000000585d : nvdlistx!nvCoprocThunk+0x4f5
000000b5150f5900 00007ffa14472451 : 0000000000000000 00007ffa1449bb00 0000000000000104 000000b5150f6fe5 : nvdlistx!QueryMuxDListForApplication+0x282a
000000b5150f5c70 00007ffa1447124d : 00007ffa1449bb08 000001df0659f0d0 0000000000000001 0000000000000000 : nvdlistx!QueryMuxDListForApplication+0x849
000000b5150f9560 00007ffa1447b4e3 : 0000000000000001 0000000000000000 0000000000000000 0000000000000001 : nvdlistx+0x124d
000000b5150f9c40 00007ffa6a478bcf : 00007ffa14470000 0000000000000001 0000000000000000 000000007ffe0385 : nvdlistx!nvCoprocThunk+0x6e2b
000000b5150f9ca0 00007ffa6a4bd67d : 000001df472011e0 00007ffa14470000 0000000000000001 0000000000000000 : ntdll!LdrpCallInitRoutine+0x6b
000000b5150f9d10 00007ffa6a4bd42e : 000001df48bdf2c0 00007ffa6a47ed00 000001df00000000 000001df47201228 : ntdll!LdrpInitializeNode+0x1c9
000000b5150f9e60 00007ffa6a47db5d : 0000000000000000 0000000000000000 000000b5150fa080 000000b5150f9fd8 : ntdll!LdrpInitializeGraphRecurse+0x42
000000b5150f9ea0 00007ffa6a478e70 : 000000b5150f9fd8 0000000000000000 0000000000000000 0000000000000600 : ntdll!LdrpPrepareModuleForExecution+0xc5
000000b5150f9ee0 00007ffa6a4690dc : 000000b5150f9fe0 000000b5150fa180 000000b500000600 000000000177fa00 : ntdll!LdrpLoadDllInternal+0x20c
000000b5150f9f80 00007ffa6a47a78a : 0000000000000000 0000000000000001 0000000000000000 000000b5150fa749 : ntdll!LdrpLoadDll+0xb0
000000b5150fa140 00007ffa6756b2a2 : 000000b5150fa270 0000000000000000 0000000000000003 0000000000000000 : ntdll!LdrLoadDll+0xfa
000000b5150fa230 00007ffa5973f764 : 0065006300000000 000001df31dfda70 0000000000000001 0000000000000000 : KERNELBASE!LoadLibraryExW+0x172
000000b5150fa2a0 00007ffa5973f019 : 00000000400000c0 000000000001b54f 00007ffa612d9680 00007ffa612a3a40 : directxdatabasehelper!QueryIHVDListAnswer+0x268
000000b5150fa5c0 00007ffa6128301b : 000001df31dfda70 000001df00000001 00007ffa612d9680 00007ffa612a3a40 : directxdatabasehelper!QueryFinalGPUPreferenceDecision+0x4f9
000000b5150fa680 00007ffa61283cea : 000001df00000002 0000000000000000 000001df2e541838 000000000001b129 : dxgi!CAdjustEnumForHybrid::AdjustEnumForHybrid+0x2cb
000000b5150fa7b0 00007ffa61280804 : 0000000000000000 00007ffa68034c50 0000000000000140 0000000000000140 : dxgi!CDXGIFactory::SampleAdaptersWorker+0x62a
000000b5150faf40 00007ffa6127ff5a : 0000000000000000 000001df2e541720 000000b5150fb470 000001df2e541720 : dxgi!CDXGIFactory::SampleAdapters+0x104
000000b5150fb210 00007ffa6127f86b : 000000b5150fb3f0 000001df2e541720 000001df2e541720 000001df2e541720 : dxgi!CDXGIFactory::Initialize+0xfa
000000b5150fb380 00007ffa6128f494 : 000001df49dab510 0000000000000000 00007ff9a21b7480 0000000000000005 : dxgi!CreateDXGIFactoryImpl+0x123
000000b5150fb430 00007ff9a1dd2610 : 000001df49dab510 0000000000000000 0000000000000000 0000000000000000 : dxgi!CreateDXGIFactoryActualImpl2+0x54
000000b5150fb460 00007ff9a1dd3eb0 : 0000000000000003 000001df49dab510 0000000000000000 000001df49dab510 : onnxruntime!onnxruntime::anonymous namespace'::GetDeviceInfoD3D12+0x90 000000b5150fb710 00007ff9a1db158a : 0000000000000000 0000000000000000 000000b5150fc270 00007ffa67c43ee6 : onnxruntime!onnxruntime::DeviceDiscovery::DiscoverDevicesForPlatform+0x190 000000b5150fb9d0 00007ff9a1db14dc : 0000000000000000 0000000000000000 000000b5150fc270 00007ffa67bc1966 : onnxruntime!onnxruntime::DeviceDiscovery::GetDevices'::2'::<lambda_1>::operator()+0x7a 000000b5150fbea0 00007ff9a15b4c14 : 000001df3318b9b0 0000000000000000 000000b5150fc020 00007ff9a2014df2 : onnxruntime!onnxruntime::DeviceDiscovery::GetDevices+0x6c 000000b5150fbed0 00007ff9a15b519a : 000001df3318b990 000000b5150fc020 0000000000000000 000001df3318b990 : onnxruntime!onnxruntime::anonymous namespace'::SortDevicesByType+0x44
000000b5150fbf20 00007ff9a15b339f : 0000000000000002 00007ff9a1627b14 000000b5150fc318 000001df00000000 : onnxruntime!onnxruntime::Environment::EpInfo::Create+0x3fa
000000b5150fc190 00007ff9a15b3c94 : 000001df31adcbc8 000000b5150fc7f0 000000000000007a 0000000000000000 : onnxruntime!onnxruntime::Environment::RegisterExecutionProviderLibrary+0x18f
000000b5150fc2c0 00007ff9a15b0f7b : 000000b5150fc7f0 000000b5150fc7f0 000000b5150fc7f0 00007ff9a15a5780 : onnxruntime!onnxruntime::Environment::CreateAndRegisterInternalEps+0x114
000000b5150fc3b0 00007ff9a15b02b5 : 000001df4bb64de0 000000b500000000 000001df4bb64f08 00007ffa67bc1966 : onnxruntime!onnxruntime::Environment::Initialize+0x33b
000000b5150fcb30 00007ff9a1585fa9 : 000001df49adf930 000000b5150fcd30 0000000000000000 0000000000000000 : onnxruntime!onnxruntime::Environment::Create+0xc5
(Inline Function) ---------------- : ---------------- ---------------- ---------------- ---------------- : onnxruntime!std::unique_ptr<onnxruntime::common::Status::State,std::default_delete<onnxruntime::common::Status::State> >::operator bool+0x5 (Inline Function) ---------------- : ---------------- ---------------- ---------------- ---------------- : onnxruntime!std::operator==+0x5
(Inline Function) ---------------- : ---------------- ---------------- ---------------- ---------------- : onnxruntime!onnxruntime::common::Status::IsOK+0x5 000000b5150fcbb0 00007ff9a1589e34 : 000001df0000000b 0000000000019577 00000000f93c3820 000000b5150fcd30 : onnxruntime!OrtEnv::GetInstance+0x219 000000b5150fcc90 00007ffa1eb45961 : 000001df2b485330 000001df4807f6c8 000001df4807f410 000001df4807f410 : onnxruntime!OrtApis::CreateEnv+0x44 000000b5150fcd10 000001df178c13a6 : 0000000000000158 000001df178c0cfc 000001df53f13330 0000000000000000 : onnxruntime4j_jni!Java_ai_onnxruntime_OrtEnvironment_createHandle__JILjava_lang_String_2+0x61 000000b5150fcd80 0000000000000158 : 000001df178c0cfc 000001df53f13330 0000000000000000 000000b5150fce28 : 0x000001df178c13a
000000b5150fcd88 000001df178c0cfc : 000001df53f13330 0000000000000000 000000b5150fce28 000001df178bf930 : 0x158
000000b5150fcd90 000001df53f13330 : 0000000000000000 000000b5150fce28 000001df178bf930 000001df53f107a8 : 0x000001df178c0cfc 000000b5150fcd98 0000000000000000 : 000000b5150fce28 000001df178bf930 000001df53f107a8 000001df178c0cb6 : 0x000001df53f13330

SYMBOL_NAME: nvdlistx+16468

MODULE_NAME: nvdlistx

IMAGE_NAME: nvdlistx.dll

STACK_COMMAND: ~320s; .ecxr ; kb

BUCKET_ID_MODPRIVATE: 1

FAILURE_BUCKET_ID: FAIL_FAST_INVALID_ARG_c0000409_nvdlistx.dll!Unknown

OS_VERSION: 10.0.22621.1

BUILDLAB_STR: ni_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

IMAGE_VERSION: 32.0.15.8092

FAILURE_ID_HASH: {cf9eecd2-0ac0-9895-14d8-c525a9611fca}

Followup: MachineOwner

[Craigacp]

Which artifact is being used onnxruntime-gpu or onnxruntime?

Also are there multiple applications using ONNX Runtime inside the same Karaf instance?

[Craigacp]

The EP discovery infrastructure is new in 1.22 and 1.23, looks like it's going wrong for some reason in your environment. I'm not familiar with Karaf, are you running it on the same computer, and if so is Karaf limiting the OS access in some way?

[ChristianMueller2]

I am running it on the same computer. The unit tests are working on this computer.
If karaf is doing so I do not know how.

[ChristianMueller2]

Just tested the version 1.21.1. That works!

[Craigacp]

Ok. I'm not sure what goes on in the C library to make the EP discovery happen, but maybe @skottmckay can help diagnose this issue?

[skottmckay]

With plugin EPs any EP (CPU/GPU/NPU) could be added at runtime, so the device discovery code that puts together a list of potential hardware devices that an EP could use is checking for GPU devices using D3D12 and DXCore infrastructure.

https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/platform/windows/device_discovery.cc

It only queries for info though. It's not directly attempting to use the GPU device.

I'm not familiar with Karaf or why it would break the CreateDXGIFactory2 call here:

onnxruntime/onnxruntime/core/platform/windows/device_discovery.cc

Line 336 in edfb6f5

if (FAILED(CreateDXGIFactory2(0, IID_PPV_ARGS(&factory)))) {