Unable to choose current device because CUDAService is disabled.

[silviodonato]

In CMSSW_11_3_X_2020-12-08-2300, we are getting in wf 136.885522, 136.888522, 10824.522, 11634.522

----- Begin Fatal Exception 09-Dec-2020 10:02:40 CET-----------------------
An exception of category 'CUDAError' occurred while
   [0] Processing  Event run: 320822 lumi: 40 event: 64112784 stream: 2
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module RecHitTask/'recHitPreRecoTask'
   [3] Prefetching for module HcalCPURecHitsProducer/'hbheprereco'
   [4] Prefetching for module HBHERecHitProducerGPU/'hbheRecHitProducerGPU'
   [5] Calling method for module HcalDigisProducerGPU/'hcalDigisGPU'
   [6] Calling cms::cuda::chooseDevice()
Exception Message:
Unable to choose current device because CUDAService is disabled. If CUDAService was not explicitly
disabled in the configuration, the probable cause is that there is no GPU or there is some problem
in the CUDA runtime or drivers.
----- End Fatal Exception -------------------------------------------------

It seems related to #31720 . It sounds like a kind of expected error due to missing GPU in the IB test machines.

[cmsbuild]

A new Issue was created by @silviodonato Silvio Donato.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[silviodonato]

assign heterogeneous, core

[cmsbuild]

New categories assigned: heterogeneous,core

@Dr15Jones,@smuzaffar,@makortel,@makortel,@fwyzard you have been requested to review this Pull request/Issue and eventually sign? Thanks

[fwyzard]

The workflows 136.885522, 136.888522, 10824.522, 11634.522 are explicitly running the HCAL reconstruction on GPU - so it looks like they are behaving as expected.

How would you like them to behave when there are no GPUs ?

[fwyzard]

Here are some options:

1. keep the GPU-only failing as they do now
2. implement a way to mark the GPU-only workflows as "expected failures"
3. implement a way to avoid running the GPU-only workflows
4. change the workflows to run on GPU if available, and fall back to CPU otherwise

[fwyzard]

I would guess that

• 2. requires changes to the testing and reporting infrastructures
• 3. could be done either in the test infrastructure, or in cmsDriver / runTheMatrix ?
• 4. is doable in CMSSW; the plan was to add an automatic GPU/CPU workflow, instead of replacing these ones, but we could also replace them instead

[makortel]

I think 4. would in principle be a viable alternative, but I also think we should find a way to run matrix workflows on GPU resources, and when that is done, 4. has a risk of not catching certain kind of problems because the "problematic workflow" could fall back to CPU and succeed.

Therefore I think we should look into 3., extending it to running GPU-only workflows on GPU resources (similarly to the "GPU Unit Tests" we have in IBs).

If solving 3. is not quick, it could be useful to do 2..

@smuzaffar, what do you think?

[makortel]

A possible straightforward way to implement 3. would be to define a separate -w set for GPU-only workflows. Then the GPU-only workflows would not be run by default, and it would be easy to run only them with runTheMatrix.py -w gpu.

[smuzaffar]

+1 for -w gpu

[silviodonato]

assign pdmv
because this change will affect runTheMatrix.py