[GPU] Multiple RelVals failing with memory allocation error

[aandvalenzuela]

Hello,

There are multiple RelVals failing with the following exception in GPU IBs:

----- Begin Fatal Exception 05-Feb-2024 04:19:50 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 366727 lumi: 89 event: 131642946 stream: 3
   [1] Running path 'MC_Run3_PFScoutingPixelTracking_v22'
   [2] Calling method for module HBHERecHitProducerGPU/'hltHbherecoGPU'
Exception Message:
A std::exception was thrown.

/data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5569e690981e3c5d49d7743adaadedca/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_GPU_X_2024-02-04-2300/src/HeterogeneousCore/CUDAUtilities/src/CachingDeviceAllocator.h, line 489:
cudaCheck(error = cudaMalloc(&search_key.d_ptr, search_key.bytes));
cudaErrorMemoryAllocation: out of memory
----- End Fatal Exception -------------------------------------------------

It seems caused by modifications in #43804.

FYI, @iarspider

Thanks,
Andrea

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @aandvalenzuela Andrea Valenzuela.

@Dr15Jones, @sextonkennedy, @rappoccio, @makortel, @antoniovilela, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[VinInn]

do not see any mechanism that makes #43804 interfering with Relvals...

[mmusich]

do not see any mechanism that makes #43804 interfering with Relvals...

I think these problems started to appear in CMSSW_14_0_X_2024-01-30-2300.
And in that IB there were a few updates related to the HLT menu:

• #43758 from @mmasciov: Add non-diagonal errors to scouting vertices
• #43788 from @cms-tsg-storm: HLT menu development for 13_3_X (1/N) [14_0_X]
• #43294 from @PixelTracksAlpaka: Pixel Alpaka Migration: Configs and Fixes [VII]

There are also different exceptions observed e.g.:

----- Begin Fatal Exception 31-Jan-2024 04:02:11 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 2 event: 101 stream: 3
   [1] Running path 'HLT_DoubleMediumDeepTauPFTauHPS30_L2NN_eta2p1_OneProng_M5to80_v2'
   [2] Calling method for module CAHitNtupletCUDAPhase1/'hltPixelTracksGPU'
Exception Message:
A std::exception was thrown.

/data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/bc88fe327b7ccd90d4bda9e20e6ec926/opt/cmssw/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_GPU_X_2024-01-30-2300/src/RecoTracker/PixelSeeding/plugins/CAHitNtupletGeneratorKernels.cu, line 64:
cudaCheck(cudaGetLastError());
cudaErrorMemoryAllocation: out of memory
----- End Fatal Exception -------------------------------------------------

but they all seem to be related to cudaErrorMemoryAllocation: out of memory.

[makortel]

assign heterogeneous, hlt

[cmsbuild]

New categories assigned: heterogeneous,hlt

@Martin-Grunewald,@mmusich,@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

I think these problems started to appear in CMSSW_14_0_X_2024-01-30-2300.

Indeed

• CMSSW_14_0_X_2024-01-29-2300 showed no failures
• CMSSW_14_0_X_2024-01-30-2300 showed 6 failures
• CMSSW_14_0_X_2024-01-31-2300 showed no failures
• CMSSW_14_0_X_2024-02-02-2300 showed 5 failures
• CMSSW_14_0_X_2024-02-04-2300 showed 10 failures

(although is difficult to say what happened before 01-29)

[slava77]

type tracking
(even though the association is not strong; it does look like related to the pixel tracking Alpaka migration)

[makortel]

@AdrianoDee @fwyzard Do I understand correctly that the workflows added in #43294 (.402, .403, .404) are not run (yet?) in GPU IBs?

Did #43788 change anything wrt. GPU modules (like more instances)? (on a cursory look I didn't catch any, but could have easily missed something subtle)