Remove obsolete CUDA-using modules from DQM/SiPixelHeterogeneous #49697

makortel · 2025-12-22T17:06:08Z

PR description:

This PR removes CUDA-depending modules from DQM/SiPixelHeterogeneous. The inclusion of these modules in runTheMatrix workflow configurations came up in failures following CUDADataFormats dictionary removal in #49656 (comment). Since all direct CUDA components are slated for removal (#45844), this PR suggests to remove them. These components seem to have been superseded by more generic ones in #45206.

Resolves cms-sw/framework-team#1742

PR validation:

Workflows 11634.5 and 34434.5 succeeded.

These modules seem to have been supersed by more generic ones: - SiPixelCompareVertexSoA -> SiPixelCompareVertices - SiPixel*CompareRecHitsSoA -> SiPixel*CompareRecHits - SiPixel*CompareTrackSoA -> SiPixel*CompareTracks - SiPixel*MonitorRecHitsSoA -> SiPixel*MonitorRecHitsSoAAlpaka - SiPixel*MonitorTrackSoA -> SiPixel*MonitorTrackSoAAlpaka - SiPixelMonitorVertexSoA -> SiPixelMonitorVertexSoAAlpaka

cmsbuild · 2025-12-22T17:06:36Z

cms-bot internal usage

makortel · 2025-12-22T17:07:32Z

FYI @cms-sw/heterogeneous-l2

cmsbuild · 2025-12-22T17:07:47Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49697/47255

There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/SiPixelHeterogeneous/plugins/BuildFile.xml modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/plugins/SiPixelCompareRecHitsSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/plugins/SiPixelCompareTrackSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/plugins/SiPixelCompareVertexSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/plugins/SiPixelMonitorRecHitsSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/plugins/SiPixelMonitorTrackSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/plugins/SiPixelMonitorVertexSoA.cc modified in PR(s): Remove legacy CUDA modules for pixel track and vertex reconstruction #45853
- File DQM/SiPixelHeterogeneous/python/SiPixelHeterogenousDQM_FirstStep_cff.py modified in PR(s): Remove the configuration of the legacy CUDA workflows #46076

cmsbuild · 2025-12-22T17:07:54Z

A new Pull Request was created by @makortel for master.

It involves the following packages:

DQM/SiPixelHeterogeneous (dqm)

@cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@fioriNTU, @idebruyn, @jandrea, @mmusich, @threus this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

makortel · 2025-12-22T17:09:22Z

test parameters:

workflows = 136.8855,136.8885,11634.5,34434.5
enable = gpu

makortel · 2025-12-22T17:09:28Z

@cmsbuild, please test

makortel · 2025-12-22T17:10:37Z

DQM/SiPixelHeterogeneous/python/SiPixelHeterogenousDQM_FirstStep_cff.py


 # Run-3 sequence
-monitorpixelSoASource = cms.Sequence(siPixelPhase1MonitorRecHitsSoA * siPixelPhase1MonitorTrackSoA * siPixelMonitorVertexSoA)
+monitorpixelSoASource = cms.Sequence()


It is not clear to me if these empty sequences serve any purpose anymore other than being placeholders that are toReplaceWith() below with various modifiers.

makortel · 2025-12-22T17:11:08Z

DQM/SiPixelHeterogeneous/python/SiPixelHeterogenousDQM_FirstStep_cff.py

 monitorpixelSoACompareSource = cms.Sequence(siPixelPhase1MonitorRawDataACPU *
                                            siPixelPhase1MonitorRawDataAGPU *
-                                            siPixelPhase1MonitorRecHitsSoACPU *
-                                            siPixelPhase1MonitorRecHitsSoAGPU *
-                                            siPixelPhase1CompareRecHitsSoA *
-                                            siPixelPhase1MonitorTrackSoAGPU *
-                                            siPixelPhase1MonitorTrackSoACPU *
-                                            siPixelPhase1CompareTrackSoA *
-                                            siPixelMonitorVertexSoACPU *
-                                            siPixelMonitorVertexSoAGPU *
-                                            siPixelCompareVertexSoA *
                                            siPixelPhase1RawDataErrorComparator)


It is not clear to me if the remaining 3 modules in this Sequence would be useful, or if it would be better to remove them as well.

cmsbuild · 2025-12-22T19:11:07Z

-1

Failed Tests: RelVals-NVIDIA_T4
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-046ac5/50392/summary.html
COMMIT: d99a3f5
CMSSW: CMSSW_16_1_X_2025-12-22-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49697/50392/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-NVIDIA_T4

ValueError: Undefined workflows: 29834.751, 29834.404, 29834.402, 29834.704, 29834.403

Comparison Summary

There are some workflows for which there are errors in the baseline:
11634.5 step 3
136.8855 step 3
136.8885 step 3
34434.5 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 99 lines to the logs
Reco comparison results: 12 differences found in the comparisons
Reco comparison had 4 failed jobs
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4280553
DQMHistoTests: Total failures: 15
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4280518
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 239 log files, 204 edm output root files, 53 DQM output files
TriggerResults: no differences found

makortel · 2025-12-22T19:24:16Z

ValueError: Undefined workflows: 29834.751, 29834.404, 29834.402, 29834.704, 29834.403

I wonder what this means in practice

makortel · 2025-12-22T19:26:27Z

ValueError: Undefined workflows: 29834.751, 29834.404, 29834.402, 29834.704, 29834.403

I wonder what this means in practice

I opened an issue #49700

makortel · 2025-12-29T14:44:46Z

@cmsbuild, please test

cms-sw/cms-bot#2636 was merged

makortel · 2025-12-29T23:34:22Z

CPU comparison differences are related to #47071

-1

In RelVal-INPUT tests the workflow the input file for 2025.0020001_RunEGamma02025D_10k is unreachable (already in the IBs)

On NVIDIA H100 the runTheMatrix tests fail with

----- Begin Fatal Exception 29-Dec-2025 17:06:58 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'MC_Ele5_Open_Unseeded'
   [2] Calling method for module HGCalSoARecHitsLayerClustersProducer@alpaka/'hltHgcalSoARecHitsLayerClustersProducer'
Exception Message:
A std::exception was thrown.
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.0.0-8493f1d11d0378dc14d6ea6ecfc69ac5/include/alpaka/mem/buf/uniformCudaHip/traits/BufUniformCudaHipRtTraits.hpp(283) 'TApi::mallocAsync(&memPtr, static_cast<std::size_t>(width) * sizeof(TElem), queue.getNativeHandle())' returned error  : 'cudaErrorNotSupported': 'operation not supported'!
----- End Fatal Exception -------------------------------------------------

that could be an infrastructure problem.

makortel · 2025-12-29T23:34:39Z

ignore tests-rejected with external-failure

smuzaffar · 2026-01-05T14:03:13Z

please test

cmsbuild · 2026-01-06T10:00:11Z

-1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-046ac5/50433/summary.html
COMMIT: d99a3f5
CMSSW: CMSSW_16_1_X_2026-01-05-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49697/50433/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
11634.5 step 3
136.8855 step 3
136.8885 step 3
34434.5 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 100 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 4 differences found in the comparisons
Reco comparison had 4 failed jobs
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4280553
DQMHistoTests: Total failures: 72
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4280461
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 239 log files, 204 edm output root files, 53 DQM output files
TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 247 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 11
DQMHistoTests: Total histograms compared: 149371
DQMHistoTests: Total failures: 27569
DQMHistoTests: Total nulls: 8
DQMHistoTests: Total successes: 121794
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
Checked 42 log files, 45 edm output root files, 11 DQM output files
TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 236 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 11
DQMHistoTests: Total histograms compared: 149371
DQMHistoTests: Total failures: 36928
DQMHistoTests: Total nulls: 9
DQMHistoTests: Total successes: 112434
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
Checked 42 log files, 45 edm output root files, 11 DQM output files
TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

You potentially removed 10 lines from the logs
Reco comparison results: 222 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 11
DQMHistoTests: Total histograms compared: 149371
DQMHistoTests: Total failures: 35529
DQMHistoTests: Total nulls: 8
DQMHistoTests: Total successes: 113834
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
Checked 42 log files, 45 edm output root files, 11 DQM output files
TriggerResults: no differences found

makortel · 2026-01-06T14:29:57Z

-1

Several of the 34634.X workflows continue to fail on NVIDIA H100 as in #49697 (comment). Maybe it's not an infrastructure problem (since other workflows succeed), but something else that would be better investigated separately from this PR.

makortel · 2026-01-06T14:31:24Z

-1

Several of the 34634.X workflows continue to fail on NVIDIA H100 as in #49697 (comment). Maybe it's not an infrastructure problem (since other workflows succeed), but something else that would be better investigated separately from this PR.

Those workflows succeed in IBs though

makortel · 2026-01-06T14:34:31Z

@cms-sw/dqm-l2 Could you please review and sign? This PR resolves the IB failures in the 4 workflows listed in #49697 (comment).

gabrielmscampos · 2026-01-06T15:02:17Z

+dqm

Regarding your comment on the monitorpixelSoACompareSource modules, I'll look for a SiPixel contact to take a look into it. Since I don't think it is urgent and this fixes the IB failures listed, I'm signing it right away.

cmsbuild · 2026-01-06T15:02:40Z

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

fwyzard · 2026-01-06T15:33:25Z

@cms-sw/tracking-pog-l2 can you review the DQM sequences highlighted by @makortel comments ?

mandrenguyen · 2026-01-07T21:34:24Z

+1

makortel mentioned this pull request Dec 22, 2025

Remove most dictionaries of CUDADataFormats #49656

Merged

cmsbuild added this to the CMSSW_16_1_X milestone Dec 22, 2025

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending trk labels Dec 22, 2025

makortel mentioned this pull request Dec 22, 2025

Drop support for native CUDA code #45844

Open

52 tasks

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 22, 2025

cmsbuild added tests-started and removed tests-pending labels Dec 22, 2025

makortel commented Dec 22, 2025

View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Dec 22, 2025

makortel mentioned this pull request Dec 22, 2025

GPU PR tests failing with "Undefined workflows" #49700

Closed

cmsbuild removed the tests-rejected label Dec 29, 2025

cmsbuild added the tests-rejected label Dec 29, 2025

cmsbuild added tests-approved tests-external-failure and removed tests-rejected labels Dec 29, 2025

cmsbuild added tests-started and removed tests-approved tests-external-failure labels Jan 5, 2026

cmsbuild added tests-approved tests-external-failure and removed tests-started labels Jan 6, 2026

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jan 6, 2026

cmsbuild added orp-approved and removed orp-pending labels Jan 7, 2026

cmsbuild merged commit d82850b into cms-sw:master Jan 7, 2026
24 of 25 checks passed

makortel deleted the cudaDQMRemove branch January 7, 2026 21:36

mmusich mentioned this pull request Jan 26, 2026

Clean-up DQM/SiPixelHeterogeneous plugins #49936

Merged

Remove obsolete CUDA-using modules from DQM/SiPixelHeterogeneous #49697

Remove obsolete CUDA-using modules from DQM/SiPixelHeterogeneous #49697

Uh oh!

Conversation

makortel commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Uh oh!

cmsbuild commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

makortel commented Dec 22, 2025

Uh oh!

cmsbuild commented Dec 22, 2025

Uh oh!

cmsbuild commented Dec 22, 2025

Uh oh!

makortel commented Dec 22, 2025

Uh oh!

makortel commented Dec 22, 2025

Uh oh!

makortel Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

makortel Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Dec 22, 2025

Failed RelVals-NVIDIA_T4

Comparison Summary

Uh oh!

makortel commented Dec 22, 2025

Uh oh!

makortel commented Dec 22, 2025

Uh oh!

makortel commented Dec 29, 2025

Uh oh!

makortel commented Dec 29, 2025

Uh oh!

makortel commented Dec 29, 2025

Uh oh!

smuzaffar commented Jan 5, 2026

Uh oh!

cmsbuild commented Jan 6, 2026

Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Uh oh!

makortel commented Jan 6, 2026

Uh oh!

makortel commented Jan 6, 2026

Uh oh!

makortel commented Jan 6, 2026

Uh oh!

gabrielmscampos commented Jan 6, 2026

Uh oh!

cmsbuild commented Jan 6, 2026

Uh oh!

fwyzard commented Jan 6, 2026

Uh oh!

mandrenguyen commented Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

makortel commented Dec 22, 2025 •

edited

Loading

cmsbuild commented Dec 22, 2025 •

edited

Loading