CUDA, ROCm and Alpaka-related updates #40832

fwyzard · 2023-02-21T09:13:30Z

PR description:

CUDA, ROCm and Alpaka-related updates:

add PortableTestObjects data formats for the ROCm Alpaka backend
add support for AMD GPUs via the ProcessAcceleratorROCm
update the description of the "backend" parameter
introduce different exit status codes for cudaIsEnabled and rocmIsEnabled
add abstract interfaces for the CUDA and ROCm services
do not load the CUDA or ROCm services if the backends are not available
update the PhysicsTools/TensorFlow tests accordingly

PR validation:

All unit tests should pass.

Backport status

To be backported to CMSSW 13.0.x for data taking.

fwyzard · 2023-02-21T09:14:27Z

please test

cmsbuild · 2023-02-21T09:21:23Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40832/34287

This PR adds an extra 32KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/StandardSequences/python/Accelerators_cff.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File DataFormats/PortableTestObjects/src/alpaka/classes_rocm.h modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File DataFormats/PortableTestObjects/src/alpaka/classes_rocm_def.xml modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/AlpakaCore/python/ProcessAcceleratorAlpaka.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/AlpakaCore/src/module_backend_config.cc modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDACore/python/ProcessAcceleratorCUDA.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/interface/CUDAService.h modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/plugins/BuildFile.xml modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/plugins/CUDAServiceDisabled.cc modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/plugins/CUDAServiceEnabled.cc modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/src/CUDAService.cc modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725, New Heterogeneous Memory Pool #37952
- File HeterogeneousCore/CUDAServices/plugins/plugins.cc modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/test/BuildFile.xml modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/test/testCUDAService.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/test/testCUDAService.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/test/testCUDAServiceDisabled.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/test/testCUDAServiceDisabled.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAServices/test/test_main.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAUtilities/bin/cudaIsEnabled.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAUtilities/bin/cudaIsEnabled_fallback.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAUtilities/interface/EventCache.h modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/CUDAUtilities/interface/StreamCache.h modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/Common/interface/PlatformStatus.h modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/Common/python/PlatformStatus.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/ROCmCore/python/ProcessAcceleratorROCm.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/ROCmCore/python/ProcessAcceleratorROCm_cfi.py modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/ROCmUtilities/bin/rocmIsEnabled.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725
- File HeterogeneousCore/ROCmUtilities/bin/rocmIsEnabled_fallback.cpp modified in PR(s): CUDA, ROCm and Alpaka-related updates [13.0.x] #40725

cmsbuild · 2023-02-21T09:21:55Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

Configuration/StandardSequences (operations)
DataFormats/PortableTestObjects (heterogeneous)
HeterogeneousCore/AlpakaCore (heterogeneous)
HeterogeneousCore/CUDACore (heterogeneous)
HeterogeneousCore/CUDAServices (heterogeneous)
HeterogeneousCore/CUDAUtilities (heterogeneous)
HeterogeneousCore/Common (heterogeneous)
HeterogeneousCore/ROCmCore (heterogeneous)
HeterogeneousCore/ROCmUtilities (heterogeneous)
PhysicsTools/TensorFlow (reconstruction)

@perrotta, @rappoccio, @makortel, @clacaputo, @fwyzard, @mandrenguyen, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@fabiocos, @VourMa, @makortel, @felicepantaleo, @riga, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @mmusich, @sameasy, @mtosi, @dgulhan, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2023-02-21T10:54:10Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9dbeaa/30774/summary.html
COMMIT: 3ac1296
CMSSW: CMSSW_13_1_X_2023-02-20-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9dbeaa/30774/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9dbeaa/30774/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testHeterogeneousCoreCUDATestWriteRead had ERRORS
---> test testHeterogeneousCoreAlpakaTestModules had ERRORS
---> test testHeterogeneousCoreAlpakaTestWriteRead had ERRORS
---> test TestCUDATest had ERRORS
and more ...

RelVals

139.001

----- Begin Fatal Exception 21-Feb-2023 11:04:07 CET-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin CUDAService
Exception Message:
unable to load /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so because /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

11634.0

----- Begin Fatal Exception 21-Feb-2023 11:17:29 CET-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin CUDAService
Exception Message:
unable to load /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so because /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

11634.7

----- Begin Fatal Exception 21-Feb-2023 11:17:29 CET-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin CUDAService
Exception Message:
unable to load /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so because /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

139.002139.002_RunZeroBias2021/step2_RunZeroBias2021.log
139.001139.001_RunMinimumBias2021/step2_RunMinimumBias2021.log
139.004139.004_RunNoBPTX2021/step2_RunNoBPTX2021.log

Expand to see more relval errors ...

AddOn Tests

hlt_mc_GRun

----- Begin Fatal Exception 21-Feb-2023 11:05:38 CET-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin CUDAService
Exception Message:
unable to load /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so because /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

hlt_mc_GRun

----- Begin Fatal Exception 21-Feb-2023 11:06:26 CET-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin CUDAService
Exception Message:
unable to load /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so because /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

hlt_mc_HIon

----- Begin Fatal Exception 21-Feb-2023 11:05:33 CET-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin CUDAService
Exception Message:
unable to load /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so because /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40832/30774/CMSSW_13_1_X_2023-02-20-2300/lib/el8_amd64_gcc11/poisoned/plugin-poisoned-HeterogeneousCoreCUDAServicesPlugins.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

cmsbuild · 2023-02-28T14:59:18Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40832/34377

This PR adds an extra 12KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HeterogeneousCore/CUDAServices/src/CUDAService.cc modified in PR(s): New Heterogeneous Memory Pool #37952
- File RecoPixelVertexing/PixelTriplets/plugins/CAHitNtupletGeneratorOnGPU.cc modified in PR(s): New Heterogeneous Memory Pool #37952
- File RecoVertex/BeamSpotProducer/plugins/BeamSpotToCUDA.cc modified in PR(s): New Heterogeneous Memory Pool #37952

cmsbuild · 2023-02-28T15:00:01Z

Pull request #40832 was updated. @perrotta, @malbouis, @rappoccio, @yuanchao, @ChrisMisan, @clacaputo, @tvami, @AdrianoDee, @srimanob, @saumyaphor4252, @francescobrivio, @mandrenguyen, @fabiocos, @davidlange6 can you please check and sign again.

mandrenguyen · 2023-02-28T16:11:21Z

+1
resign

tvami · 2023-02-28T16:11:21Z

+alca

AdrianoDee · 2023-02-28T18:24:39Z

+upgrade
(upgrade-wise only changes to few headers)

cmsbuild · 2023-02-28T18:50:27Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9dbeaa/30954/summary.html
COMMIT: 1c30d1c
CMSSW: CMSSW_13_1_X_2023-02-27-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40832/30954/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 29 lines from the logs
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3529699
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3529671
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

rappoccio · 2023-02-28T20:57:23Z

+1

cmsbuild · 2023-02-28T20:57:58Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

cmsbuild added this to the CMSSW_13_1_X milestone Feb 21, 2023

fwyzard mentioned this pull request Feb 21, 2023

CUDA, ROCm and Alpaka-related updates [13.0.x] #40725

Merged

cmsbuild added reconstruction-pending operations-pending pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending labels Feb 21, 2023

cmsbuild added tests-started and removed tests-pending labels Feb 21, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 21, 2023

cmsbuild added tests-rejected and removed tests-started labels Feb 21, 2023

fwyzard added 7 commits February 23, 2023 16:31

Clean up HcalRawToDigiGPU headers

d872328

CUDAService is no longer required by PixelCPEFast

189eac5

Drop the deviceWithMostFreeMemory() methods

a636f4a

Add data formats for the ROCm Alpaka backend

6e5e357

Add support for AMD GPUs via the ProcessAcceleratorROCm

279fb84

Update the "backend" parameter description

c91a0dd

Use different exit codes for different conditions

a2fe732

fwyzard force-pushed the more_Alpaka_updates_131x branch from 3ac1296 to d9d14ff Compare February 23, 2023 15:58

cmsbuild added alca-pending and removed tests-rejected code-checks-approved labels Feb 23, 2023

cmsbuild added tests-started heterogeneous-approved labels Feb 28, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 28, 2023

cmsbuild added alca-approved reconstruction-approved and removed reconstruction-pending alca-pending labels Feb 28, 2023

cmsbuild added upgrade-approved and removed upgrade-pending labels Feb 28, 2023

cmsbuild added tests-approved and removed tests-started labels Feb 28, 2023

cmsbuild added operations-approved fully-signed orp-approved and removed operations-pending pending-signatures orp-pending labels Feb 28, 2023

cmsbuild merged commit 3d761d8 into cms-sw:master Feb 28, 2023

makortel mentioned this pull request Mar 2, 2023

Allow access to const edm::{Event,EventSetup} from device::{Event,EventSetup} #40849

Merged

fwyzard deleted the more_Alpaka_updates_131x branch March 9, 2023 21:48

grzanka mentioned this pull request Apr 4, 2023

PPS diamond mapping fix for 2023 run (backport) [13_0_X] #41188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA, ROCm and Alpaka-related updates #40832

CUDA, ROCm and Alpaka-related updates #40832

Uh oh!

fwyzard commented Feb 21, 2023 •

edited

Loading

Uh oh!

fwyzard commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

mandrenguyen commented Feb 28, 2023

Uh oh!

tvami commented Feb 28, 2023

Uh oh!

AdrianoDee commented Feb 28, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

rappoccio commented Feb 28, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

CUDA, ROCm and Alpaka-related updates #40832

CUDA, ROCm and Alpaka-related updates #40832

Uh oh!

Conversation

fwyzard commented Feb 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Backport status

Uh oh!

fwyzard commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 21, 2023

Uh oh!

cmsbuild commented Feb 21, 2023

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

mandrenguyen commented Feb 28, 2023

Uh oh!

tvami commented Feb 28, 2023

Uh oh!

AdrianoDee commented Feb 28, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Comparison Summary

Uh oh!

rappoccio commented Feb 28, 2023

Uh oh!

cmsbuild commented Feb 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

fwyzard commented Feb 21, 2023 •

edited

Loading