Update CUDA to version 12.9.0 and cuDNN to version 9.9.0 #9856

fwyzard · 2025-05-13T09:53:32Z

Drop support for the NVTX2 library.

Update PyTorch and its extensions to find the NVTX3 headers.

fwyzard · 2025-05-13T09:53:47Z

please test for el8_aarch64_gcc12

cmsbuild · 2025-05-13T09:53:56Z

A new Pull Request was created by @fwyzard for branch IB/CMSSW_15_1_X/master.

@iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2025-05-13T09:53:57Z

cms-bot internal usage

fwyzard · 2025-05-13T10:07:19Z

please test for el8_amd64_gcc14

fwyzard · 2025-05-13T10:07:49Z

please test for el10_amd64_gcc14

fwyzard · 2025-05-13T10:08:18Z

please test

fwyzard · 2025-05-13T10:08:39Z

GPU tests already passed in #9833 .

cmsbuild · 2025-05-13T10:20:04Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46080/summary.html
COMMIT: 9f730e3
CMSSW: CMSSW_15_1_X_2025-05-08-2300/el10_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9856/46080/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Requested to quit.
Requested to quit.
* The action "build-external+rdma-core+57.0-b28ed27528b25e00c98373c700e92f5f" was not completed successfully because Failed to build rdma-core. Log file in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el10_amd64_gcc14/external/rdma-core/57.0-b28ed27528b25e00c98373c700e92f5f/log. Final lines of the log file:
warning: Macro expanded in comment on line 404: %{pkginstroot}/lib64

error: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/SPECS/external/rdma-core/57.0-b28ed27528b25e00c98373c700e92f5f/spec: line 418: Patch number not specified: patch -p1
0<  (%patch)

* The action "install-external+rdma-core+57.0-b28ed27528b25e00c98373c700e92f5f" was not completed successfully because The following dependencies could not complete:
build-external+rdma-core+57.0-b28ed27528b25e00c98373c700e92f5f
* The action "download-upload-store-external+rdma-core+57.0-b28ed27528b25e00c98373c700e92f5f-1-1.el10_amd64_gcc14.rpm" was not completed successfully because The following dependencies could not complete:

cmsbuild · 2025-05-13T13:01:11Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46077/summary.html
COMMIT: 9f730e3
CMSSW: CMSSW_15_1_X_2025-05-12-2300/el8_aarch64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9856/46077/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

external/onnxruntime_external_deps.cmake:550 (include)
CMakeLists.txt:614 (include)


-- Configuring incomplete, errors occurred!
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.4kd5xI (%build)


RPM build errors:
Macro expanded in comment on line 404: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

cmsbuild · 2025-05-13T14:39:06Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46081/summary.html
COMMIT: 9f730e3
CMSSW: CMSSW_15_1_X_2025-05-13-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9856/46081/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 14 lines to the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4038163
DQMHistoTests: Total failures: 41
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4038102
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-05-13T15:14:08Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46079/summary.html
COMMIT: 9f730e3
CMSSW: CMSSW_15_1_X_2025-05-12-1100/el8_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9856/46079/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

external/onnxruntime_external_deps.cmake:550 (include)
CMakeLists.txt:614 (include)


-- Configuring incomplete, errors occurred!
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.9ehWHx (%build)

RPM build warnings:
Macro expanded in comment on line 404: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

fwyzard · 2025-05-14T09:08:52Z

The onnxruntime failure should be addressed by #9863 .

cmsbuild · 2025-05-14T16:09:38Z

Pull request #9856 was updated.

fwyzard · 2025-05-14T16:11:49Z

please test

cmsbuild · 2025-05-14T16:11:58Z

Pull request #9856 was updated.

cmsbuild · 2025-05-16T11:03:12Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46190/summary.html
COMMIT: d19948e
CMSSW: CMSSW_15_1_X_2025-05-08-2300/el10_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9856/46190/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46190/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46190/git-merge-result

Unit Tests

I found 4 errors in the following unit tests:

---> test test-das-selected-lumis had ERRORS
---> test test_edmPickEvents had ERRORS
---> test testAOTTools had ERRORS
and more ...

iarspider · 2025-05-16T11:11:52Z

@fwyzard FYI - this PR has also reached a limit on number of commit statuses.
@smuzaffar can we make bot post a comment if we reach commit status limit + add a label that bot would detect at start and not waste too many api calls on such PR?

fwyzard · 2025-05-16T11:14:16Z

by the way, what does it mean "number of commit statuses" exactly ?

fwyzard · 2025-05-16T11:15:29Z

what I mean is, how many "commit statuses" does a result like #9856 (comment) take ?

Drop support for nvtx2 in favour of nvtx3, which is header-only.

…tub libraries

fwyzard · 2025-05-16T11:16:45Z

Rebased to include the latest updates.

cmsbuild · 2025-05-16T11:16:55Z

Pull request #9856 was updated.

fwyzard · 2025-05-16T11:17:01Z

please test for el10_amd64_gcc14

smuzaffar · 2025-05-16T12:23:47Z

by the way, what does it mean "number of commit statuses" exactly ?

@fwyzard , the Pr checks you see at the bottom of PR are commit statuses. e.g. cms/9856/el10_amd64_gcc14/optional is one staus which might have been update multiple times. bot update each status multiple times during PR tests . The combined commit status limit is 1000 per commit. So with many tests and many archs we can reach this limit. We are looking in to other solutions ( e.g. storing the status somewhere else and then update once all tests are done) to avoid this

smuzaffar · 2025-05-16T12:25:15Z

@smuzaffar can we make bot post a comment if we reach commit status limit + add a label that bot would detect at start and not waste too many api calls on such PR?

No, bot might not have rights to push to user repo/branch. We should work on storing commit status at cmssdt server

fwyzard · 2025-05-16T12:25:38Z

No problem.
I'm not complaining, just trying to understand how quickly the tests would reach the limit.

fwyzard · 2025-05-16T12:26:28Z

No, bot might not have rights to push to user repo/branch. We should work on storing commit status at cmssdt server

But how about posting a comment on the GitHub PR ?

smuzaffar · 2025-05-16T12:28:30Z

But how about posting a comment on the GitHub PR ?

yes that bot can do

cmsbuild · 2025-05-16T20:43:40Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46213/summary.html
COMMIT: 8801d64
CMSSW: CMSSW_15_1_X_2025-05-08-2300/el10_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9856/46213/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46213/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-972068/46213/git-merge-result

Unit Tests

I found 4 errors in the following unit tests:

---> test test-das-selected-lumis had ERRORS
---> test test_edmPickEvents had ERRORS
---> test testAOTTools had ERRORS
and more ...

Comparison Summary

Summary:

You potentially added 1131 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 101635 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4015624
DQMHistoTests: Total failures: 562507
DQMHistoTests: Total nulls: 506
DQMHistoTests: Total successes: 3452591
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 432.935 KiB( 49 files compared)
DQMHistoSizes: changed ( 13034.0 ): 0.879 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 139.001,... ): 0.001 KiB HLT/Filters
DQMHistoSizes: changed ( 140.045,... ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.042 ): 0.035 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 145.014,... ): 6.954 KiB HLT/Filters
DQMHistoSizes: changed ( 145.014,... ): 4.246 KiB L1TEMU/L1TdeStage2EMTF
DQMHistoSizes: changed ( 145.014,... ): 0.492 KiB L1T/L1TStage2uGT
DQMHistoSizes: changed ( 145.408 ): -0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 145.5 ): 0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 145.604 ): 0.152 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 145.713 ): ...
Checked 215 log files, 184 edm output root files, 50 DQM output files

smuzaffar · 2025-05-17T07:23:56Z

merging this as 15.1.0.pre3 and FASTPU flavor has been uploaded

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels May 13, 2025

fwyzard changed the title ~~[do not merge] Update CUDA to version 12.9.0 and cuDNN to version 9.9.0~~ Update CUDA to version 12.9.0 and cuDNN to version 9.9.0 May 13, 2025

fwyzard mentioned this pull request May 13, 2025

Update CUDA to version 12.9.0 and cuDNN to version 9.9.0 #9833

Closed

cmsbuild added tests-approved and removed tests-started labels May 13, 2025

fwyzard mentioned this pull request May 14, 2025

PyTorch: fix configuration of CUDA and cuDNN [15.0.x] #9859

Merged

fwyzard force-pushed the IB/CMSSW_15_1_X/master_cuda_12.9.0_for_ARM branch from 9f730e3 to 4089780 Compare May 14, 2025 16:09

cmsbuild added tests-pending and removed tests-approved labels May 14, 2025

fwyzard force-pushed the IB/CMSSW_15_1_X/master_cuda_12.9.0_for_ARM branch from 4089780 to a267965 Compare May 14, 2025 16:11

cmsbuild added tests-started and removed tests-pending labels May 14, 2025

fwyzard added 4 commits May 16, 2025 13:16

Update CUDA to version 12.9.0

e92e675

Drop support for nvtx2 in favour of nvtx3, which is header-only.

Update cuDNN to version 9.9.0

c80ebd6

PyTorch: fix use of NVTXv3

05125b2

CUDA: use the redistributable CUDA driver libraries in place of the s…

8801d64

…tub libraries

fwyzard force-pushed the IB/CMSSW_15_1_X/master_cuda_12.9.0_for_ARM branch from d19948e to 8801d64 Compare May 16, 2025 11:16

cmsbuild added tests-pending and removed tests-approved requires-external labels May 16, 2025

cmsbuild added tests-started and removed tests-pending labels May 16, 2025

cmsbuild added tests-rejected and removed tests-started labels May 16, 2025

smuzaffar merged commit d684de6 into cms-sw:IB/CMSSW_15_1_X/master May 17, 2025
8 of 9 checks passed

fwyzard deleted the IB/CMSSW_15_1_X/master_cuda_12.9.0_for_ARM branch May 17, 2025 07:31

cmsbuild mentioned this pull request May 19, 2025

[ROOT636_X] Updated root to tip of branch v6-36-00-patches #9874

Merged

fwyzard mentioned this pull request May 20, 2025

Do not look for libnvToolsExt.{so,a} #9875

Closed

Update CUDA to version 12.9.0 and cuDNN to version 9.9.0 #9856

Update CUDA to version 12.9.0 and cuDNN to version 9.9.0 #9856

Uh oh!

Conversation

fwyzard commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented May 13, 2025

Uh oh!

cmsbuild commented May 13, 2025

Uh oh!

cmsbuild commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented May 13, 2025

Uh oh!

fwyzard commented May 13, 2025

Uh oh!

fwyzard commented May 13, 2025

Uh oh!

fwyzard commented May 13, 2025

Uh oh!

cmsbuild commented May 13, 2025

External Build

Uh oh!

cmsbuild commented May 13, 2025

External Build

Uh oh!

cmsbuild commented May 13, 2025

Comparison Summary

Uh oh!

cmsbuild commented May 13, 2025

External Build

Uh oh!

fwyzard commented May 14, 2025

Uh oh!

cmsbuild commented May 14, 2025

Uh oh!

fwyzard commented May 14, 2025

Uh oh!

cmsbuild commented May 14, 2025

Uh oh!

cmsbuild commented May 16, 2025

Unit Tests

Uh oh!

iarspider commented May 16, 2025

Uh oh!

fwyzard commented May 16, 2025

Uh oh!

fwyzard commented May 16, 2025

Uh oh!

fwyzard commented May 16, 2025

Uh oh!

cmsbuild commented May 16, 2025

Uh oh!

fwyzard commented May 16, 2025

Uh oh!

smuzaffar commented May 16, 2025

Uh oh!

smuzaffar commented May 16, 2025

Uh oh!

fwyzard commented May 16, 2025

Uh oh!

fwyzard commented May 16, 2025

Uh oh!

smuzaffar commented May 16, 2025

Uh oh!

cmsbuild commented May 16, 2025

Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented May 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

fwyzard commented May 13, 2025 •

edited

Loading

cmsbuild commented May 13, 2025 •

edited

Loading