Update cuDNN to v8.3.3.40 for CUDA 11.5 #7749

iarspider · 2022-04-05T13:48:57Z

No description provided.

iarspider · 2022-04-05T13:49:06Z

please test

cmsbuild · 2022-04-05T13:49:15Z

A new Pull Request was created by @iarspider for branch IB/CMSSW_12_4_X/master.

@smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2022-04-05T19:11:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23661/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-05-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23661/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23661/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23661/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3156 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3593039
DQMHistoTests: Total failures: 9298
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3583719
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 45 edm output root files, 48 DQM output files
TriggerResults: found differences in 1 / 47 workflows

iarspider · 2022-04-05T20:01:28Z

enable gpu

iarspider · 2022-04-05T20:01:34Z

please test

cmsbuild · 2022-04-05T23:55:04Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23668/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-05-1100/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23668/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23668/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23668/git-merge-result

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19874
DQMHistoTests: Total failures: 973
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 18901
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3156 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3593039
DQMHistoTests: Total failures: 9298
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3583719
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 45 edm output root files, 48 DQM output files
TriggerResults: found differences in 1 / 47 workflows

smuzaffar · 2022-04-06T07:01:05Z

please test

cmsbuild · 2022-04-06T08:45:30Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23677/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-05-2300/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23677/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

25202.0

----- Begin Fatal Exception 06-Apr-2022 10:10:49 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MixingModule/'mix'
   [3] Calling RootInputFileSequence::initTheFile()
   [4] Calling StorageFactory::open()
   [5] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/relval/CMSSW_10_6_0/RelValMinBias_13/GEN-SIM/106X_mcRun2_asymptotic_v3-v1/10000/C0533456-718E-F440-9748-4DEAF7EECEFE.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_10_6_0/RelValMinBias_13/GEN-SIM/106X_mcRun2_asymptotic_v3-v1/10000/C0533456-718E-F440-9748-4DEAF7EECEFE.root
      [b] Input file root://cms-xrd-global.cern.ch//store/relval/CMSSW_10_6_0/RelValMinBias_13/GEN-SIM/106X_mcRun2_asymptotic_v3-v1/10000/C0533456-718E-F440-9748-4DEAF7EECEFE.root could not be opened.
      [c] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/relval/CMSSW_10_6_0/RelValMinBias_13/GEN-SIM/106X_mcRun2_asymptotic_v3-v1/10000/C0533456-718E-F440-9748-4DEAF7EECEFE.root', flags=0x10, permissions=0660) => error '[ERROR] Operation expired' (errno=0, code=206). No additional data servers were found.
      [d] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_10_6_0/RelValMinBias_13/GEN-SIM/106X_mcRun2_asymptotic_v3-v1/10000/C0533456-718E-F440-9748-4DEAF7EECEFE.root?tried=&xrdcl.requuid=2abe1f93-b407-46f4-89bd-9833c33120e3
      [e] Problematic data server: cms-xrd-global.cern.ch:1094
      [f] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19874
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19865
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-04-06T15:35:32Z

please test

cmsbuild · 2022-04-06T21:55:26Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23702/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-06-1100/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23702/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23702/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23702/git-merge-result

GPU Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19874
DQMHistoTests: Total failures: 1195
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 18679
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 3 / 3 workflows

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 13989 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3593033
DQMHistoTests: Total failures: 66571
DQMHistoTests: Total nulls: 3
DQMHistoTests: Total successes: 3526437
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.016 KiB( 47 files compared)
DQMHistoSizes: changed ( 136.874 ): -0.004 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 138.4 ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 138.5 ): -0.008 KiB JetMET/SUSYDQM
Checked 200 log files, 45 edm output root files, 48 DQM output files
TriggerResults: found differences in 11 / 47 workflows

smuzaffar · 2022-04-07T09:19:10Z

please test

cmsbuild · 2022-04-08T00:28:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23716/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-06-2300/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23716/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19874
DQMHistoTests: Total failures: 996
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 18878
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3593039
DQMHistoTests: Total failures: 19
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3592997
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 47 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 200 log files, 45 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-04-08T06:23:39Z

+externals
comparison differences are in message logger mostly

cmsbuild · 2022-04-08T06:24:03Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_4_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2022-04-08T08:17:50Z

please test for slc7_aarch64_gcc11

smuzaffar · 2022-04-08T11:36:46Z

please test for slc7_ppc64le_gcc11

cmsbuild · 2022-04-08T13:17:49Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23763/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-07-2300/slc7_aarch64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23763/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23763/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23763/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test import-tables had ERRORS
---> test testTables had ERRORS
and more ...

RelVals

4.53

----- Begin Fatal Exception 08-Apr-2022 14:33:47 CEST-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 08-Apr-2022 14:43:09 CEST-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 08-Apr-2022 14:49:02 CEST-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

cmsbuild · 2022-04-09T02:01:14Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23781/summary.html
COMMIT: 0d7e6c9
CMSSW: CMSSW_12_4_X_2022-04-07-2300/slc7_ppc64le_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7749/23781/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23781/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20340b/23781/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test DRNTest had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test import-tables had ERRORS
---> test testTables had ERRORS
and more ...

smuzaffar · 2022-04-09T15:13:20Z

@fwyzard this looks good to go in, any objections on this update?

fwyzard · 2022-04-09T15:43:31Z

OK for me - I don't have any particular tests for cuDNN.

Update cuDNN to v8.3.3.40 for CUDA 11.5

0d7e6c9

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Apr 5, 2022

cmsbuild added tests-approved and removed tests-started labels Apr 5, 2022

cmsbuild added tests-started and removed tests-approved labels Apr 5, 2022

cmsbuild added tests-approved and removed tests-started labels Apr 5, 2022

cmsbuild added tests-started and removed tests-approved labels Apr 6, 2022

cmsbuild added tests-rejected and removed tests-started labels Apr 6, 2022

cmsbuild added tests-started and removed tests-rejected labels Apr 6, 2022

cmsbuild added tests-approved and removed tests-started labels Apr 6, 2022

cmsbuild added tests-started and removed tests-approved labels Apr 7, 2022

cmsbuild added tests-approved and removed tests-started labels Apr 8, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Apr 8, 2022

smuzaffar merged commit ea806e6 into IB/CMSSW_12_4_X/master Apr 9, 2022

smuzaffar deleted the update-cudnn-8.3.3 branch April 9, 2022 16:15

This was referenced Apr 10, 2022

[Spack] Update py-packages to the latest version #7767

Merged

Fix py-tables issue for aarch64/ppc64le #7774

Merged

Update cuDNN to v8.3.3.40 for CUDA 11.5 #7749

Update cuDNN to v8.3.3.40 for CUDA 11.5 #7749

Uh oh!

Conversation

iarspider commented Apr 5, 2022

Uh oh!

iarspider commented Apr 5, 2022

Uh oh!

cmsbuild commented Apr 5, 2022

Uh oh!

cmsbuild commented Apr 5, 2022

Comparison Summary

Uh oh!

iarspider commented Apr 5, 2022

Uh oh!

iarspider commented Apr 5, 2022

Uh oh!

cmsbuild commented Apr 5, 2022

GPU Comparison Summary

Comparison Summary

Uh oh!

smuzaffar commented Apr 6, 2022

Uh oh!

cmsbuild commented Apr 6, 2022

RelVals

GPU Comparison Summary

Uh oh!

smuzaffar commented Apr 6, 2022

Uh oh!

cmsbuild commented Apr 6, 2022

GPU Comparison Summary

Comparison Summary

Uh oh!

smuzaffar commented Apr 7, 2022

Uh oh!

cmsbuild commented Apr 8, 2022

GPU Comparison Summary

Comparison Summary

Uh oh!

smuzaffar commented Apr 8, 2022

Uh oh!

cmsbuild commented Apr 8, 2022

Uh oh!

smuzaffar commented Apr 8, 2022

Uh oh!

smuzaffar commented Apr 8, 2022

Uh oh!

cmsbuild commented Apr 8, 2022

Unit Tests

RelVals

Uh oh!

cmsbuild commented Apr 9, 2022

Unit Tests

Uh oh!

smuzaffar commented Apr 9, 2022

Uh oh!

fwyzard commented Apr 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants