Update onnxruntime to 1.10.0 #7695

iarspider · 2022-03-16T09:16:33Z

iarspider · 2022-03-16T09:16:41Z

please test

cmsbuild · 2022-03-16T09:16:57Z

A new Pull Request was created by @iarspider for branch IB/CMSSW_12_4_X/master.

@smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

iarspider · 2022-03-16T09:35:11Z

please test

cmsbuild · 2022-03-16T09:35:21Z

Pull request #7695 was updated.

cmsbuild · 2022-03-16T10:19:08Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23154/summary.html
COMMIT: 8c52813
CMSSW: CMSSW_12_4_X_2022-03-15-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7695/23154/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ '[' 0 -ne 0 ']'
+ cd onnxruntime-1.10.0
+ chmod -Rf a+rX,u+w,g-w,o-w .
+ patch -p1
/pool/condor/dir_254446/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wcvNVI: line 41: /pool/condor/dir_254446/jenkins/workspace/ib-run-pr-tests/testBuildDir/SOURCES/external/onnxruntime/1.10.0-679904dd98f448b23cfea64f83c7e900/de4089f8cbe0baffe56a363cc3a41595cc8f0809.patch: No such file or directory
error: Bad exit status from /pool/condor/dir_254446/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wcvNVI (%prep)


RPM build errors:
Bad exit status from /pool/condor/dir_254446/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wcvNVI (%prep)

smuzaffar · 2022-03-16T10:23:14Z

@iarspider , you also need to remove https://github.com/cms-sw/cmsdist/pull/7695/files#diff-5b95b8765ca364b1c0447afc4ee95f43880cb14da555b8f94bf6fa5668dd81ffR16 line

iarspider · 2022-03-16T10:27:12Z

please test

cmsbuild · 2022-03-16T10:27:18Z

Pull request #7695 was updated.

cmsbuild · 2022-03-16T15:17:35Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23156/summary.html
COMMIT: 1b1fc1e
CMSSW: CMSSW_12_4_X_2022-03-15-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7695/23156/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update aiohttp to 3.8.1 #7690

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23156/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23156/git-merge-result

Comparison Summary

The workflows 1001.0, 1000.0, 136.88811, 136.874, 136.8311, 136.793, 136.7611, 136.731, 4.22 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695377
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695353
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-03-16T16:06:18Z

please test
lets rerun the tests using new Ib

smuzaffar · 2022-03-16T16:07:31Z

please test for slc7_aarch64_gcc11

smuzaffar · 2022-03-16T16:09:20Z

please test

smuzaffar · 2022-03-16T16:09:34Z

please test for slc7_ppc64le_gcc11

cmsbuild · 2022-03-16T18:06:31Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23166/summary.html
COMMIT: 1b1fc1e
CMSSW: CMSSW_12_4_X_2022-03-15-2300/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7695/23166/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23166/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23166/git-merge-result

RelVals

4.53

----- Begin Fatal Exception 16-Mar-2022 18:19:51 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 16-Mar-2022 18:29:30 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 16-Mar-2022 18:35:09 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

cmsbuild · 2022-03-16T18:27:55Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23167/summary.html
COMMIT: 1b1fc1e
CMSSW: CMSSW_12_4_X_2022-03-15-2300/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7695/23167/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23167/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23167/git-merge-result

cmsbuild · 2022-03-16T20:28:43Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23165/summary.html
COMMIT: 1b1fc1e
CMSSW: CMSSW_12_4_X_2022-03-16-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7695/23165/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23165/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23165/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695377
DQMHistoTests: Total failures: 10
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3695344
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 48 workflows

smuzaffar · 2022-03-16T22:02:39Z

enable gpu,threading,profiling

smuzaffar · 2022-03-16T22:02:45Z

please test

cmsbuild · 2022-03-17T06:44:35Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23168/summary.html
COMMIT: 1b1fc1e
CMSSW: CMSSW_12_4_X_2022-03-16-1100/slc7_amd64_gcc10
Additional Tests: GPU,THREADING,PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7695/23168/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23168/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0510ab/23168/git-merge-result

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19811
DQMHistoTests: Total failures: 1630
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 18181
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695377
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3695341
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 48 workflows

smuzaffar · 2022-03-17T07:14:06Z

+externals
@qliphy @perrotta , this is ready to go in. Feel free to merge it for 12.4.X and let me know if a backport of it is needed for 12.3.X

cmsbuild · 2022-03-17T07:14:28Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_4_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-03-17T09:35:37Z

+1

perrotta · 2022-03-17T09:36:58Z

@qliphy @perrotta , this is ready to go in. Feel free to merge it for 12.4.X and let me know if a backport of it is needed for 12.3.X

Thank you @smuzaffar
I would say that a backport to 12_3_X is in order, but let me check with the main users, @cms-sw/reconstruction-l2 , if they envisage any possible counterindication.

jpata · 2022-03-17T09:43:05Z

for my understanding, what would be the reason for the backport? is the new version solving some stability issue we had in 12_3, or is it aimed to retroactively support cms-sw/cmssw#36963?

@riga @yongbinfeng may know more

Update onnxruntime to 1.10.0

13a2d49

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Mar 16, 2022

Add missing commit

8c52813

cmsbuild added tests-rejected and removed tests-started labels Mar 16, 2022

remove patch

1b1fc1e

cmsbuild added tests-started and removed tests-rejected labels Mar 16, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 16, 2022

cmsbuild added tests-started and removed tests-approved labels Mar 16, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 16, 2022

cmsbuild added tests-started and removed tests-approved labels Mar 16, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 17, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Mar 17, 2022

cmsbuild added orp-approved and removed orp-pending labels Mar 17, 2022

cmsbuild merged commit 0734b14 into IB/CMSSW_12_4_X/master Mar 17, 2022

smuzaffar deleted the update-onnxruntime-v1.10.0 branch March 24, 2022 22:22

Update onnxruntime to 1.10.0 #7695

Update onnxruntime to 1.10.0 #7695

Uh oh!

Conversation

iarspider commented Mar 16, 2022

Uh oh!

iarspider commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

Uh oh!

iarspider commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

External Build

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

iarspider commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

Comparison Summary

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

RelVals

Uh oh!

cmsbuild commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 16, 2022

Comparison Summary

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

smuzaffar commented Mar 16, 2022

Uh oh!

cmsbuild commented Mar 17, 2022

GPU Comparison Summary

Comparison Summary

Uh oh!

smuzaffar commented Mar 17, 2022

Uh oh!

cmsbuild commented Mar 17, 2022

Uh oh!

perrotta commented Mar 17, 2022

Uh oh!

perrotta commented Mar 17, 2022

Uh oh!

jpata commented Mar 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants