Update OpenMPI and other network components #7544

fwyzard · 2022-01-13T18:15:30Z

Update OpenMPI to version 4.1.2

This is a bugfix release, see https://www.mail-archive.com/announce@lists.open-mpi.org/msg00145.html for the details.

Update UCX to version 1.12.0

This is a feature release, for a list of all changes, updates and bug fixes since 1.10.1 see:

Update RDMA Core to version 36.3

This is a bugfix release, for the changes since version 36.0 see:

Update GDRCopy to version 2.3

This is a bugfix release, for the changes since version 2.2 see https://github.com/NVIDIA/gdrcopy/releases/tag/v2.3 .

Update Hardware Locality library to version 2.7.0

This is a major release, see https://raw.githubusercontent.com/open-mpi/hwloc/v2.7/NEWS for all the changes.

This is a bugfix release, see https://www.mail-archive.com/announce@lists.open-mpi.org/msg00145.html for the details.

fwyzard · 2022-01-13T18:15:40Z

please test

cmsbuild · 2022-01-13T18:15:55Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_12_3_X/master.

@smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2022-01-13T18:16:33Z

hold

cmsbuild · 2022-01-13T18:16:52Z

Pull request has been put on hold by @fwyzard
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

fwyzard · 2022-01-13T18:17:09Z

I plan to submit various updates that eventually should be integrated together, but I would prefer to test them adding only one or two at a time.

This is a feature release, for a list of all changes, updates and bug fixes since 1.10.1 see: - https://github.com/openucx/ucx/releases/tag/v1.11.0 - https://github.com/openucx/ucx/releases/tag/v1.12.0

This is a bugfix release, for the changes since version 36.0 see: - https://github.com/linux-rdma/rdma-core/releases/tag/v36.1 - https://github.com/linux-rdma/rdma-core/releases/tag/v36.3

This is a bugfix release, for the changes since version 2.2 see https://github.com/NVIDIA/gdrcopy/releases/tag/v2.3 .

cmsbuild · 2022-01-14T06:18:51Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21700/summary.html
COMMIT: 718d0b1
CMSSW: CMSSW_12_3_X_2022-01-12-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21700/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update UTM to version 0.10.0 #7543

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21700/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21700/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3461659
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3461637
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-01-14T06:54:18Z

Pull request #7544 was updated.

fwyzard · 2022-01-14T06:54:43Z

please test

cmsbuild · 2022-01-14T16:25:08Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21708/summary.html
COMMIT: af61320
CMSSW: CMSSW_12_3_X_2022-01-13-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21708/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3461659
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3461631
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

fwyzard · 2022-01-14T16:48:36Z

please test

fwyzard · 2022-01-14T16:50:44Z

unhold

cmsbuild · 2022-01-14T18:07:11Z

Pull request #7544 was updated.

cmsbuild · 2022-01-15T04:00:43Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21735/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-14-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21735/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21735/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21735/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3461659
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3461637
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-01-23T21:39:19Z

please test

smuzaffar · 2022-01-23T21:39:52Z

please test for slc7_ppc64le_gcc11

smuzaffar · 2022-01-23T21:40:09Z

please test for slc7_aarch64_gcc11

cmsbuild · 2022-01-24T04:42:46Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21948/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-23-0000/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21948/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21948/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21948/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testFWCoreUtilities had ERRORS
---> test TestFWCoreFrameworkGlobalStreamOne had ERRORS
---> test materialBudgetTrackerPlots had ERRORS

cmsbuild · 2022-01-24T05:58:16Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21947/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-23-0000/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21947/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21947/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21947/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test materialBudgetTrackerPlots had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3465668
DQMHistoTests: Total failures: 249
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3465396
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 42 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-01-24T13:42:42Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21949/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-23-0000/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21949/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw New package: libzmq #7550

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21949/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21949/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test testFWCoreUtilities had ERRORS

RelVals

4.53

----- Begin Fatal Exception 24-Jan-2022 13:33:18 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 24-Jan-2022 13:43:01 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 24-Jan-2022 13:56:32 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

fwyzard · 2022-01-24T13:48:48Z

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21949/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-23-0000/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21949/install.sh to create a dev area with all the needed externals and cmssw changes.

The MPI unit tests did pass:

fwyzard · 2022-01-24T13:50:44Z

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21947/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-23-0000/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21947/install.sh to create a dev area with all the needed externals and cmssw changes.

The MPI unit tests did pass:

fwyzard · 2022-01-24T13:52:18Z

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eca18/21948/summary.html
COMMIT: 4c15d9b
CMSSW: CMSSW_12_3_X_2022-01-23-0000/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7544/21948/install.sh to create a dev area with all the needed externals and cmssw changes.

The MPI unit tests did pass:

smuzaffar · 2022-01-24T13:55:34Z

+externals
@fwyzard , this looks good to go in, are you done with updates?

cmsbuild · 2022-01-24T13:55:56Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_3_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

fwyzard · 2022-01-24T13:58:03Z

@smuzaffar yes, thanks, I unheld the PR when it was ready from my side.

smuzaffar · 2022-01-24T14:08:30Z

this looks good to go in, unit test failure for materialBudgetTrackerPlots is not related to changes here

Update OpenMPI to version 4.1.2

718d0b1

This is a bugfix release, see https://www.mail-archive.com/announce@lists.open-mpi.org/msg00145.html for the details.

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Jan 13, 2022

cmsbuild added the hold label Jan 13, 2022

fwyzard added 3 commits January 13, 2022 19:18

Update UCX to version 1.12.0

af61320

This is a feature release, for a list of all changes, updates and bug fixes since 1.10.1 see: - https://github.com/openucx/ucx/releases/tag/v1.11.0 - https://github.com/openucx/ucx/releases/tag/v1.12.0

Update RDMA Core to version 36.3

878e90c

This is a bugfix release, for the changes since version 36.0 see: - https://github.com/linux-rdma/rdma-core/releases/tag/v36.1 - https://github.com/linux-rdma/rdma-core/releases/tag/v36.3

Update GDRCopy to version 2.3

7ab9035

This is a bugfix release, for the changes since version 2.2 see https://github.com/NVIDIA/gdrcopy/releases/tag/v2.3 .

cmsbuild added tests-approved tests-pending and removed tests-started tests-approved labels Jan 14, 2022

cmsbuild added tests-started and removed tests-pending labels Jan 14, 2022

cmsbuild added tests-approved tests-started and removed tests-started tests-approved labels Jan 14, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 15, 2022

cmsbuild added tests-started and removed tests-approved labels Jan 23, 2022

cmsbuild added tests-rejected and removed tests-started labels Jan 24, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jan 24, 2022

smuzaffar merged commit d167cec into cms-sw:IB/CMSSW_12_3_X/master Jan 24, 2022

This was referenced Jan 24, 2022

[AArch64] build cmssw aarch64 with -mno-outline-atomics file #7582

Closed

Updated boost to 1.78 #7529

Merged

Update alpaka to the HEAD of the develop branch as of 2022.01.24 #7581

Merged

fwyzard deleted the IB/CMSSW_12_3_X/master_MPI_updates branch April 1, 2022 11:58

Update OpenMPI and other network components #7544

Update OpenMPI and other network components #7544

Uh oh!

Conversation

fwyzard commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update OpenMPI to version 4.1.2

Update UCX to version 1.12.0

Update RDMA Core to version 36.3

Update GDRCopy to version 2.3

Update Hardware Locality library to version 2.7.0

Uh oh!

fwyzard commented Jan 13, 2022

Uh oh!

cmsbuild commented Jan 13, 2022

Uh oh!

fwyzard commented Jan 13, 2022

Uh oh!

cmsbuild commented Jan 13, 2022

Uh oh!

fwyzard commented Jan 13, 2022

Uh oh!

cmsbuild commented Jan 14, 2022

Comparison Summary

Uh oh!

cmsbuild commented Jan 14, 2022

Uh oh!

fwyzard commented Jan 14, 2022

Uh oh!

cmsbuild commented Jan 14, 2022

Comparison Summary

Uh oh!

fwyzard commented Jan 14, 2022

Uh oh!

fwyzard commented Jan 14, 2022

Uh oh!

cmsbuild commented Jan 14, 2022

Uh oh!

cmsbuild commented Jan 15, 2022

Comparison Summary

Uh oh!

smuzaffar commented Jan 23, 2022

Uh oh!

smuzaffar commented Jan 23, 2022

Uh oh!

smuzaffar commented Jan 23, 2022

Uh oh!

cmsbuild commented Jan 24, 2022

Unit Tests

Uh oh!

cmsbuild commented Jan 24, 2022

Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Jan 24, 2022

Unit Tests

RelVals

Uh oh!

fwyzard commented Jan 24, 2022

Uh oh!

fwyzard commented Jan 24, 2022

Uh oh!

fwyzard commented Jan 24, 2022

Uh oh!

smuzaffar commented Jan 24, 2022

Uh oh!

cmsbuild commented Jan 24, 2022

Uh oh!

fwyzard commented Jan 24, 2022

Uh oh!

smuzaffar commented Jan 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fwyzard commented Jan 13, 2022 •

edited

Loading