Update jemalloc to version 5.3.0 #7953

fwyzard · 2022-06-23T07:37:15Z

This release contains many speed and space optimizations, from micro optimizations on common paths to rework of internal data structures and locking schemes, and many more too detailed to list below. Multiple percent of system level metric improvements were measured in tested production workloads.
The release has gone through large-scale production testing.

See https://github.com/jemalloc/jemalloc/releases/tag/5.3.0 for more details.

Simplify and make more uniform the syntax of the jemalloc{,-debug,-prof}.spec files.

fwyzard · 2022-06-23T07:37:28Z

please test

cmsbuild · 2022-06-23T07:37:37Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_12_5_X/master.

@smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2022-06-23T07:40:59Z

please test for el8_amd64_gcc11

fwyzard · 2022-06-23T07:41:33Z

please test for el8_aarch64_gcc10

fwyzard · 2022-06-23T07:41:48Z

please test for el8_ppc64le_gcc10

fwyzard · 2022-06-23T07:42:15Z

please test for slc7_amd64_gcc10

fwyzard · 2022-06-23T07:49:14Z

@gartung FYI
The release notes mention some extra functionality for the flame graphs as well.

cmsbuild · 2022-06-23T10:35:23Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25723/summary.html
COMMIT: 0cc974e
CMSSW: CMSSW_12_5_X_2022-06-22-2300/el8_aarch64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7953/25723/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2022-06-23T11:43:31Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25726/summary.html
COMMIT: 0cc974e
CMSSW: CMSSW_12_5_X_2022-06-22-2300/el8_ppc64le_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7953/25726/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2022-06-23T12:36:08Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25717/summary.html
COMMIT: 0cc974e
CMSSW: CMSSW_12_5_X_2022-06-22-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7953/25717/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659307
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3659277
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-06-23T16:17:00Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25719/summary.html
COMMIT: 0cc974e
CMSSW: CMSSW_12_5_X_2022-06-22-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7953/25719/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 72981 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659307
DQMHistoTests: Total failures: 468162
DQMHistoTests: Total nulls: 373
DQMHistoTests: Total successes: 3190750
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.31499999999999995 KiB( 49 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.063 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11834.0 ): -2.372 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.006 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): 2.009 KiB SiStrip/MechanicalView
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: found differences in 14 / 49 workflows

cmsbuild · 2022-06-23T16:30:26Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25718/summary.html
COMMIT: 0cc974e
CMSSW: CMSSW_12_5_X_2022-06-22-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7953/25718/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 63130 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659099
DQMHistoTests: Total failures: 311068
DQMHistoTests: Total nulls: 151
DQMHistoTests: Total successes: 3347858
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 61.06600000000001 KiB( 49 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.352 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11834.0 ): 0.527 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 23234.0,... ): 6.885 KiB MTD/Tracks
DQMHistoSizes: changed ( 23234.0,... ): 0.742 KiB MTD/BTL
DQMHistoSizes: changed ( 250202.181 ): -0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.012 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: found differences in 14 / 49 workflows

smuzaffar · 2022-06-24T09:47:02Z

+externals

cmsbuild · 2022-06-24T09:47:26Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_5_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2022-06-24T09:47:37Z

@fwyzard , PR tests look good, let me know when you are done with your tests

fwyzard · 2022-06-26T04:25:11Z

Running a GPU-enabled HLT menu on one of the new nodes, jemalloc 5.3.0 is ~1% faster than 5.2.1.

jemalloc 5.2.1

Running 10 times over 10100 events with 8 jobs, each with 32 threads, 32 streams and 1 GPUs
   621.7 ±   3.5 ev/s

jemalloc 5.3.0

Running 10 times over 10100 events with 8 jobs, each with 32 threads, 32 streams and 1 GPUs
   629.7 ±   6.5 ev/s

fwyzard · 2022-06-26T04:26:38Z

I don't know if anyone is interested in measuring the impact on the simulation or the offline reconstruction.

From my point of view, we can merge the PR.

fwyzard · 2022-06-26T09:32:20Z

By the way, just a reminder of what is the performance without jemalloc, using cmsRunGlibC:

Running 10 times over 10100 events with 8 jobs, each with 32 threads, 32 streams and 1 GPUs
   577.8 ±   2.7 ev/s (based on 9 measurements)

smuzaffar · 2022-06-26T17:20:24Z

jemalloc-prof.spec

-Requires: libunwind
 ## INCLUDE jemalloc-common
 ### RPM external jemalloc-prof %{jemalloc_version}
+Requires: libunwind


@fwyzard , we need to keep this statement before ## INCLUDE jemalloc-common to make sure the it is defined before the %prep section defined in the jemalloc-common . Can you please revert this change ( and may be add a comment why it needs to be on top)

This release contains many speed and space optimizations, from micro optimizations on common paths to rework of internal data structures and locking schemes, and many more too detailed to list below. Multiple percent of system level metric improvements were measured in tested production workloads. The release has gone through large-scale production testing. See https://github.com/jemalloc/jemalloc/releases/tag/5.3.0 for more details.

fwyzard · 2022-06-26T20:42:01Z

please test

cmsbuild · 2022-06-26T20:42:16Z

Pull request #7953 was updated.

cmsbuild · 2022-06-27T07:26:29Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25813/summary.html
COMMIT: 412a6f8
CMSSW: CMSSW_12_5_X_2022-06-26-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7953/25813/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Numpy version 1.22.4, cython 0.29.30 #7956

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25813/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25813/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testhep_ml had ERRORS

Comparison Summary

There are some workflows for which there are errors in the baseline:
39634.9 step 4
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 11205 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659995
DQMHistoTests: Total failures: 6454
DQMHistoTests: Total nulls: 5
DQMHistoTests: Total successes: 3653514
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.039 KiB( 49 files compared)
DQMHistoSizes: changed ( 138.4 ): -0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 138.5 ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 4.53 ): -0.023 KiB JetMET/SUSYDQM
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: found differences in 3 / 49 workflows

smuzaffar · 2022-06-27T09:50:06Z

please test

cmsbuild · 2022-06-27T18:03:56Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25830/summary.html
COMMIT: 412a6f8
CMSSW: CMSSW_12_5_X_2022-06-26-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7953/25830/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw [BuildRules] Fixed code and header checks for alpaka #7960

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25830/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5b2cd5/25830/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659995
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3659971
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-06-27T20:17:34Z

+externals

cmsbuild · 2022-06-27T20:17:55Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_5_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Jun 23, 2022

fwyzard mentioned this pull request Jun 23, 2022

Update jemalloc to version 5.3.0 [12.4.x] #7954

Merged

cmsbuild added tests-approved and removed tests-started labels Jun 23, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jun 24, 2022

smuzaffar reviewed Jun 26, 2022

View reviewed changes

fwyzard force-pushed the IB/CMSSW_12_5_X/master_jemalloc_530 branch from 0cc974e to 412a6f8 Compare June 26, 2022 20:41

cmsbuild added externals-pending pending-signatures tests-started and removed tests-approved externals-approved fully-signed labels Jun 26, 2022

cmsbuild added tests-rejected and removed tests-started labels Jun 27, 2022

cmsbuild added tests-started and removed tests-rejected labels Jun 27, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 27, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jun 27, 2022

smuzaffar merged commit 644f773 into cms-sw:IB/CMSSW_12_5_X/master Jun 27, 2022

fwyzard deleted the IB/CMSSW_12_5_X/master_jemalloc_530 branch July 6, 2022 22:44

Update jemalloc to version 5.3.0 #7953

Update jemalloc to version 5.3.0 #7953

Uh oh!

Conversation

fwyzard commented Jun 23, 2022

Uh oh!

fwyzard commented Jun 23, 2022

Uh oh!

cmsbuild commented Jun 23, 2022

Uh oh!

fwyzard commented Jun 23, 2022

Uh oh!

fwyzard commented Jun 23, 2022

Uh oh!

fwyzard commented Jun 23, 2022

Uh oh!

fwyzard commented Jun 23, 2022

Uh oh!

fwyzard commented Jun 23, 2022

Uh oh!

cmsbuild commented Jun 23, 2022

Uh oh!

cmsbuild commented Jun 23, 2022

Uh oh!

cmsbuild commented Jun 23, 2022

Comparison Summary

Uh oh!

cmsbuild commented Jun 23, 2022

Comparison Summary

Uh oh!

cmsbuild commented Jun 23, 2022

Comparison Summary

Uh oh!

smuzaffar commented Jun 24, 2022

Uh oh!

cmsbuild commented Jun 24, 2022

Uh oh!

smuzaffar commented Jun 24, 2022

Uh oh!

fwyzard commented Jun 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jemalloc 5.2.1

jemalloc 5.3.0

Uh oh!

fwyzard commented Jun 26, 2022

Uh oh!

fwyzard commented Jun 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar Jun 26, 2022

Choose a reason for hiding this comment

Uh oh!

fwyzard Jun 26, 2022

Choose a reason for hiding this comment

Uh oh!

fwyzard commented Jun 26, 2022

Uh oh!

cmsbuild commented Jun 26, 2022

Uh oh!

cmsbuild commented Jun 27, 2022

Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented Jun 27, 2022

Uh oh!

cmsbuild commented Jun 27, 2022

Comparison Summary

Uh oh!

smuzaffar commented Jun 27, 2022

Uh oh!

cmsbuild commented Jun 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fwyzard commented Jun 26, 2022 •

edited

Loading

fwyzard commented Jun 26, 2022 •

edited

Loading