Create new ECAL DQM GpuTask to monitor and compare CPU and GPU generated ECAL Rec Hits #35947

alejands · 2021-11-01T21:49:55Z

PR description:

PPD has requested DQM subsystems to monitor several GPU-enabled collections being introduced in CMSSW_12_1_X. We have introduced a new EcalMonitorTask called GpuTask designed to take in CPU and GPU generated rec hits and produce plots comparing several rec hit quantities for each run. We don't explicitly plot the GPU rec hit values for the sake of memory efficiency. Additional plots may be added in the future.

This task will not run by default on the regular Online DQM workflow.

PR validation:

One caveat to this version of the code is that we have not been able to test it on actual GPU rec hits. There appears to be no workflow currently that produces both types of rec hit collections. We have been informed by Thomas Reis that CPU and GPU ECAL rec hits are the same data type, so our testing is done by changing the input tags so that both collections are the CPU rec hits.

This code was run with the runTheMatrix workflow 10842.512 and the new plots look as expected.

Backport

This is a backport of #35946 to include this code in the 12_1_X release.

…d Ecal Rec Hits

cmsbuild · 2021-11-01T21:50:15Z

A new Pull Request was created by @alejands (Alejandro Sanchez) for CMSSW_12_1_X.

It involves the following packages:

DQM/EcalMonitorTasks (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please review it and eventually sign? Thanks.
@rchatter, @simonepigazzini, @thomreis, @argiro this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Backported from Create new ECAL DQM GpuTask to monitor and compare CPU and GPU generated ECAL Rec Hits #35946

emanueleusai · 2021-11-01T23:31:38Z

please test

cmsbuild · 2021-11-02T06:12:40Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-788a01/20168/summary.html
COMMIT: a47505d
CMSSW: CMSSW_12_1_X_2021-11-01-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35947/20168/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

Summary:

You potentially added 4033 lines to the logs
Reco comparison results: 88 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 32
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2901385
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 207.673 KiB( 41 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 3.350 KiB EcalBarrel/EBGpuTask
DQMHistoSizes: changed ( 1000.0,... ): 3.350 KiB EcalEndcap/EEGpuTask
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-11-02T16:18:01Z

backport

perrotta · 2021-11-02T16:53:20Z

backport of #35946

alejands · 2021-11-02T17:28:12Z

We believe that the tests failed due to the GPU ecalRecHits not being enabled in CMSSW at the moment. The @cuda version is commented out here: RecoLocalCalo/EcalRecProducers/python/ecalRecHit_cff.py#43. For the time being, we'll modify the GPU input tags to also read from the CPU ecalRecHits. This can be quickly changed back when the GPU rec hits are available.

cmsbuild · 2021-11-02T17:29:58Z

Pull request #35947 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please check and sign again.

jfernan2 · 2021-11-02T17:32:15Z

@alejands the UnitTest failed due to the issue fixed in this PR #35921 not to your code, so I would bring back the code as you had it. Thanks

alejands · 2021-11-02T17:47:26Z

@jfernan2 I have removed the modification commit from the commit history

jfernan2 · 2021-11-02T17:47:29Z

please test

cmsbuild · 2021-11-02T20:59:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-788a01/20200/summary.html
COMMIT: a47505d
CMSSW: CMSSW_12_1_X_2021-11-02-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35947/20200/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 3983 lines to the logs
Reco comparison results: 90 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 33
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901385
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 207.669 KiB( 41 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 3.350 KiB EcalBarrel/EBGpuTask
DQMHistoSizes: changed ( 1000.0,... ): 3.350 KiB EcalEndcap/EEGpuTask
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

cmsbuild · 2021-11-03T07:56:20Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_1_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

jfernan2 · 2021-11-03T10:00:11Z

@perrotta @qliphy I am not sure if a test with enable gpu may give some check to this PR given its nature

jfernan2 · 2021-11-03T15:03:54Z

-1
Same comment as in master PR

jfernan2 · 2021-11-03T16:24:44Z

enable gpu

cmsbuild · 2021-11-03T16:46:48Z

Pull request #35947 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please check and sign again.

jfernan2 · 2021-11-03T16:54:32Z

please test

cmsbuild · 2021-11-03T19:58:16Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-788a01/20232/summary.html
COMMIT: 5c364e9
CMSSW: CMSSW_12_1_X_2021-11-03-1100/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35947/20232/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19782
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19776
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 3.064 KiB( 3 files compared)
DQMHistoSizes: changed ( 11634.512 ): 1.532 KiB EcalBarrel/EBGpuTask
DQMHistoSizes: changed ( 11634.512 ): 1.532 KiB EcalEndcap/EEGpuTask
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901411
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 94.984 KiB( 41 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 1.532 KiB EcalBarrel/EBGpuTask
DQMHistoSizes: changed ( 1000.0,... ): 1.532 KiB EcalEndcap/EEGpuTask
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

qliphy · 2022-01-10T01:17:39Z

please test

alejands · 2022-01-10T01:26:19Z

@qliphy I have significant updates to this code that I'm about to submit as a new PR very soon. Perhaps it's best to close this PR and submit it all together

qliphy · 2022-01-10T01:30:56Z

please abort

qliphy · 2022-01-10T01:31:26Z

@qliphy I have significant updates to this code that I'm about to submit as a new PR very soon. Perhaps it's best to close this PR and submit it all together

Ok, thanks @alejands

Create new GpuTask module to monitor and compare CPU and GPU generate…

a47505d

…d Ecal Rec Hits

cmsbuild added this to the CMSSW_12_1_X milestone Nov 1, 2021

cmsbuild added dqm-pending orp-pending pending-signatures tests-pending labels Nov 1, 2021

alejands mentioned this pull request Nov 1, 2021

Create new ECAL DQM GpuTask to monitor and compare CPU and GPU generated ECAL Rec Hits #35946

Closed

cmsbuild added tests-started and removed tests-pending labels Nov 1, 2021

cmsbuild added tests-rejected and removed tests-started labels Nov 2, 2021

cmsbuild added the backport label Nov 2, 2021

cmsbuild added tests-pending and removed tests-rejected labels Nov 2, 2021

alejands force-pushed the gpuTask_12_1_X branch from ecbe217 to a47505d Compare November 2, 2021 17:44

cmsbuild added tests-rejected and removed tests-pending labels Nov 2, 2021

cmsbuild added tests-started and removed tests-rejected labels Nov 2, 2021

cmsbuild added dqm-rejected pending-signatures and removed dqm-approved fully-signed labels Nov 3, 2021

Add switch to GpuTask to prevent unnecessary warnings when not in use

5c364e9

cmsbuild added dqm-pending tests-pending and removed dqm-rejected tests-approved labels Nov 3, 2021

cmsbuild added tests-started and removed tests-pending labels Nov 3, 2021

cmsbuild added tests-approved and removed tests-started labels Nov 3, 2021

cmsbuild added tests-started and removed tests-approved labels Jan 10, 2022

cmsbuild added tests-pending and removed tests-started labels Jan 10, 2022

alejands closed this Jan 18, 2022

alejands deleted the gpuTask_12_1_X branch June 23, 2022 23:46

Create new ECAL DQM GpuTask to monitor and compare CPU and GPU generated ECAL Rec Hits #35947

Create new ECAL DQM GpuTask to monitor and compare CPU and GPU generated ECAL Rec Hits #35947

Uh oh!

Conversation

alejands commented Nov 1, 2021

PR description:

PR validation:

Backport

Uh oh!

cmsbuild commented Nov 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emanueleusai commented Nov 1, 2021

Uh oh!

cmsbuild commented Nov 2, 2021

Unit Tests

Comparison Summary

Uh oh!

jfernan2 commented Nov 2, 2021

Uh oh!

perrotta commented Nov 2, 2021

Uh oh!

alejands commented Nov 2, 2021

Uh oh!

cmsbuild commented Nov 2, 2021

Uh oh!

jfernan2 commented Nov 2, 2021

Uh oh!

alejands commented Nov 2, 2021

Uh oh!

jfernan2 commented Nov 2, 2021

Uh oh!

cmsbuild commented Nov 2, 2021

Comparison Summary

Uh oh!

cmsbuild commented Nov 3, 2021

Uh oh!

jfernan2 commented Nov 3, 2021

Uh oh!

jfernan2 commented Nov 3, 2021

Uh oh!

jfernan2 commented Nov 3, 2021

Uh oh!

cmsbuild commented Nov 3, 2021

Uh oh!

jfernan2 commented Nov 3, 2021

Uh oh!

cmsbuild commented Nov 3, 2021

GPU Comparison Summary

Comparison Summary

Uh oh!

qliphy commented Jan 10, 2022

Uh oh!

alejands commented Jan 10, 2022

Uh oh!

qliphy commented Jan 10, 2022

Uh oh!

qliphy commented Jan 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cmsbuild commented Nov 1, 2021 •

edited

Loading