HCAL Run3: removing redundant hbhereco=HBHEIsolatedNoiseReflagger from wf's #30138

abdoulline · 2020-06-07T21:22:54Z

PR description:

Existing for Phase2 customization to exclude HBHEIsolatedNoiseReflagger (only relevant for Phase0 HPDs HCAL subdetectors), formerly known as "hbhereco" producer, has been backported to Run3 to save CPU and disk space: one HBHERecHits collection "hbhereco" now produced instead of two "hbheprereco" + "hbhereco" (effectively identical for >= Run3) previously.

NB: HLT for Run3 is already using solely "hbhereco"=HBHEPhase1Reconstructor.

PR validation:

runTheMatrix.py -l limited OK

No backport needed

cmsbuild · 2020-06-07T21:23:16Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-07T21:30:13Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30138/15924

This PR adds an extra 12KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File RecoLocalCalo/Configuration/python/hcalLocalReco_cff.py modified in PR(s): Hcal calibration cache #30060

cmsbuild · 2020-06-07T21:30:36Z

A new Pull Request was created by @abdoulline (Salavat Abdullin) for master.

It involves the following packages:

RecoJets/JetProducers
RecoLocalCalo/Configuration

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@rappoccio, @apsallid, @jdolen, @ahinzmann, @rovere, @jdamgov, @yslai, @nhanvtran, @gkasieczka, @clelange, @lecriste, @schoef, @mariadalfonso, @seemasharmafnal this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

slava77 · 2020-06-08T00:03:12Z

@cmsbuild please test

cmsbuild · 2020-06-08T00:03:31Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6867/console Started: 2020/06/08 02:05

cmsbuild · 2020-06-08T01:17:57Z

+1
Tested at: 59c44f6
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6555e3/6867/summary.html
CMSSW: CMSSW_11_2_X_2020-06-07-0000
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-08T01:18:00Z

Comparison job queued.

cmsbuild · 2020-06-08T03:02:50Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6555e3/6867/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2780497
DQMHistoTests: Total failures: 74
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2780373
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

abdoulline · 2020-06-08T06:51:39Z

Some difference observed only in old (obsolete ?) GlobalRecHitsV
(with getManyByType(hbhe) and interation over all collections [1])
for 11634.0 (2021) and 12434.0 (2023) wfs.

Possible explanation: this obsolete analyser [1] picks up and iterate over all existing HBHERecHits collections. Previously there were three of them: (i) hbheprereco, (ii) hbhereco and (iii)"reducedHcalRecHits" (wih label "hbhereco").
Now remains only the latter two. In addition, (iii) becomes smaller, as RecHits with redundant (set, but not used for Run3, Phase2 ) "HBHEIsolatedNoise" flag [2] are not added to this collection anymore.

[1]
https://cmssdt.cern.ch/lxr/source/Validation/GlobalRecHits/src/GlobalRecHitsProducer.cc#0526

[2]
https://cmssdt.cern.ch/lxr/source/DataFormats/METReco/interface/HcalPhase1FlagLabels.h#0009

abdoulline · 2020-06-08T08:41:53Z

Will follow (to be added to this PR) later today:
HCAL Run3-related DQM configs adjustment (move of existing Phase2 customization to Run3)

cmsbuild · 2020-06-08T10:39:08Z

Pull request #30138 was updated. @perrotta, @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU, @slava77 can you please check and sign again.

slava77 · 2020-06-08T12:14:58Z

@cmsbuild please test

cmsbuild · 2020-06-08T12:15:22Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6875/console Started: 2020/06/08 15:58

cmsbuild · 2020-06-08T16:05:51Z

+1
Tested at: 98a7464
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6555e3/6875/summary.html
CMSSW: CMSSW_11_2_X_2020-06-08-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-08T16:05:53Z

Comparison job queued.

cmsbuild · 2020-06-08T18:09:23Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6555e3/6875/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 20 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2779875
DQMHistoTests: Total failures: 33
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2779792
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -416.46 KiB( 35 files compared)
DQMHistoSizes: changed ( 11634.0,... ): -198.366 KiB JetMET/MET
DQMHistoSizes: changed ( 11634.0,... ): -9.864 KiB HcalNoiseRatesD/HcalNoiseRatesTask
Checked 152 log files, 16 edm output root files, 36 DQM output files

jfernan2 · 2020-06-09T08:00:30Z

Thanks @abdoulline for this cleaning.
There is a small reduction in number of entries for 4/5 plots, not sure if relevant. I ask you just in case it is not expected:
https://tinyurl.com/y9vaf5n3
https://tinyurl.com/ydz988jl

abdoulline · 2020-06-09T08:24:01Z

Thanks @abdoulline for this cleaning.
There is a small reduction in number of entries for 4/5 plots, not sure if relevant. I ask you just in case it is not expected:
https://tinyurl.com/y9vaf5n3
https://tinyurl.com/ydz988jl

@jfernan2
Yes, I've commented on them earlier in this thread (after the first round, yet without DQM changes),
giving an explanation: #30138 (comment)

jfernan2 · 2020-06-09T08:36:10Z

Thanks @abdoulline Sorry, I had understood your last commit would fix that too. OK

jfernan2 · 2020-06-09T08:36:15Z

+1

perrotta · 2020-06-12T07:38:10Z

+1

Removed redundant collection from Run3 HCal local reco
Jenkins tests pass: the reco outputs for Run3 wfs witness the removal of "hbheprereco", as desired; no other changes are observed

cmsbuild · 2020-06-12T07:38:36Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-06-12T08:56:51Z

+1

removing hbhereco=HBHEIsolatedNoiseReflagger from Run3

59c44f6

cmsbuild added this to the CMSSW_11_2_X milestone Jun 7, 2020

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jun 7, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 7, 2020

abdoulline mentioned this pull request Jun 7, 2020

HCAL only wf #30136

Merged

cmsbuild added tests-started and removed tests-pending labels Jun 8, 2020

cmsbuild added tests-approved and removed tests-started labels Jun 8, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jun 8, 2020

DQM HBHE-no-noise customization move to Run3

98a7464

cmsbuild removed code-checks-approved comparison-available labels Jun 8, 2020

cmsbuild added tests-started and removed tests-pending labels Jun 8, 2020

cmsbuild added tests-approved and removed tests-started labels Jun 8, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jun 8, 2020

cmsbuild added dqm-approved and removed dqm-pending labels Jun 9, 2020

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Jun 12, 2020

cmsbuild added orp-approved and removed orp-pending labels Jun 12, 2020

cmsbuild merged commit db2015d into cms-sw:master Jun 12, 2020

abdoulline mentioned this pull request Jul 30, 2020

CPU and disk optimization: TrackExtrapolator #30968

Open

abdoulline deleted the remove_HBHEIsolatedNoiseReflagger_fromRun3 branch June 6, 2021 17:30

HCAL Run3: removing redundant hbhereco=HBHEIsolatedNoiseReflagger from wf's #30138

HCAL Run3: removing redundant hbhereco=HBHEIsolatedNoiseReflagger from wf's #30138

Uh oh!

Conversation

abdoulline commented Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

No backport needed

Uh oh!

cmsbuild commented Jun 7, 2020

Uh oh!

cmsbuild commented Jun 7, 2020

Uh oh!

cmsbuild commented Jun 7, 2020

Uh oh!

slava77 commented Jun 8, 2020

Uh oh!

cmsbuild commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

abdoulline commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abdoulline commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

slava77 commented Jun 8, 2020

Uh oh!

cmsbuild commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

cmsbuild commented Jun 8, 2020

Uh oh!

jfernan2 commented Jun 9, 2020

Uh oh!

abdoulline commented Jun 9, 2020

Uh oh!

jfernan2 commented Jun 9, 2020

Uh oh!

jfernan2 commented Jun 9, 2020

Uh oh!

perrotta commented Jun 12, 2020

Uh oh!

cmsbuild commented Jun 12, 2020

Uh oh!

silviodonato commented Jun 12, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

abdoulline commented Jun 7, 2020 •

edited

Loading

cmsbuild commented Jun 8, 2020 •

edited

Loading

abdoulline commented Jun 8, 2020 •

edited

Loading

abdoulline commented Jun 8, 2020 •

edited

Loading

cmsbuild commented Jun 8, 2020 •

edited

Loading