Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44782

hqucms · 2024-04-19T00:45:43Z

PR description:

Building on the implementation by @Dr15Jones in #44575 and some discussions there, I modified the usage of SimpleFlatTableProducer to read concrete data types instead of the base reco::Candidate. This avoids the large number of lazyEval = True previously needed in #44575.

PR validation:

Tested for all NANO workflows:

runTheMatrix.py --ibeos --nEvents 100 -w nano -i all -l 2500.0,2500.001,2500.002,2500.01,2500.011,2500.012,2500.1,2500.2,2500.21,2500.211,2500.3,2500.301,2500.31,2500.311,2500.312,2500.313,2500.32,2500.321,2500.322,2500.323,2500.324,2500.325,2500.326,2500.327,2500.4,2500.401,2500.402,2500.403,2500.404,2500.405,2500.5,2500.51

The T0 had a failure from the string parser which we have previously attributed to using the lazy option of the string parser. Added option to turn on lazy parsing from the configuration. This is an 'opt' in since lazy parsing can be problematic so best to explicitly state it must be used.

cmsbuild · 2024-04-19T00:46:09Z

cms-bot internal usage

cmsbuild · 2024-04-19T00:54:44Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44782/40016

This PR adds an extra 108KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DPGAnalysis/MuonTools/interface/MuDigiBaseProducer.h modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File DPGAnalysis/MuonTools/interface/MuLocalRecoBaseProducer.h modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File DPGAnalysis/MuonTools/python/common_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File DPGAnalysis/MuonTools/python/nano_mu_reco_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/interface/SimpleFlatTableProducer.h modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/boostedTaus_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/btvMC_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/common_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/custom_btv_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/custom_jme_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/custom_muon_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/electrons_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/fsrPhotons_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/genparticles_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/isotracks_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/jetMC_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/jetsAK4_CHS_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/jetsAK4_Puppi_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/jetsAK8_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/leptonTimeLifeInfo_common_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/lowPtElectrons_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/met_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/muons_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/photons_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/run3scouting_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/simpleSingletonCandidateFlatTableProducer_cfi.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/taus_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575, Add Unified ParT to tau NanoAOD #44691
- File PhysicsTools/NanoAOD/python/vertices_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575

cmsbuild · 2024-04-19T00:55:11Z

A new Pull Request was created by @hqucms for master.

It involves the following packages:

CalibTracker/SiStripCommon (alca)
DPGAnalysis/MuonTools (xpog)
DataFormats/Candidate (reconstruction)
PhysicsTools/NanoAOD (xpog)

@vlimant, @cmsbuild, @hqucms, @saumyaphor4252, @perrotta, @jfernan2, @mandrenguyen, @consuegs can you please review it and eventually sign? Thanks.
@rsreds, @battibass, @robervalwalsh, @tocheng, @missirol, @AnnikaStein, @gpetruc, @yuanchao, @gbenelli, @rovere, @mmusich, @echabert this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

hqucms · 2024-04-19T00:57:31Z

enable nano

hqucms · 2024-04-19T06:42:24Z

please test

cmsbuild · 2024-04-19T10:27:27Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2ec77b/38946/summary.html
COMMIT: 66624e6
CMSSW: CMSSW_14_1_X_2024-04-18-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44782/38946/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 92 lines from the logs
Reco comparison results: 43 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3319599
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3319576
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 15
DQMHistoTests: Total histograms compared: 16792
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 16792
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
Checked 55 log files, 32 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.0	2.774	2.774	0.000 ( +0.0% )	3.57	3.51	+1.7%	2.259	2.310
2500.001	2.887	2.887	0.000 ( +0.0% )	3.20	3.13	+2.4%	2.688	2.625
2500.002	2.834	2.834	0.000 ( +0.0% )	3.34	3.27	+1.9%	2.685	2.627
2500.01	1.440	1.440	0.000 ( +0.0% )	6.26	6.03	+4.0%	2.329	2.297
2500.011	1.898	1.898	0.000 ( +0.0% )	3.37	3.27	+2.9%	2.465	2.445
2500.012	1.754	1.754	0.000 ( +0.0% )	4.92	4.75	+3.6%	2.431	2.360
2500.1	2.345	2.345	0.000 ( +0.0% )	4.44	4.49	-0.9%	2.070	2.074
2500.2	2.449	2.449	0.000 ( +0.0% )	5.25	5.14	+2.3%	1.986	1.922
2500.21	1.279	1.279	0.000 ( +0.0% )	3.51	3.49	+0.5%	2.270	2.343
2500.211	1.659	1.659	0.000 ( +0.0% )	3.28	3.13	+4.9%	2.345	2.240
2500.3	2.219	2.219	0.000 ( +0.0% )	10.01	9.65	+3.8%	1.966	2.000
2500.301	2.822	2.822	0.000 ( +0.0% )	8.59	8.08	+6.3%	1.956	1.951
2500.31	7.164	7.164	0.000 ( +0.0% )	1.53	1.30	+18.0%	1.709	1.704
2500.311	1.568	1.568	0.000 ( +0.0% )	8.88	4.78	+85.7%	1.061	1.055
2500.312	540.457	540.457	0.000 ( +0.0% )	0.54	0.53	+1.2%	1.605	1.602
2500.313	817.694	817.694	0.000 ( +0.0% )	0.76	0.74	+3.1%	1.592	1.614
2500.32	1.341	1.341	0.000 ( +0.0% )	13.16	12.64	+4.1%	2.364	2.333
2500.321	1.748	1.748	0.000 ( +0.0% )	10.05	8.99	+11.8%	2.453	2.447
2500.322	1.240	1.240	0.000 ( +0.0% )	8.96	9.23	-2.8%	2.231	2.210
2500.323	7.772	7.772	0.000 ( +0.0% )	3.73	3.21	+16.4%	1.817	1.928
2500.324	1.870	1.870	0.000 ( +0.0% )	9.70	9.02	+7.5%	2.193	2.188
2500.325	4.156	4.156	0.000 ( +0.0% )	4.51	4.22	+6.8%	2.145	2.147
2500.326	3.309	3.309	0.000 ( +0.0% )	1.66	1.65	+0.7%	2.153	2.119
2500.327	1.804	1.804	0.000 ( +0.0% )	10.03	9.29	+8.0%	2.341	2.327
2500.4	2.363	2.363	0.000 ( +0.0% )	9.43	9.21	+2.4%	1.830	1.812
2500.401	1.891	1.891	0.000 ( +0.0% )	8.69	8.12	+7.0%	1.689	1.707
2500.402	2.937	2.937	0.000 ( +0.0% )	8.86	8.05	+10.1%	1.870	1.782
2500.403	8.687	8.687	0.000 ( +0.0% )	3.17	2.96	+7.0%	1.928	1.786
2500.404	5.438	5.438	0.000 ( +0.0% )	1.22	1.21	+1.2%	1.833	1.810
2500.405	2.847	2.847	0.000 ( +0.0% )	8.92	8.22	+8.4%	1.909	1.910
2500.5	5.194	5.194	0.000 ( +0.0% )	16.37	15.77	+3.8%	1.535	1.564
2500.51	9.120	9.120	0.000 ( +0.0% )	9.84	9.59	+2.6%	1.473	1.512

hqucms · 2024-04-19T10:51:16Z

+1

perrotta · 2024-04-19T12:26:45Z

+alca

Updates in CalibTracker/SiStripCommon/plugins/SealModules.cc look correct

hqucms · 2024-04-23T06:38:58Z

+1

cmsbuild · 2024-04-23T06:39:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

antoniovilela · 2024-04-24T11:07:07Z

+1

antoniovilela · 2024-04-25T12:13:24Z

@cms-sw/xpog-l2 @hqucms

We need a quick fix for 14_1_X and 14_0_X for the ProductNotFound error in RelVals:

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_14_0_X_2024-04-24-2300/pyRelValMatrixLogs/run/546.0_DYToLL_M-50_13TeV_pythia8/step2_DYToLL_M-50_13TeV_pythia8.log#/

vlimant · 2024-04-25T13:17:59Z

looks like unrelated to this PR ; what else has gone in that IB ?

Dr15Jones · 2024-04-25T13:21:20Z

The list of PRs added to that build can be found here
https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_14_1_X

Dr15Jones · 2024-04-25T13:41:24Z

@vlimant it could be this PR if the type of the data product now being asked for by the module does not actually match the type in the edm::Event.

vlimant · 2024-04-25T13:50:22Z

indeed, the type is reco::GenMETCollection in the NANOGEN, and std::vector<pat::MET> in standard nano

hqucms · 2024-04-25T13:56:55Z

Fixes here:

master: Fix metMCTable in NanoGEN #44850
14_0_X: [14_0_X] Fix metMCTable in NanoGEN #44851

Dr15Jones · 2024-04-25T13:57:05Z

Given the values being used in the cut string, just using SimpleCandidateFlatTableProducer would still work without having to use lazy evaluation. Alternatively, switching to SimpleMETFlatTableProducer.

hqucms · 2024-04-25T14:00:11Z

just using SimpleCandidateFlatTableProducer would still work without having to use lazy evaluation

yes that's what I did :)

Dr15Jones and others added 2 commits April 16, 2024 10:06

Use explicit types for SimpleFlatTableProducer.

66624e6

cmsbuild added this to the CMSSW_14_1_X milestone Apr 19, 2024

cmsbuild added reconstruction-pending alca-pending pending-signatures tests-pending orp-pending code-checks-pending xpog-pending trk labels Apr 19, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 19, 2024

cmsbuild added tests-started and removed tests-pending labels Apr 19, 2024

cmsbuild added tests-approved and removed tests-started labels Apr 19, 2024

cmsbuild added xpog-approved and removed xpog-pending labels Apr 19, 2024

hqucms mentioned this pull request Apr 19, 2024

Explicit lazy parsing in SimpleFlatTableProducer #44575

Closed

cmsbuild added alca-approved and removed alca-pending labels Apr 19, 2024

cmsbuild added fully-signed xpog-approved and removed pending-signatures xpog-pending labels Apr 23, 2024

hqucms mentioned this pull request Apr 23, 2024

[14_0_X] Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44807

Merged

cmsbuild added orp-approved and removed orp-pending labels Apr 24, 2024

cmsbuild merged commit 21cb3af into cms-sw:master Apr 24, 2024

This was referenced Apr 24, 2024

Add script to merge the JSON files produced by the FastTimerService #44836

Merged

Phase2 OMTF emulator #44557

Merged

Adaptions for Pythia 8.311 #43786

Merged

This was referenced Apr 25, 2024

Fix metMCTable in NanoGEN #44850

Merged

[14_0_X] Fix metMCTable in NanoGEN #44851

Merged

antoniovilela mentioned this pull request Apr 25, 2024

Build CMSSW_14_0_6 #44852

Closed

cmsbuild mentioned this pull request Apr 26, 2024

[GCC13] Fix Woverloaded-virtual warning #44581

Merged

vlimant mentioned this pull request Sep 6, 2024

Cut parser error in ROOT master IB #33084

Open

This was referenced Oct 22, 2024

Updating the SimpleTableProducers p2l1pfp/FastPUPPI#72

Closed

[142X] Updating the SimpleTableProducers p2l1pfp/FastPUPPI#73

Merged

patinkaew mentioned this pull request Dec 11, 2024

Add support for adding variable-size attributes (std::vector) within objects for NanoAOD #46702

Merged

Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44782

Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44782

Uh oh!

Conversation

hqucms commented Apr 19, 2024

PR description:

PR validation:

Uh oh!

cmsbuild commented Apr 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Apr 19, 2024

Uh oh!

cmsbuild commented Apr 19, 2024

Uh oh!

hqucms commented Apr 19, 2024

Uh oh!

hqucms commented Apr 19, 2024

Uh oh!

cmsbuild commented Apr 19, 2024

Comparison Summary

NANO Comparison Summary

Uh oh!

hqucms commented Apr 19, 2024

Uh oh!

perrotta commented Apr 19, 2024

Uh oh!

hqucms commented Apr 23, 2024

Uh oh!

cmsbuild commented Apr 23, 2024

Uh oh!

antoniovilela commented Apr 24, 2024

Uh oh!

antoniovilela commented Apr 25, 2024

Uh oh!

vlimant commented Apr 25, 2024

Uh oh!

Dr15Jones commented Apr 25, 2024

Uh oh!

Dr15Jones commented Apr 25, 2024

Uh oh!

vlimant commented Apr 25, 2024

Uh oh!

hqucms commented Apr 25, 2024

Uh oh!

Dr15Jones commented Apr 25, 2024

Uh oh!

hqucms commented Apr 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cmsbuild commented Apr 19, 2024 •

edited

Loading