Event Display DQM clients freezes due particleFlowClusterOOTECALUncorrected module

[pmandrik]

Following [0] issue e-log report for SPLASH run 345922 we found that this stuck can be reproduced at lxplus:

scram project CMSSW_12_0_2_patch2
cd CMSSW_12_0_2_patch2/src; cmsenv
git cms-addpkg DQM
git fetch official-cmssw refs/pull/35674/*:refs/gh-remotes/pull/35674/*; git cms-merge-topic 35674
git fetch official-cmssw refs/pull/35365/*:refs/gh-remotes/pull/35674/*; git cms-merge-topic 35365
cp -r /afs/cern.ch/user/p/pmandrik/public/temp/data .
mkdir upload

Then build and set SPLASH run settings following [1] and set in the DQM/Integration/python/clients/visualization-live_cfg.py output folder and skipFirstLumis option (also to simplify debugging numberOfThreads = cms.untracked.uint32(1)):

outDir = "./upload"  
source.skipFirstLumis = False

Execute:
cmsRun DQM/Integration/python/clients/visualization-live_cfg.py runInputDir=./data runNumber=345922 runkey=cosmic_run scanOnce=True

Using the process.Tracer = cms.Service("Tracer") we can see that cmsRun always freezes at Event 2219 Lumi 96 in module label = 'particleFlowClusterOOTECALUncorrected' :
starting: processing event for module: stream = 0 label = 'particleFlowClusterOOTECALUncorrected' id = 510

Scenario: ppEra_Run3, GT: "120X_dataRun3_Express_v2"

[0] http://cmsonline.cern.ch/cms-elog/1125794
[1] https://twiki.cern.ch/twiki/bin/view/CMS/DQMOnline#SPLASH_run

[cmsbuild]

A new Issue was created by @pmandrik .

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@ahmad3213,@rvenditti,@emanueleusai,@pbo0,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[jfernan2]

assign reconstruction
Since the stuck is at a RECO module

[cmsbuild]

New categories assigned: reconstruction

@slava77,@jpata you have been requested to review this Pull request/Issue and eventually sign? Thanks

[slava77]

@cms-sw/ecal-dpg-l2 @cms-sw/pf-l2
please take a note

There will be more examples from a more directly available data files in
https://hypernews.cern.ch/HyperNews/CMS/get/tier0-Ops/2293.html

[thomreis]

Event 2219 in LS 96 was the first splash event in that run. ECAL was pre-firing on that event, which means that basically all 30k RecHits are OOT and almost none in time. Perhaps this has an effect on the module.

[slava77]

Event 2219 in LS 96 was the first splash event in that run. ECAL was pre-firing on that event, which means that basically all 30k RecHits are OOT and almost none in time. Perhaps this has an effect on the module.

perhaps some "input is too large" detection is needed for the module (unless there is an obvious fixable runaway in combinatorics).
@cms-sw/pf-l2 @laurenhay @marksan87 please check

[slava77]

I think I found a "bug".
... testing