Skip to content

Conversation

@brusale
Copy link
Contributor

@brusale brusale commented Apr 13, 2025

PR description:

This PR adds the possibility of running part of the TICL reconstruction framework in the barrel region. In particular, it introduces the ticl_barrel process modifier with which one can run CLUE in ECAL and HCAL (HO not included), also at HLT, and the corresponding associators and validation chain.

With this PR also the first workflows for TICL-barrel are introduced. These workflows run the modules mentioned above at step2 and step3, and with the HARVESTING one can produce the DQM plots for the LayerClusters in the barrel calorimeters.

PR validation:

Tested on wfs 29694.203, 29634.0 (these two with and without ticl_barrel process modifier) and 29753.209 (introduced by this PR).

@felicepantaleo @rovere @amecca

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 13, 2025

cms-bot internal usage

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44499

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @brusale for master.

It involves the following packages:

  • Configuration/Generator (generators)
  • Configuration/ProcessModifiers (operations)
  • Configuration/PyReleaseValidation (upgrade, pdmv)
  • DataFormats/CaloRecHit (reconstruction)
  • DataFormats/HGCalReco (upgrade, reconstruction)
  • HLTrigger/Configuration (hlt)
  • RecoHGCal/TICL (upgrade, reconstruction)
  • RecoLocalCalo/HGCalRecAlgos (upgrade, reconstruction)
  • RecoLocalCalo/HGCalRecProducers (upgrade, reconstruction)
  • RecoParticleFlow/PFClusterProducer (reconstruction)
  • SimCalorimetry/HGCalAssociatorProducers (simulation, upgrade)
  • SimGeneral/CaloAnalysis (simulation)
  • Validation/Configuration (simulation, dqm)
  • Validation/HGCalValidation (dqm)

@AdrianoDee, @Martin-Grunewald, @Moanwar, @antoniovagnerini, @antoniovilela, @bbilin, @civanch, @cmsbuild, @davidlange6, @DickyChant, @fabiocos, @jfernan2, @kpedro88, @lviliani, @mandrenguyen, @mdhildreth, @menglu21, @miquork, @mkirsano, @mmusich, @rappoccio, @rseidita, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @ReyerBand, @SohamBhattacharya, @VourMa, @apsallid, @argiro, @bsunanda, @cseez, @denizsun, @edjtscott, @fabiocos, @felicepantaleo, @forthommel, @hatakeyamak, @lecriste, @lgray, @makortel, @martinamalberti, @missirol, @mmarionncern, @mmusich, @pfs, @rchatter, @rovere, @salimcerci, @sameasy, @seemasharmafnal, @sethzenz, @slomeo, @sobhatta, @thomreis, @vandreev11, @wang0jin, @youyingli this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@mmusich
Copy link
Contributor

mmusich commented Apr 13, 2025

test parameters:

  • relvals_opt = --what upgrade
  • workflows = 29694.203, 29634.0, 29753.209, 29634.756
  • enable = hlt_p2_integration, hlt_p2_timing

@mmusich
Copy link
Contributor

mmusich commented Apr 13, 2025

@cmsbuild, please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: ClangBuild
Size: This PR adds an extra 512KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-724d13/45531/summary.html
COMMIT: 33de313
CMSSW: CMSSW_15_1_X_2025-04-13-0000/el8_amd64_gcc12
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47859/45531/install.sh to create a dev area with all the needed externals and cmssw changes.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

@mmusich
Copy link
Contributor

mmusich commented Jul 7, 2025

@bbilin @mkirsano @menglu21 @lviliani please review.

@mandrenguyen
Copy link
Contributor

+1

@mandrenguyen
Copy link
Contributor

merge

@cmsbuild cmsbuild merged commit 6bbdfff into cms-sw:master Jul 8, 2025
13 checks passed
@mandrenguyen
Copy link
Contributor

@cms-sw/generators-l2 Please feel free to sign (or object) retroactively. We wanted to get this one merged as it had to be rebased already several times.

@mmusich
Copy link
Contributor

mmusich commented Jul 9, 2025

It appears that this PR broke several wf, in the CMSSW_15_1_X_2025-07-08-2300 IB, e.g.:

  • configuration errors related to layerClustersEE and time_layerclustersEE in the HLT steps of the phase-2 workflows exercising the alpaka modifier
  • product not found (particleFlowRecHitECAL) in serveral .21 workflows

we're working on a fix.

@mmusich
Copy link
Contributor

mmusich commented Jul 9, 2025

  • configuration errors related to layerClustersEE and time_layerclustersEE in the HLT steps of the phase-2 workflows exercising the alpaka modifier

this issue is fixed at #48509

@mmusich
Copy link
Contributor

mmusich commented Jul 9, 2025

It appears that this PR broke several wf, in the CMSSW_15_1_X_2025-07-08-2300 IB, e.g.:

@smuzaffar in the CLANG-flavoured IB CMSSW_15_1_CLANG_X_2025-07-08-2300 we see a crash of the type:

MSG
09-Jul-2025 01:26:29 CEST  Closed file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_14_1_0/RelValTTbar_14TeV/GEN-SIM/141X_mcRun4_realistic_v1_STD_RegeneratedGS_2026D110_noPU-v1/2580000/3bc4d61f-f2d1-49c4-8111-8a00535e3cb4.root
----- Begin Fatal Exception 09-Jul-2025 01:26:29 CEST-----------------------
An exception of category 'PluginCacheError' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=BarrelLayerClusterProducer label='hltBarrelLayerClustersEB'
Exception Message:
The plugin 'EBCLUE' should have been in loadable
 '/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02897/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so'
 but was not there.  This means the plugin cache is incorrect.  Please run 'edmPluginRefresh /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02897/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so'
----- End Fatal Exception -------------------------------------------------

can you clarify what's the action item for this?

@smuzaffar
Copy link
Contributor

smuzaffar commented Jul 9, 2025

@mmusich , may be @makortel can guide you in better direction but what I see is that once you have a DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, EBCLUEAlgo, "EBCLUE"); then in the plugin library one should end up with some symbols which framework tries to load e.g. I see for
DEFINE_EDM_VALIDATED_PLUGIN(BaseCkfTrajectoryBuilderFactory, MuonCkfTrajectoryBuilder, "MuonCkfTrajectoryBuilder"); we have [a] but for EBCLUE I do not see any such symbols [b].

May be issue is with https://github.com/cms-sw/cmssw/blob/master/RecoParticleFlow/PFClusterProducer/plugins/BarrelCLUEAlgo.h#L188-L189 but as I wrote @makortel might have better clue

[a]

[muzaffar@cmsdev31 el8_amd64_gcc12]$ nm -D pluginRecoMuonL3TrackFinderPlugins.so | c++filt | grep 'MuonCkfTrajectoryBuilder::'
000000000001b5f0 T MuonCkfTrajectoryBuilder::fillPSetDescription(edm::ParameterSetDescription&)
000000000001b760 T MuonCkfTrajectoryBuilder::setEvent_(edm::Event const&, edm::EventSetup const&)
000000000001af90 T MuonCkfTrajectoryBuilder::MuonCkfTrajectoryBuilder(edm::ParameterSet const&, edm::ConsumesCollector&)
000000000001af90 T MuonCkfTrajectoryBuilder::MuonCkfTrajectoryBuilder(edm::ParameterSet const&, edm::ConsumesCollector&)
000000000001b5d0 T MuonCkfTrajectoryBuilder::~MuonCkfTrajectoryBuilder()
000000000001b540 T MuonCkfTrajectoryBuilder::~MuonCkfTrajectoryBuilder()
000000000001b540 T MuonCkfTrajectoryBuilder::~MuonCkfTrajectoryBuilder()
000000000001b900 T MuonCkfTrajectoryBuilder::collectMeasurement(DetLayer const*, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, TrajectoryStateOnSurface const&, std::vector<TrajectoryMeasurement, std::allocator<TrajectoryMeasurement> >&, int&, Propagator const*) const
000000000001be60 T MuonCkfTrajectoryBuilder::findCompatibleMeasurements(TrajectorySeed const&, TempTrajectory const&, std::vector<TrajectoryMeasurement, std::allocator<TrajectoryMeasurement> >&) const

[b]

Singularity> nm -D lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so | c++filt | grep EBCLU | wc -l
0

@mmusich
Copy link
Contributor

mmusich commented Jul 9, 2025

May be issue is with https://github.com/cms-sw/cmssw/blob/master/RecoParticleFlow/PFClusterProducer/plugins/BarrelCLUEAlgo.h#L188-L189 but as I wrote @makortel might have better clue

thanks for the analysis. We are a little puzzled though:

  1. as far as I can tell the issue is visible only in the CLANG IB.

  2. the Barrel algorithm was implemented in complete synchrony to what already exists for the End Caps (HGCal), see

EDM_REGISTER_VALIDATED_PLUGINFACTORY(HGCalLayerClusterAlgoFactory, "HGCalLayerClusterAlgoFactory");
DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, EBCLUEAlgo, "EBCLUE");
DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, HBCLUEAlgo, "HBCLUE");

and

// explicit template instantiation
extern template class BarrelCLUEAlgoT<EBLayerTiles>;
extern template class BarrelCLUEAlgoT<HBLayerTiles>;
using EBCLUEAlgo = BarrelCLUEAlgoT<EBLayerTiles>;
using HBCLUEAlgo = BarrelCLUEAlgoT<HBLayerTiles>;

VS

EDM_REGISTER_VALIDATED_PLUGINFACTORY(HGCalLayerClusterAlgoFactory, "HGCalLayerClusterAlgoFactory");
DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, HGCalImagingAlgo, "Imaging");
DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, HGCalSiCLUEAlgo, "SiCLUE");
DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, HGCalSciCLUEAlgo, "SciCLUE");
DEFINE_EDM_VALIDATED_PLUGIN(HGCalLayerClusterAlgoFactory, HFNoseCLUEAlgo, "HFNoseCLUE");

and

// explicit template instantiation
extern template class HGCalCLUEAlgoT<HGCalSiliconLayerTiles, HGCalSiliconStrategy>;
extern template class HGCalCLUEAlgoT<HGCalScintillatorLayerTiles, HGCalScintillatorStrategy>;
extern template class HGCalCLUEAlgoT<HFNoseLayerTiles, HGCalSiliconStrategy>;
using HGCalSiCLUEAlgo = HGCalCLUEAlgoT<HGCalSiliconLayerTiles, HGCalSiliconStrategy>;
using HGCalSciCLUEAlgo = HGCalCLUEAlgoT<HGCalScintillatorLayerTiles, HGCalScintillatorStrategy>;
using HFNoseCLUEAlgo = HGCalCLUEAlgoT<HFNoseLayerTiles, HGCalSiliconStrategy>;

@akritkbehera
Copy link
Contributor

akritkbehera commented Jul 9, 2025

Multiple RelVals are failing on IB CMSSW_15_1 2025-07-08-2300

What failed Description
RelVal 29634.21, 29834.21, 29834.9921 An exception of category ProductNotFound occurred.
RelVals 29634_X Configuration Error: Illegal parameters found in configuration. The parameters are named: layerClustersEE, time_layerclustersEE.
less step3_TTbar_14TeV+Run4D110_ProdLike.log
%MSG-w TrackListMerger:  TrackListMerger:earlyGeneralTracks  09-Jul-2025 11:12:14 CEST Run: 1 Event: 701
TrackCollection InputTag:  label = jetCoreRegionalStepTracks, instance =  not found
%MSG
----- Begin Fatal Exception 09-Jul-2025 12:32:56 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=MergeClusterProducer label='hltMergeLayerClusters'
Exception Message:
Illegal parameters found in configuration.  The parameters are named:
 'layerClustersEE'
 'time_layerclustersEE'
You could be trying to use parameter names that are not
allowed for this plugin or they could be misspelled.

Full Logs:
Product Not Found
Configuration Error

@mmusich
Copy link
Contributor

mmusich commented Jul 9, 2025

@akritkbehera what's the reason of this post?
It was already reported at #47859 (comment).

By the way:

less step3_TTbar_14TeV+Run4D110_ProdLike.log
%MSG-w TrackListMerger:  TrackListMerger:earlyGeneralTracks  09-Jul-2025 11:12:14 CEST Run: 1 Event: 701
TrackCollection InputTag:  label = jetCoreRegionalStepTracks, instance =  not found
%MSG

is unrelated.

@akritkbehera
Copy link
Contributor

akritkbehera commented Jul 9, 2025

Got it, thanks for pointing it out. I missed that comment. @mmusich

@smuzaffar
Copy link
Contributor

@mmusich , looks like fix in #48509 also fixes the CLANG_X relvals. So some strange reason the non-clang relvals were failing before loading EBCLUE plugin e.g. I added print statement and for non-clang I get

SMA2:EcalDetIdToBeRecoveredProducer    /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02897/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-07-07-2300/lib/el8_amd64_gcc12/pluginRecoLocalCaloEcalRecProducersPlugins.so
SMA2:HBHEPhase1Reconstructor    /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02897/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-07-07-2300/lib/el8_amd64_gcc12/pluginRecoLocalCaloHcalRecProducers.so
SMA2:HGCalUncalibRecHitProducer    /build/muz/x/CMSSW_15_1_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoLocalCaloHGCalRecProducersPlugins.so
SMA2:alpaka_serial_sync::HGCalSoARecHitsProducer    /build/muz/x/CMSSW_15_1_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoLocalCaloHGCalRecProducersPluginsPortableSerialSync.so
SMA2:BarrelLayerClusterProducer    /build/muz/x/CMSSW_15_1_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so
09-Jul-2025 12:34:54 CEST  Closed file root://eoscms.cern.ch//store/user/cmsbuild/store/relval/CMSSW_14_1_0/RelValTTbar_14TeV/GEN-SIM/141X_mcRun4_realistic_v1_STD_RegeneratedGS_2026D110_noPU-v1/2580000/3bc4d61f-f2d1-49c4-8111-8a00535e3cb4.root
----- Begin Fatal Exception 09-Jul-2025 12:34:54 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=MergeClusterProducer label='hltMergeLayerClusters'
Exception Message:
Illegal parameters found in configuration.  The parameters are named:
 'layerClustersEE'
 'time_layerclustersEE'
You could be trying to use parameter names that are not
allowed for this plugin or they could be misspelled.
----- End Fatal Exception -------------------------------------------------

and for clang I get

SMA2:alpaka_serial_sync::HGCalSoARecHitsProducer    /build/muz/x/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoLocalCaloHGCalRecProducersPluginsPortableSerialSync.so
SMA2:BarrelLayerClusterProducer    /build/muz/x/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so
SMA1:EBCLUE    /build/muz/x/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so
09-Jul-2025 12:25:31 CEST  Closed file root://eoscms.cern.ch//store/user/cmsbuild/store/relval/CMSSW_14_1_0/RelValTTbar_14TeV/GEN-SIM/141X_mcRun4_realistic_v1_STD_RegeneratedGS_2026D110_noPU-v1/2580000/3bc4d61f-f2d1-49c4-8111-8a00535e3cb4.root
----- Begin Fatal Exception 09-Jul-2025 12:25:31 CEST-----------------------
An exception of category 'PluginCacheError' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=BarrelLayerClusterProducer label='hltBarrelLayerClustersEB'
Exception Message:
The plugin 'EBCLUE' should have been in loadable
 '/build/muz/x/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so'
 but was not there.  This means the plugin cache is incorrect.  Please run 'edmPluginRefresh /build/muz/x/CMSSW_15_1_CLANG_X_2025-07-08-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so'
----- End Fatal Exception -------------------------------------------------

@mmusich
Copy link
Contributor

mmusich commented Jul 9, 2025

  • product not found (particleFlowRecHitECAL) in serveral .21 workflows

this issue is followed-up in #48517

@iarspider
Copy link
Contributor

@mmusich @makortel We still observe IB failures for CLANG_X : link

----- Begin Fatal Exception 16-Jul-2025 01:40:11 CEST-----------------------
An exception of category 'PluginCacheError' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=BarrelLayerClusterProducer label='hltBarrelLayerClustersEB'
Exception Message:
The plugin 'EBCLUE' should have been in loadable
 '/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02898/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_CLANG_X_2025-07-14-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so'
 but was not there.  This means the plugin cache is incorrect.  Please run 'edmPluginRefresh /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02898/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_CLANG_X_2025-07-14-2300/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFClusterProducerPlugins.so'
----- End Fatal Exception -------------------------------------------------

Is there a PR or a separate issue where this is being investigated? Thanks!

@makortel
Copy link
Contributor

Is there a PR or a separate issue where this is being investigated? Thanks!

I don't think there is a separate issue, could you open one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.