[NGT] Extension of CA Pixel Tracking to Phase 2 Outer Tracker barrel

[Parsifal-2045]

PR description:

This PR was co-developed by: @AdrianoDee @JanGerritSchulz @mmusich @rovere

The Phase 2 CA Extension for PixelTracks enables the usage of recHits found in the first 3 layers of the Outer Tracker barrel in addition to the pixel recHits in the CA algorithm. As part of this work, the regular CA has also been updated and its parameters re-tuned for Phase 2. In particular, two more connections have been added in the very forward region bringing the total number of layer pairs to 57; a few more cuts have also been added and others have been re-tuned. The newly optimised, non-extended CA is proposed as the new default pixel tracking configuration, producing tracks with 4+ hits.

The CA extension provides a more radical improvement, bringing the total number of layer pairs to 71, taking advantage of 3 more layers in the barrel, while still producing tracks with 4+ hits. It is not enabled by default, but can be tested using in workflows *.7511, *.771, *.774, and *.775, which are also part of the IB matrix or by enabling the phase2CAExtension procModifier.

Full talks with discussions on physics performance as well as the overall impact on general tracks have been recently given at the HLT Upgrade and Tracking POG. These slides also contain links to a large set of plots and comparisons.

Main developments

To support the extension, two new plugins have been developed:

• Phase2OTRecHitsSoAConverter converts the recHits on the P modules of the Outer Tracker barrel detector into SoA format using the same layout and following the same ordering logic as what is already in use in the pixel
• SiPixelRecHitExtendedAlpaka merges the existing trackingRecHitsSoACollection for pixel hits with the newly converted one so that each column is extended with the hits from the OT, meaning that the CA interface with the input collection does not need to be changed

Other modules have been modified to support the extension, including:

• PixelTrackProducerFromSoAAlpaka, responsible for converting the PixelTracks in SoA format to legacy. It can now be configured to include the OT recHits in the legacy tracks or discard them
• A new TrackerTraits, phase2OT, has been introduce to be able to differentiate parameters, classes and functions, between regular and extended CA
• When the CA extension is used together with the trackingLST procModifier, the OT recHits are removed from PixelTracks fed in input to LST as to avoid a large increase in duplicates
• The initialStep high-purity selection is applied to extended PixelTracks to further reduce their fake rate. Note that this is a temporary solution until a proper DNN will be applied directly to the PixelTracks in SoA format in a follow-up PR.

Both the regular and extended CA have been optimised and new cuts have been introduced to improve fake rejection such as:

• Cuts on inner/outer max/min z (r) for layer pairs in the barrel (endcap)
• Layer pair depended pT cuts at the doublet building level
• adaptive chi2 cut based on the number of hits of the track

The meaning of the ngtScouting procModifier has been changed as follows:

• when enabled, ngtScouting, promotes PixelTracks as general tracks directly without any other pattern recognition or fitting step
  • enabling ngtScouting,phase2CAExtension uses extended PixelTracks as general tracks
  • enabling ngtScouting,phase2CAExtension,trackingLST merges extended PixelTracks with LST T5s and uses the resulting collection as general tracks

A few workflows have been added to test the extension and have been included in the IB matrix

• *.7511: HLT phase-2 timing menu, with PixelTracks CA Extension
• *.774: HLT phase-2 NGT Scouting menu Alpaka variant, with PixelTracks CA Extension as GeneralTracks
• *.775: HLT phase-2 NGT Scouting menu Alpaka variant, with Pixeltracks CA Extension + LST T5s as GeneralTracks

Physics performance

Run 3

Since we expect no changes in Run 3 performance, we verified that the Run 3 CA performs exactly the same using the following recipe:

#!/bin/bash -ex

cmsDriver.py step2 -s HLT:@relval2025,VALIDATION:hltMultiTrackValidation \
    --conditions auto:phase1_2025_realistic \
    --datatier DQMIO \
    -n 1000 \
    --eventcontent DQMIO \
    --geometry DB:Extended \
    --era Run3_2025 \
    --filein file:/eos/cms/store/relval/CMSSW_15_1_0_pre5/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_151X_mcRun3_2025_realistic_v4_STD_2025_PU-v2/2590000/0ecc755f-c744-48bb-8c46-abddb997dc46.root \
    --fileout file:step2.root \
    --nThreads 32 \
    --process HLTX \
    --inputCommands='keep *, drop *_hlt*_*_HLT, drop triggerTriggerFilterObjectWithRefs_l1t*_*_HLT' \
    >step2.log 2>&1

cmsDriver.py step3 -s HARVESTING:postProcessorHLTtrackingSequence \
    --conditions auto:phase1_2025_realistic \
    --mc \
    --geometry DB:Extended \
    --scenario pp \
    --filetype DQM \
    --era Run3_2025 \
    -n 1000 \
    --filein file:step2.root \
    --fileout file:step3.root >step3.log 2>&1

and comparing the DQM outputs.

Phase 2

Performance measured on a TTbar D110 PU Run4 RelVal sample (EDM input):

/RelValTTbar_14TeV/CMSSW_15_1_0_pre4-PU_150X_mcRun4_realistic_v1_STD_Run4D110_PU-v1/GEN-SIM-DIGI-RAW 

A detailed report on physics performance can be found in the talks linked above, here a few results are reported for completeness. Firstly, a comparison between legacy PixelTracks, new alpaka-based default, and extended CA (all plots

The impact on general tracks is more nuanced, here we only show a few configurations which were highlighted during the talks linked above and in the following discussions (all plots). Note, that the first 3 configurations run 2 tracking iterations (initialStep + highPtTriplets), while the last configuration singleIterCAExtensionLST runs a single tracking iteration where LST is seeded by extended PixelTracks only.

Timing

Measurements run on 2x AMD EPYC 9534 with 4x L40 GPUs. Configuration: 16 jobs with 16 threads/16 streams each, sample of 1000 TTbar 200PU events.

We have tested about 20 different configurations: the most interesting comparisons can be found in the talks linked above while all the results are here. We only show a comparison between legacy, the proposed new default and the promising singleIterCAExtLST configuration for the records

Legacy	New Default	singleIterCAExtLST

Memory usage

GPU memory usage have been measured while running the timing measurements using the output from nvidia-smi, thus the setup is exactly the same.

The non-extend optimised CA proposed as the new baseline shows a memory consumption comparable with what was already in release after #47611. The extension, including more layer pair connections and doubling the maximum number of doublets and tracks produced shows a more noticeable memory increase of about 34%

PR validation:

This PR has been tested locally running both addOnTests.py and runTheMatrix.py -l limited -i all --ibeos, both commands resulting in all tests passing. The newly introduced workflows have been tested with:

runTheMatrix.py -w upgrade -l 29634.7511,29634.774,29634.775

Update 25/09

We have updated the track selection settings for the default Pixel Tracking in Phase-2 (with Patatrack, without extension) by changing the $\chi^2$ / ndof requirement (depends on number of hits per track). It was previously set identical to the extended Pixel Tracking, which led to efficiency losses in the barrel due to shorter tracks. The change in efficiency for the new default before and after the commit is shown below. More validation plots can be found here.

The small efficiency losses at $|\eta|$ > 4 are due to an added requirement for quadruplets to not skip layers. This part of the track selection is planned to be replaced soon by a DNN to improve the track selection further.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48921/46075

• Found files with invalid states:

  • Validation/RecoParticleFlow/plugins/PFCandidateAnalyzerHLTDQM.cc:
    • Added: f3ecfb1
    • Deleted: e85a0ba
  • RecoTracker/PixelSeeding/plugins/alpaka/CASimpleCell.h:
    • Added: 6cf6f08
    • Deleted: f3ecfb1
  • HLTrigger/Configuration/python/HLT_75e33/modules/hltPhase2PixelTracksSelectionHighPurity_cfi.py:
    • Added: 7d5c1cf
    • Deleted: 06dd61a

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): CA Extension to strips #47090
  • File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): Fixed Number of Events Data Wfs for Testing #48490, [Do not merge] For cms-bot PR testing #48637
  • File DataFormats/TrackSoA/interface/alpaka/TrackUtilities.h modified in PR(s): CA Extension to strips #47090
  • File Geometry/CommonTopologies/interface/SimplePixelTopology.h modified in PR(s): CA Extension to strips #47090
  • File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): CA Extension to strips #47090
  • File RecoTracker/PixelSeeding/plugins/alpaka/BrokenLineFit.dev.cc modified in PR(s): CA Extension to strips #47090, Updated SoA View accessors from raw pointers to span #48377, Removed workaround for illegal memory access #48854
  • File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtuplet.cc modified in PR(s): Use TkPixelCPERecord for PixelCPEFastParams* #46852, CA Extension to strips #47090
  • File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtupletGenerator.cc modified in PR(s): CA Extension to strips #47090
  • File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtupletGeneratorKernels.dev.cc modified in PR(s): CA Extension to strips #47090, Updated SoA View accessors from raw pointers to span #48377
  • File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtupletGeneratorKernelsImpl.h modified in PR(s): CA Extension to strips #47090, Updated SoA View accessors from raw pointers to span #48377
  • File RecoTracker/PixelSeeding/plugins/alpaka/CAPixelDoubletsAlgos.h modified in PR(s): CA Extension to strips #47090
  • File RecoTracker/PixelSeeding/plugins/alpaka/HelixFit.cc modified in PR(s): CA Extension to strips #47090, Updated SoA View accessors from raw pointers to span #48377
  • File RecoTracker/PixelSeeding/plugins/alpaka/RiemannFit.dev.cc modified in PR(s): CA Extension to strips #47090
  • File RecoTracker/PixelTrackFitting/plugins/PixelTrackProducerFromSoAAlpaka.cc modified in PR(s): CA Extension to strips #47090, Updated SoA View accessors from raw pointers to span #48377

[cmsbuild]

A new Pull Request was created by @Parsifal-2045 for master.

It involves the following packages:

• Configuration/ProcessModifiers (operations)
• Configuration/PyReleaseValidation (pdmv, upgrade)
• DataFormats/SiPixelClusterSoA (reconstruction, heterogeneous)
• DataFormats/TrackSoA (reconstruction, heterogeneous)
• Geometry/CommonTopologies (geometry)
• HLTrigger/Configuration (hlt)
• RecoLocalTracker/Phase2TrackerRecHits (reconstruction, upgrade)
• RecoLocalTracker/SiPixelRecHits (reconstruction)
• RecoTracker/Configuration (reconstruction)
• RecoTracker/PixelSeeding (reconstruction)
• RecoTracker/PixelTrackFitting (reconstruction)
• RecoTracker/TkSeedGenerator (reconstruction)
• Validation/RecoTrack (dqm)

@AdrianoDee, @Dr15Jones, @Martin-Grunewald, @Moanwar, @antoniovagnerini, @bsunanda, @civanch, @cmsbuild, @ctarricone, @davidlange6, @DickyChant, @fabiocos, @ftenchini, @fwyzard, @gabrielmscampos, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @miquork, @mmusich, @nothingface0, @rseidita, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @Martin-Grunewald, @SohamBhattacharya, @VinInn, @VourMa, @bsunanda, @dgulhan, @dkotlins, @fabiocos, @felicepantaleo, @ferencek, @gpetruc, @makortel, @martinamalberti, @missirol, @mmusich, @mroguljic, @mtosi, @rovere, @slomeo, @threus, @tsusa, @tvami, @wmtford this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[mmusich]

type ngt

[mmusich]

test parameters:

• enable = gpu, hlt_p2_integration, hlt_p2_timing
• workflows = ph2_hlt

[mmusich]

allow @Parsifal-2045 test rights

[mmusich]

@cmsbuild, please test

[mmusich]

+hlt

[civanch]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1

[fwyzard]

🎉

[makortel]

This PR is a suspect for assertion failures in IBs, see #49266

[mmusich]

This PR is a suspect for assertion failures in IBs, see #49266

#49272 hopefully will fix that.