LST: add trackToTrack comparisons for GPU vs CPU tracking #48508

slava77 · 2025-07-08T23:13:59Z

the monitoring sequence is added in combination with trackingLST proc modifier for the offline setup.
In presence of the DQM (TrackingOfflineDQMClient) sequence this would trigger running a clone of the regular LST tracking in HighPtTripletStepTaskSerialSync (so that a no-DQM variant, e.g. used for timing, would not run the extra variant of this iteration).

DQM plots should show up in the /DQMData/Run 1/Tracking/Run summary/TrackBuilding/ValidationWRTSerialSync/highPtTripletStep folder

I made an attempt for a somewhat generic setup by introducing a trackToTrackCPUSequence, which can eventually be populated with other comparisons.

The implementation and population of the HighPtTripletStepTaskSerialSync task is a bit tedious,

Tested with 29834.704
The efficiency and deltaeta plots are shown below.

this PR is supposed to address the Implement a "GPU vs CPU" workflow part of #46746 (a followup to the LST integration PR #45117 (comment)) and was somewhat high on the wish list related to the presentation in the GPU meeting May 19

A specific implementation was not particularly well specified; so, this may become an RFC.

@VourMa @fwyzard

…rackingLST

cmsbuild · 2025-07-08T23:14:23Z

cms-bot internal usage

slava77 · 2025-07-08T23:14:45Z

test parameters:

enable_tests = gpu
workflows_gpu = 29634.704,29834.704
workflows = 29634.703,29834.703,29834.755,29634.757,29834.757
relvals_opt = -w upgrade,standard
relvals_opt_gpu = -w upgrade,standard

cmsbuild · 2025-07-08T23:15:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48508/45442

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): CA Extension to strips #47090

cmsbuild · 2025-07-08T23:16:12Z

A new Pull Request was created by @slava77 for master.

It involves the following packages:

Configuration/PyReleaseValidation (upgrade, pdmv)
DQM/TrackingMonitorClient (dqm)
DQM/TrackingMonitorSource (dqm)
RecoTracker/IterativeTracking (reconstruction)

@AdrianoDee, @Moanwar, @antoniovagnerini, @cmsbuild, @ctarricone, @DickyChant, @jfernan2, @mandrenguyen, @miquork, @rseidita, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @Martin-Grunewald, @VinInn, @VourMa, @arossi83, @dgulhan, @fabiocos, @felicepantaleo, @fioriNTU, @gpetruc, @idebruyn, @jandrea, @makortel, @missirol, @mmusich, @mtosi, @richa2710, @rovere, @slomeo, @sroychow, @threus this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

slava77 · 2025-07-08T23:16:27Z

@cmsbuild please test

cmsbuild · 2025-07-09T03:35:32Z

-1

Failed Tests: RelVals rocmUnitTests
Size: This PR adds an extra 56KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8c869e/47138/summary.html
COMMIT: 3d93e7e
CMSSW: CMSSW_15_1_X_2025-07-08-2300/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48508/47138/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

29834.757

----- Begin Fatal Exception 09-Jul-2025 04:18:05 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=MergeClusterProducer label='hltMergeLayerClusters'
Exception Message:
Illegal parameters found in configuration.  The parameters are named:
 'layerClustersEE'
 'time_layerclustersEE'
You could be trying to use parameter names that are not
allowed for this plugin or they could be misspelled.
----- End Fatal Exception -------------------------------------------------

29834.755

----- Begin Fatal Exception 09-Jul-2025 04:18:05 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=MergeClusterProducer label='hltMergeLayerClusters'
Exception Message:
Illegal parameters found in configuration.  The parameters are named:
 'layerClustersEE'
 'time_layerclustersEE'
You could be trying to use parameter names that are not
allowed for this plugin or they could be misspelled.
----- End Fatal Exception -------------------------------------------------

29634.757

----- Begin Fatal Exception 09-Jul-2025 04:18:04 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=MergeClusterProducer label='hltMergeLayerClusters'
Exception Message:
Illegal parameters found in configuration.  The parameters are named:
 'layerClustersEE'
 'time_layerclustersEE'
You could be trying to use parameter names that are not
allowed for this plugin or they could be misspelled.
----- End Fatal Exception -------------------------------------------------

ROCm Unit Tests

I found 0 errors in the following unit tests:

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 183 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 117626
DQMHistoTests: Total failures: 14384
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 103242
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 9638.794 KiB( 8 files compared)
DQMHistoSizes: changed ( 29634.704,... ): 4819.397 KiB Tracking/TrackBuilding
Checked 32 log files, 36 edm output root files, 9 DQM output files
TriggerResults: found differences in 1 / 8 workflows

ROCM Comparison Summary

Summary:

You potentially added 5 lines to the logs
Reco comparison results: 149 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 117626
DQMHistoTests: Total failures: 13183
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 104443
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 9638.794 KiB( 8 files compared)
DQMHistoSizes: changed ( 29634.704,... ): 4819.397 KiB Tracking/TrackBuilding
Checked 32 log files, 36 edm output root files, 9 DQM output files
TriggerResults: no differences found

mmusich · 2025-07-09T06:26:41Z

-1

failures are unrelated see #47859 (comment)

slava77 · 2025-07-14T17:59:03Z

@cmsbuild please test

cmsbuild · 2025-07-14T20:25:53Z

-1

Failed Tests: rocmUnitTests
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8c869e/47220/summary.html
COMMIT: 3d93e7e
CMSSW: CMSSW_15_1_X_2025-07-14-1100/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48508/47220/install.sh to create a dev area with all the needed externals and cmssw changes.

ROCm Unit Tests

I found 0 errors in the following unit tests:

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 57
DQMHistoTests: Total histograms compared: 4447845
DQMHistoTests: Total failures: 4
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4447821
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 9634.59 KiB( 56 files compared)
DQMHistoSizes: changed ( 29634.703,... ): 4817.295 KiB Tracking/TrackBuilding
Checked 240 log files, 206 edm output root files, 57 DQM output files
TriggerResults: no differences found

CUDA Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 175 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 117626
DQMHistoTests: Total failures: 12871
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 104755
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 9638.794 KiB( 8 files compared)
DQMHistoSizes: changed ( 29634.704,... ): 4819.397 KiB Tracking/TrackBuilding
Checked 32 log files, 36 edm output root files, 9 DQM output files
TriggerResults: no differences found

ROCM Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 134 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 117626
DQMHistoTests: Total failures: 10754
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 106872
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 9638.794 KiB( 8 files compared)
DQMHistoSizes: changed ( 29634.704,... ): 4819.397 KiB Tracking/TrackBuilding
Checked 32 log files, 36 edm output root files, 9 DQM output files
TriggerResults: no differences found

jfernan2 · 2025-07-25T16:26:19Z

+1

Moanwar · 2025-07-27T08:12:56Z

+Upgrade

slava77 · 2025-07-31T12:38:41Z

it would be nice to get some feedback (or an expectation of when it may be coming)

@cms-sw/dqm-l2 @cms-sw/pdmv-l2 signatures are missing

I'm not sure still if @cms-sw/heterogeneous-l2 is going to assign this PR

fwyzard · 2025-07-31T12:47:08Z

From a cursory look I don't think we need to assign to and sign for @cms-sw/heterogeneous-l2 .

slava77 · 2025-08-04T22:27:17Z

it would be nice to get some feedback (or an expectation of when it may be coming)

@cms-sw/dqm-l2 @cms-sw/pdmv-l2 signatures are missing

@cms-sw/orp-l2 this was open for 4 weeks
without comments/feedback from DQM and PDMV. Are the L2s away or is there an issue with this PR?

AdrianoDee · 2025-08-11T08:46:42Z

+pdmv

slava77 · 2025-08-12T16:19:46Z

@cms-sw/dqm-l2 (now also directly @antoniovagnerini @rseidita @ctarricone)
your signature or comments are needed.
Please clarify on the status of the review of this PR.
Thank you

rseidita · 2025-08-12T17:37:43Z

+dqm

cmsbuild · 2025-08-12T17:38:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-08-13T06:24:16Z

+1

add trackToTrack comparisons for GPU vs CPU tracking, populated for t…

3d93e7e

…rackingLST

cmsbuild added this to the CMSSW_15_1_X milestone Jul 8, 2025

cmsbuild added reconstruction-pending dqm-pending pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending code-checks-pending tracking labels Jul 8, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 8, 2025

cmsbuild added tests-started and removed tests-pending labels Jul 8, 2025

cmsbuild added tests-rejected and removed tests-started labels Jul 9, 2025

cmsbuild added tests-started and removed tests-rejected labels Jul 14, 2025

cmsbuild added tests-rejected and removed tests-started labels Jul 14, 2025

cmsbuild added the tests-approved label Jul 25, 2025

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jul 25, 2025

cmsbuild added upgrade-approved and removed upgrade-pending labels Jul 27, 2025

This was referenced Aug 6, 2025

Add a relval test for the NGT scouting menu with multiple process modifiers #48687

Merged

add ageing customization to ph2_hlt workflows #48695

Merged

HLT:75e33_trackingOnly and a related reference workflow #48706

Merged

cmsbuild added pdmv-approved and removed pdmv-pending labels Aug 11, 2025

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Aug 12, 2025

cmsbuild added orp-approved and removed orp-pending labels Aug 13, 2025

cmsbuild merged commit d63a45c into cms-sw:master Aug 13, 2025
17 checks passed

VourMa mentioned this pull request Aug 13, 2025

Follow-up tasks for the LST algorithm #46746

Open

cmsbuild mentioned this pull request Aug 13, 2025

Adapt HGCal rechit producer for HGCal testbeam data #48606

Merged

fwyzard mentioned this pull request Aug 15, 2025

[GPU] No accelerator for cuda_async/rocm_async #48743

Closed

LST: add trackToTrack comparisons for GPU vs CPU tracking #48508

LST: add trackToTrack comparisons for GPU vs CPU tracking #48508

Uh oh!

Conversation

slava77 commented Jul 8, 2025

Uh oh!

cmsbuild commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slava77 commented Jul 8, 2025

Uh oh!

cmsbuild commented Jul 8, 2025

Uh oh!

cmsbuild commented Jul 8, 2025

Uh oh!

slava77 commented Jul 8, 2025

Uh oh!

cmsbuild commented Jul 9, 2025

RelVals

ROCm Unit Tests

CUDA Comparison Summary

ROCM Comparison Summary

Uh oh!

mmusich commented Jul 9, 2025

Uh oh!

slava77 commented Jul 14, 2025

Uh oh!

cmsbuild commented Jul 14, 2025

ROCm Unit Tests

Comparison Summary

CUDA Comparison Summary

ROCM Comparison Summary

Uh oh!

jfernan2 commented Jul 25, 2025

Uh oh!

Moanwar commented Jul 27, 2025

Uh oh!

slava77 commented Jul 31, 2025

Uh oh!

fwyzard commented Jul 31, 2025

Uh oh!

slava77 commented Aug 4, 2025

Uh oh!

AdrianoDee commented Aug 11, 2025

Uh oh!

slava77 commented Aug 12, 2025

Uh oh!

rseidita commented Aug 12, 2025

Uh oh!

cmsbuild commented Aug 12, 2025

Uh oh!

mandrenguyen commented Aug 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

cmsbuild commented Jul 8, 2025 •

edited

Loading