Phase2-hgx364X Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment #49763

bsunanda · 2026-01-11T13:53:24Z

PR description:

Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment

PR validation:

Tested for the diagnostic

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Nothing special

…ong DetID->Position->DetID assignment

cmsbuild · 2026-01-11T13:53:47Z

cms-bot internal usage

cmsbuild · 2026-01-11T13:54:55Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49763/47353

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49763/47353/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49763/47353/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2026-01-11T13:59:07Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49763/47354

cmsbuild · 2026-01-11T13:59:31Z

A new Pull Request was created by @bsunanda for master.

It involves the following packages:

Geometry/HGCalCommonData (geometry)

@Dr15Jones, @bsunanda, @civanch, @cmsbuild, @kpedro88, @makortel, @mdhildreth can you please review it and eventually sign? Thanks.
@fabiocos, @martinamalberti this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

bsunanda · 2026-01-11T14:00:19Z

@cmsbuild Please test

bsunanda · 2026-01-11T14:28:33Z

@cmsbuild Please test

cmsbuild · 2026-01-11T15:52:35Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d20885/50505/summary.html
COMMIT: f31de82
CMSSW: CMSSW_16_1_X_2026-01-11-0000/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49763/50505/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

34634.999

----- Begin Fatal Exception 11-Jan-2026 16:26:31 CET-----------------------
An exception of category 'OutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'HLTriggerFinalPath'
   [2] Prefetching for module TriggerSummaryProducerAOD/'hltTriggerSummaryAOD'
   [3] Prefetching for module L1HPSPFTauProducer/'l1tHPSPFTauProducer'
   [4] Prefetching for module L1TPFCandMultiMerger/'l1tLayer1'
   [5] Prefetching for module L1TCorrelatorLayer1Producer/'l1tLayer1HGCal'
   [6] Calling method for module HGCalBackendLayer2Producer/'l1tHGCalBackEndLayer2Producer'
Exception Message:
TC X1 = 0.0713466 out of the seeding histogram bounds 0.076 - 0.58
----- End Fatal Exception -------------------------------------------------

bsunanda · 2026-01-12T02:03:40Z

This error is not due to the changes of the PR. It needs investigation

bsunanda · 2026-01-12T02:03:47Z

+geometry

cmsbuild · 2026-01-12T02:04:01Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-01-12T08:07:12Z

The exception is coming from this line:

cmssw/L1Trigger/L1THGCal/src/backend/HGCalHistoSeedingImpl.cc

Line 103 in fc98c65

    
           throw cms::Exception("OutOfBound") << "TC X1 = " << x1 << " out of the seeding histogram bounds " << minx1

It's not clear to me why this is only popping up in this PR and not others. Would you expect a change in geometry could cause us to hit the error by accident?

bsunanda · 2026-01-12T09:43:42Z

This PR is affecting the rebuilding of geometry which triggers this test. But the changes in this PR is addition of a few diagnostic messages which cannot cause the error. This error was reported earlier and needs investigation.

…

________________________________ From: Matthew Nguyen ***@***.***> Sent: 12 January 2026 13:37 To: cms-sw/cmssw ***@***.***> Cc: Sunanda Banerjee ***@***.***>; Mention ***@***.***> Subject: Re: [cms-sw/cmssw] Phase2-hgx364X Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment (PR #49763) [https://avatars.githubusercontent.com/u/5022810?s=20&v=4]mandrenguyen left a comment (cms-sw/cmssw#49763)<#49763 (comment)> The exception is coming from this line: https://github.com/cms-sw/cmssw/blob/fc98c6521423acce9e64292eee60586ed64418b1/L1Trigger/L1THGCal/src/backend/HGCalHistoSeedingImpl.cc#L103 It's not clear to me why this is only popping up in this PR and not others. Would you expect a change in geometry could cause us to hit the error by accident? — Reply to this email directly, view it on GitHub<#49763 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGMZOQJM2BGX77QJHQ4JTD4GNI4PAVCNFSM6AAAAACRKWCNVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMZXGMYTSOBQGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

makortel · 2026-01-12T14:37:53Z

The exception is coming from this line:

cmssw/L1Trigger/L1THGCal/src/backend/HGCalHistoSeedingImpl.cc

Line 103 in fc98c65

throw cms::Exception("OutOfBound") << "TC X1 = " << x1 << " out of the seeding histogram bounds " << minx1

It's not clear to me why this is only popping up in this PR and not others. Would you expect a change in geometry could cause us to hit the error by accident?

Is #41451 back?

bsunanda · 2026-01-12T15:19:50Z

Is the pileup file made with the same CMSSW version - I mean the IB + current PR?

…

________________________________ From: Matti Kortelainen ***@***.***> Sent: 12 January 2026 20:08 To: cms-sw/cmssw ***@***.***> Cc: Sunanda Banerjee ***@***.***>; Mention ***@***.***> Subject: Re: [cms-sw/cmssw] Phase2-hgx364X Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment (PR #49763) [https://avatars.githubusercontent.com/u/33993?s=20&v=4]makortel left a comment (cms-sw/cmssw#49763)<#49763 (comment)> The exception is coming from this line: https://github.com/cms-sw/cmssw/blob/fc98c6521423acce9e64292eee60586ed64418b1/L1Trigger/L1THGCal/src/backend/HGCalHistoSeedingImpl.cc#L103 It's not clear to me why this is only popping up in this PR and not others. Would you expect a change in geometry could cause us to hit the error by accident? Is #41451<#41451> back? — Reply to this email directly, view it on GitHub<#49763 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGMZOTIF3GH4P7OUU2UVTD4GOWVRAVCNFSM6AAAAACRKWCNVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMZYHA3DSOJUGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

makortel · 2026-01-12T15:26:59Z

Is the pileup file made with the same CMSSW version - I mean the IB + current PR?

The premixed pileup file itself is done with the same CMSSW version, but MinBias files used as an input for the premixing are read from storage (I see from CMSSW_16_0_0_pre2). If that mismatch is the problem, then we just have to accept the failures until a new MinBias dataset is produced.

(it would be great if someone would have time to craft another variant of the workflow that would generate the MinBias on the fly)

kpedro88 · 2026-01-12T16:18:15Z

(it would be great if someone would have time to craft another variant of the workflow that would generate the MinBias on the fly)

Actually, these exist for alternative pileup options (premixing and hybrid), just not for classical mixing... @makortel could you open an issue for this?

makortel · 2026-01-12T17:22:10Z

(it would be great if someone would have time to craft another variant of the workflow that would generate the MinBias on the fly)

Actually, these exist for alternative pileup options (premixing and hybrid), just not for classical mixing... @makortel could you open an issue for this?

I can open an issue, but just to clarify, I was talking about premixing. While the X.999 does the premixing on the fly, it relies on existing MinBias files.

makortel · 2026-01-12T17:23:59Z

(it would be great if someone would have time to craft another variant of the workflow that would generate the MinBias on the fly)

Actually, these exist for alternative pileup options (premixing and hybrid), just not for classical mixing... @makortel could you open an issue for this?

I can open an issue, but just to clarify, I was talking about premixing. While the X.999 does the premixing on the fly, it relies on existing MinBias files.

I opened #49776 for the case I had in mind. I can extend the issue to classical mixing if you want (but maybe we should move that discussion to the issue)

kpedro88 · 2026-01-12T17:38:15Z

@makortel thanks for clarifying. Yes, I would include both in the same issue.

mandrenguyen · 2026-01-12T18:50:30Z

+1
As discussed above the failure stems from the input file and gets triggered by the change in geometry. has nothing to do with this PR.

mandrenguyen · 2026-01-12T21:32:34Z

merge

akritkbehera · 2026-01-13T09:03:17Z

Some compilation errors were found in CMSSW_16_1_UBSAN_X IBs after this PR
constexpr std::string compilation error in HGCalTypes.h

cmssw/Geometry/HGCalCommonData/interface/HGCalTypes.h

Lines 168 to 173 in c828765

    
             static constexpr std::string waferType_[4] = {"HD120", "LD200", "LD300", "HD200"}; 
        
             static constexpr std::string waferTypeX_[27] = { 
        
                 "Full",      "Five",      "ChopTwo",   "ChopTwoM", "Half",     "Semi",    "Semi2",   "Three",   "Half2", 
        
                 "Five2",     "Unknown10", "LDTop",     "LDBottom", "LDLeft",   "LDRight", "LDFive",  "LDThree", "Unknown17", 
        
                 "Unknown18", "Unknown19", "Unknown20", "HDTop",    "HDBottom", "HDLeft",  "HDRight", "HDFive",  "Out"}; 
        
           };

In file included from src/Geometry/HGCalCommonData/interface/HGCalDDDConstants.h:26,
                 from src/CondFormats/HGCalObjects/interface/HeterogeneousHGCalHEFCellPositionsConditions.h:11,
                 from src/CondFormats/HGCalObjects/src/T_EventSetup_HeterogeneousHGCalHEFCellPositionsConditions.cc:1:
  src/Geometry/HGCalCommonData/interface/HGCalTypes.h:167:85: error: '(((const std::string*)(& HGCalTypes::layerTypes_)) != 0)' is not a constant expression
   167 |       "Center", "CenterB", "CenterYp", "CenterYm", "CenterR", "CenterXp", "CenterXm"};
      |                                                                                     ^
  src/Geometry/HGCalCommonData/interface/HGCalTypes.h:168:83: error: '(((const std::string*)(& HGCalTypes::waferType_)) != 0)' is not a constant expression
   168 |   static constexpr std::string waferType_[4] = {"HD120", "LD200", "LD300", "HD200"};
      |                                                                                   ^
  src/Geometry/HGCalCommonData/interface/HGCalTypes.h:172:108: error: '(((const std::string*)(& HGCalTypes::waferTypeX_)) != 0)' is not a constant expression
   172 |       "Unknown18", "Unknown19", "Unknown20", "HDTop",    "HDBottom", "HDLeft",  "HDRight", "HDFive",  "Out"};

mmusich · 2026-01-15T08:43:37Z

As discussed above the failure stems from the input file and gets triggered by the change in geometry. has nothing to do with this PR.

On the contrary, I think this PR is the primary cause of the recent turbulence in IBs.
For example in the test revert #49831, the GPU matrix passed mostly unscathed (while it fails elsewhere).
Can we consider a revert?
@bsunanda @mandrenguyen

Add the tools with some more diagnostic help to debug the issue of wr…

24b796c

…ong DetID->Position->DetID assignment

cmsbuild added this to the CMSSW_16_1_X milestone Jan 11, 2026

cmsbuild added geometry-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jan 11, 2026

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 11, 2026

Code check

f31de82

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 11, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 11, 2026

cmsbuild added tests-started and removed tests-pending labels Jan 11, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 11, 2026

cmsbuild added geometry-approved and removed geometry-pending pending-signatures labels Jan 12, 2026

cmsbuild added the fully-signed label Jan 12, 2026

cmsbuild added orp-approved and removed orp-pending labels Jan 12, 2026

cmsbuild merged commit 2cb07a0 into cms-sw:master Jan 12, 2026
9 of 10 checks passed

akritkbehera mentioned this pull request Jan 13, 2026

Use string_view to fix constexpr std::string comp error in UBSAN_X IBs #49792

Open

makortel mentioned this pull request Jan 13, 2026

RelVals 34551.85, 34634.XX failing with OutOfBound error #49795

Open

mmusich mentioned this pull request Jan 14, 2026

[DO NOT MERGE ]Revert "Phase2-hgx364X Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment" #49831

Draft

Phase2-hgx364X Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment #49763

Phase2-hgx364X Add the tools with some more diagnostic help to debug the issue of wrong DetID->Position->DetID assignment #49763

Uh oh!

Conversation

bsunanda commented Jan 11, 2026

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 11, 2026

Uh oh!

cmsbuild commented Jan 11, 2026

Uh oh!

cmsbuild commented Jan 11, 2026

Uh oh!

bsunanda commented Jan 11, 2026

Uh oh!

bsunanda commented Jan 11, 2026

Uh oh!

cmsbuild commented Jan 11, 2026

Failed RelVals

Uh oh!

bsunanda commented Jan 12, 2026

Uh oh!

bsunanda commented Jan 12, 2026

Uh oh!

cmsbuild commented Jan 12, 2026

Uh oh!

mandrenguyen commented Jan 12, 2026

Uh oh!

bsunanda commented Jan 12, 2026 via email

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

bsunanda commented Jan 12, 2026 via email

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

kpedro88 commented Jan 12, 2026

Uh oh!

makortel commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

kpedro88 commented Jan 12, 2026

Uh oh!

mandrenguyen commented Jan 12, 2026

Uh oh!

mandrenguyen commented Jan 12, 2026

Uh oh!

Uh oh!

akritkbehera commented Jan 13, 2026

Uh oh!

mmusich commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cmsbuild commented Jan 11, 2026 •

edited

Loading

makortel commented Jan 12, 2026 •

edited

Loading