SimTrack ID exceeding maximum values

[DickyChant]

During a recent production of upward-going muons with high momentum from cosmic muon generator, we see massive failures, notably suffering from the following errors:

Fatal Exception (Exit code: 8001)
An exception of category 'SimTrackManager::reallyStoreTracks' occurred while
[0] Processing Event run: 1 lumi: 89 event: 6231740 stream: 3
[1] Running path 'RAWSIMoutput_step'
[2] Prefetching for module PoolOutputModule/'RAWSIMoutput'
[3] Calling method for module OscarMTProducer/'g4SimHits'
Exception Message:
SimTrack ID 200001019 exceeds maximum allowed by PSimHit identifier200000000

A deck of requests having this issue can be found on McM, with several examples as:

https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_EXO-RunIII2024Summer24GS-00613
https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_EXO-RunIII2024Summer24GS-00665

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @DickyChant.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign simulation

[cmsbuild]

New categories assigned: simulation

@civanch,@kpedro88,@mdhildreth you have been requested to review this Pull request/Issue and eventually sign? Thanks

[tvami]

I see this offset was added in PR #42506
where in the description I read

The previous offset of 100000000 is doubled to 200000000 , in order to stay safe also in presence of Heavy Ion samples

So that number seems somewhat arbitrary.

[kpedro88]

@fabiocos can you explain the purpose/usage of k_tidOffset? (I only find it in MtdTruthAccumulator, but PSimHit is also used by other subdetectors...)

[fabiocos]

@kpedro88 I added a possible offset in PSimHit trackId so as to classify hits without the need to introduce further words. There are methods to retrieve both the offset (modulo the fixed constant multiplier) and the genuine Geant4 track id. This tool is used to enable the classification of MTD hits categories, which we discussed several times in various meetings, but it might be used by anybody.

I had chosen a large enough integer constant so as to exceed what I had found a possible Geant4 track Id in some HIN events. I am puzzled that a cosmic muon is producing such a large number of Geant4 secondaries.

[fabiocos]

The simple-minded solution is to replace this mechanism with the addition of a dedicated word to PSimHit. Given that this mechanism is used for the time being by just MTD, and not all the hits have a non-zero type, it looks like a waste of space, but of course it prevents such a possible problem. Honestly cosmics are not the place where I was expecting such issue, but may be, being upward-going, they are not simple muons but massive showers? Producing a secondary multiplicity larger than in a HIN event? Do we have an estimate of the total number of secondaries produce in such an event?

[fabiocos]

Actually I see a possible alternative keeping the present scheme, but replacing multiplicative with additive kind of types, i.e.: set an offset as something very large, not far from the numerical limit of uint32_t, and decrease it by one unit for each successive type. But before any possible solution, it would be important to understand the scenario that is producing the issue.

[kpedro88]

@fabiocos I will try to quickly inspect one of the workflows to see how many secondaries they can produce.

[kpedro88]

@DickyChant do you have the PSet for a specific failing job? Running 5K events directly from https://cms-pdmv-prod.web.cern.ch/mcm/public/restapi/requests/get_setup/EXO-RunIII2024Summer24GS-00613 did not trip the exception.