Pythia 8.315

[mseidel42]

Same updates as for Pythia 8.313 (#9612), specifically

• updated Hadronization Variations
• fixed bugs in Vincia merging, recoilStrategyRF, and QCD Colour-Reconnection model

Full changelog here: https://pythia.org/latest-manual/UpdateHistory.html#section0

[cmsbuild]

A new Pull Request was created by @mseidel42 for branch IB/CMSSW_15_1_X/master.

@akritkbehera, @cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[mseidel42]

Needs cms-sw/cmssw#48422

[smuzaffar]

test parameters:

• pull_requests = Modifications for Pythia 8.315 cmssw#48422

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13f81f/46954/summary.html
COMMIT: dbc8a1d
CMSSW: CMSSW_15_1_X_2025-06-26-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9943/46954/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw Update tag for GeneratorInterface-EvtGenInterface to V02-12-00 #9946

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13f81f/46954/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13f81f/46954/git-merge-result

RelVals

• 29634.75

----- Begin Fatal Exception 27-Jun-2025 13:32:05 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'validation_step'
   [2] Prefetching for module HGCalValidator/'hltHgcalValidator'
   [3] Calling method for module SimTrackstersProducer/'hltTiclSimTracksters'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<TrackingParticle>,edm::View<reco::Track>,double,unsigned int,edm::RefProd<std::vector<TrackingParticle> >,edm::RefToBaseProd<reco::Track>,edm::Ref<std::vector<TrackingParticle>,TrackingParticle,edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle>,TrackingParticle> >,edm::RefToBase<reco::Track> > >
Looking for module label: tpToHltGeneralTrackAssociation
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

[mseidel42]

GEN-SIM step ran fine as far as I can see

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13f81f/47002/summary.html
COMMIT: dbc8a1d
CMSSW: CMSSW_15_1_X_2025-06-29-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9943/47002/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 29634.75

----- Begin Fatal Exception 30-Jun-2025 11:51:53 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'validation_step'
   [2] Prefetching for module HGCalValidator/'hltHgcalValidator'
   [3] Calling method for module SimTrackstersProducer/'hltTiclSimTracksters'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::AssociationMap<edm::OneToManyWithQualityGeneric<std::vector<TrackingParticle>,edm::View<reco::Track>,double,unsigned int,edm::RefProd<std::vector<TrackingParticle> >,edm::RefToBaseProd<reco::Track>,edm::Ref<std::vector<TrackingParticle>,TrackingParticle,edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle>,TrackingParticle> >,edm::RefToBase<reco::Track> > >
Looking for module label: tpToHltGeneralTrackAssociation
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

[mseidel42]

Hi, could you ping someone from TICL to have a look at this?
I have no idea why their truth information fails with the new Pythia version...

[makortel]

@cmsbuild, please test

Let's refresh the tests

[mandrenguyen]

Unfortunately it's not straightforward as it might seem, especially because we have to find person power to do the validation work.

Anyway, if it's possible to have a special release for this that would be great. Otherwise we can use a pre-release.

It's possible, but is the plan really to validate with a central production campaign? Because if we end up just doing private production anyway, then this would be wasted effort.

[lviliani]

Unfortunately it's not straightforward as it might seem, especially because we have to find person power to do the validation work.
Anyway, if it's possible to have a special release for this that would be great. Otherwise we can use a pre-release.

It's possible, but is the plan really to validate with a central production campaign? Because if we end up just doing private production anyway, then this would be wasted effort.

It's an idea that we are considering for this kind of validations.
But can you clarify why it would be wasted effort? The only difference I see in the two approaches is that a pre-release will automatically end up in an official release, while a special pythia8315 release will not. But please correct me if I'm wrong, it might very well be that I'm missing something obvious.
Regarding the usage of pre-releases in mcm we will follow-up with PPD to ask if it's possible for validation campaigns.

[theofil]

The key point is that it’s much easier to find people willing to perform validation if there’s a “central” but purpose-specific CMS release that can be used with McM for producing validation samples. Expecting individuals to handle both MC generation and the subsequent validation studies on their own—and trusting that this will be completed in time before January—is unrealistic. At the moment, no one is actually committed to carrying out validation work for the Pythia upgrade.

So, what’s the benefit of including it in the pre-release now, if we’ll just need to remove it again before January 2026?

[mandrenguyen]

The benefit is that you have a pre-release to validate. Which is the point of pre-releases! In principle any feature included in a prerelease may have to be rolled back, if the validation is not successful in time. We can of course treat generators separately, if it's warranted. It creates more work, but if there's a strong reason why you can't simply benefit from pre-releases, we can discuss it

[lviliani]

The problem is that pre-releases have a short lifetime as far as I know, so it's tricky to use them in a central production campaign in McM.

Also we should bear in mind that, if there's any issue with the new version of pythia (or any other generator in CMSSW) all the relvals that are run to validate the pre-releases could potentially fail because of this.

[theofil]

I do not see a clear advantage of a pre-release compared to a dedicated pythia8315 release for validation. In both cases, private samples would still need to be produced by individuals, which slows down the process and makes validation less appealing for volunteers. A pre-release also carries the risk of prematurely entering the main release cycle, which could result in large-scale production before validation is complete and effectively leave CMS analyzers acting as a posteriori validators.

A more robust approach would be to provide a dedicated release (e.g. release-X) that can be used directly with McM for producing validation samples. This release would remain outside the main release pipeline until validation is finalized, thereby avoiding premature large-scale MC production.

[mandrenguyen]

hold
Ok, you can request a dedicated release at the next ORP meeting.

[cmsbuild]

Pull request has been put on hold by @mandrenguyen
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

[mandrenguyen]

I would really encourage @cms-sw/generators-l2 to make use 16_0_0_pre1 for pythia 8.315 in this instance AND have a discussion about dedicated gen releases going forward at the next ORP. I don't think we should hold up pythia 8.315 until we have settled on a broader strategy. But this is your call.
I would also like to point out that gen should really be represented at the ORP regularly.

[mseidel42]

Fwiw, we have a student coming to CERN this month who was supposed to do Vincia validation.
We can have him validate Pythia 8.315 vs the previous version first instead.

[makortel]

The problem is that pre-releases have a short lifetime as far as I know,

No, pre-releases are kept "forever".

I do not see a clear advantage of a pre-release compared to a dedicated pythia8315 release for validation.

Pre-release approach follows the standard procedure everyone else follows, whereas special releases create extra work for those who prepare that release. Special releases can be done, and the amount of work is not that much, but they do need to be well motivated.

[mandrenguyen]

My hesitation is that generator version changes can be frequent, and this creates extra work for the core and ORP teams, which can delay generator integration (although at the current pace perhaps the extra delay is negligible). It's possible, but there is a cost. Following the release schedule imposes deadlines (a good thing) and helps makes sure that generator developments stay in sync with other developments.

[lviliani]

No, pre-releases are kept "forever".

Ok, thanks for the info. Sorry, I was not aware of this.

I guess we could try using a pre-release, but I would like to clarify that the validation of a new generator is totally different from all the other release validations that we usually do.
It can take a long time, depending on the potential issues that are found, so it's very likely that we will have to roll back before the official release is out.

I can imagine that if we extend this kind of validation to multiple new generators (Herwig, Sherpa, ...), the current approach will most likely fail.

[makortel]

"Taking long time" could be a good-enough reason to motivate special (pre-)releases. That's not too different with what we do for GCC and ROOT, although for them the primary motivation of a special validation is the huge amount of numerical changes either update usually creates, and the long validation time is more of a matter of reality.

I'll let @smuzaffar to comment on the feasibility on practical level.

[lviliani]

Hello, after discussing with @mseidel42 and @theofil, we decided to include this in a pre-release this time. We will check how the validation will go and in case consider a stable release for the next time.

[lviliani]

+generators

[iarspider]

+externals

[mandrenguyen]

unhold

Hello, after discussing with @mseidel42 and @theofil, we decided to include this in a pre-release this time. We will check how the validation will go and in case consider a stable release for the next time.

We are happy to have that discussion. Probably folks from PPD/PdMV should be in the loop.

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_16_0_X/master IBs (tests are also fine). This pull request will be automatically merged.
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cmssw#48422

[AdrianoDee]

We are happy to have that discussion. Probably folks from PPD/PdMV should be in the loop.

Hi, sorry I missed this discussion. Ideally, the special pre-release method (as Matti was suggesting, and as we do for ROOT) is the preferred solution since any change would most probably be hidden below all the other changes occurring in the standard pre-release cycle, unless they are very catastrophic, which I hope is never the case. If we can stick to ~1 change of version per year (as it seems to be the case, looking at the PRs history), this would be bearable.