Fix PU for 2026 wfs [16_0_X]

[AdrianoDee]

PR description:

Plain backport of #49958

[AdrianoDee]

please test

[AdrianoDee]

type bug-fix

[cmsbuild]

A new Pull Request was created by @AdrianoDee for CMSSW_16_0_X.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv)

@AdrianoDee, @DickyChant, @antoniovagnerini, @miquork can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

-1

Failed Tests: Build
Size: This PR adds an extra 68KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ae6085/50956/summary.html
COMMIT: 0159c51
CMSSW: CMSSW_16_0_X_2026-01-27-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49959/50956/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @m-glowacki [Backport to #49869] Adding HLT layer of Anomaly Detection #49939

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ae6085/50956/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ae6085/50956/git-merge-result

Failed Build

I found compilation error when building:

*** Error compiling 'src/Configuration/PyReleaseValidation/python/relval_steps.py'...
Sorry: TabError: inconsistent use of tabs and spaces in indentation (relval_steps.py, line 4898)
gmake[1]: *** [config/SCRAM/GMake/Makefile.rules:2059: CompilePython] Error 1
gmake[1]: Target 'PostBuild' not remade because of errors.
gmake[1]: Leaving directory '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_16_0_X_2026-01-27-2300'
gmake: *** [config/SCRAM/GMake/Makefile.rules:1942: src] Error 2
gmake: Target 'all' not remade because of errors.
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2
Command exited with non-zero status 2
	Command being timed: "scram build -v -k -j 16"
	User time (seconds): 393.48


[AdrianoDee]

please test

[cmsbuild]

Pull request #49959 was updated. @AdrianoDee, @DickyChant, @antoniovagnerini, @miquork can you please check and sign again.

[cmsbuild]

-1

Failed Tests: RelVals
Size: This PR adds an extra 68KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ae6085/50958/summary.html
COMMIT: f37ffb6
CMSSW: CMSSW_16_0_X_2026-01-27-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49959/50958/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

• 14234.0

----- Begin Fatal Exception 28-Jan-2026 13:10:18 CET-----------------------
An exception of category 'NoSecondaryFiles' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

[AdrianoDee]

@smuzaffar something strange happened, and it seems to me it's connected to an unexpected (?) behavior of the bot. Please let me know if I'm wrong. In 14234.0 the step2 log shows an empyt pileup input

SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,PAT,NANO,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQMFS+@miniAODDQM+@nanoAODDQM
We have determined that this is simulation (if not, rerun cmsDriver.py with --data)
entry file:step1.root
the query is file dataset = /RelValMinBias_14TeV/CMSSW_13_0_11-130X_mcRun3_2023_realistic_withEarly2023BS_v1_FastSim-v1/GEN-SIM
DAS succeeded after 1 attempts 0
found files:  []
Step: SIM Spec: 
Step: RECOBEFMIX Spec: 
Step: DIGI Spec: ['pdigi_valid']

And this is normal since the RelVal has long gone. But, this dataset should have been cached by the bot a long time ago (it was not touched here). Do you understand why the mechanism is not working here?

[AdrianoDee]

please test

[cmsbuild]

Pull request #49959 was updated. @AdrianoDee, @DickyChant, @antoniovagnerini, @miquork can you please check and sign again.

[AdrianoDee]

@cms-sw/orp-l2, if this could enter in 16_0_0, it would be great (I should have solved the issue causing the failures)

[AdrianoDee]

The 2026 tests have the right configuration now https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ae6085/50983/runTheMatrix-results/18634.0_TTbar_14TeV+2026PU/cmdLog

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_16_0_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_16_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[cmsbuild]

+1

Size: This PR adds an extra 68KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ae6085/50983/summary.html
COMMIT: 1a15330
CMSSW: CMSSW_16_0_X_2026-01-28-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49959/50983/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 9 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 3439 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4149103
• DQMHistoTests: Total failures: 82412
• DQMHistoTests: Total nulls: 14
• DQMHistoTests: Total successes: 4066657
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 12.305 KiB( 52 files compared)
• DQMHistoSizes: changed ( 18634.0 ): 12.305 KiB SiStrip/MechanicalView
• Checked 227 log files, 198 edm output root files, 53 DQM output files
• TriggerResults: found differences in 2 / 51 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 2 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 18634.0_TTbar_14TeV+2026PU step2 max memory diff 575.0 exceeds +/- 90.0 MiB
• Error: Workflow 18634.0_TTbar_14TeV+2026PU step3 max memory diff 882.4 exceeds +/- 90.0 MiB

[mandrenguyen]

+1