Make `--secondfilein` and on-the-fly pileup workflows to use RNTuple when `--use-rntuple` is specified #49674

makortel · 2025-12-18T19:22:28Z

PR description:

This PR makes the runTheMatrix.py workflows that use --secondfilein and generate premixed pileup on the fly to work with the RNTupleTemp* IO components (i.e. with runTheMatrix.py --use-rntuple).

In the --secondfilein case the workflows that use it use files from DAS query, i.e. the files exist already and are in TTree format. Since we can't (presently) use different file formats for primary and secondary input files, this PR makes those workflows use TTree file format until the last step that uses --secondfilein.

In the premixed pileup case this PR makes the MixingModules to use EmbeddedRNTupleTempSource. The way I did it is a hit hacky, but given that the use of EmbeddedRootSource is coded in all the MixingModule definitions along

cmssw/SimGeneral/MixingModule/python/HiEventMixing_cff.py

Line 22 in b0efeef

input = cms.SecSource("EmbeddedRootSource",

it seemed difficult to do anything else without overhauling lot's of existing code.

Resolves cms-sw/framework-team#1633
Resolves cms-sw/framework-team#1728

PR validation:

runTheMatrix.py --limited --use-rntuple succeeded (except for some workflows that had problems in reading their remote input files)

cmsbuild · 2025-12-18T19:22:57Z

cms-bot internal usage

cmsbuild · 2025-12-18T19:24:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49674/47227

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Move dqmoffline_step from EndPath to Path #49385, Rename AdaptorConfig to TFileAdaptor #49654

cmsbuild · 2025-12-18T19:24:44Z

A new Pull Request was created by @makortel for master.

It involves the following packages:

Configuration/Applications (operations)
Configuration/PyReleaseValidation (pdmv)
FWIO/RNTupleTempInput (core)

@AdrianoDee, @DickyChant, @Dr15Jones, @antoniovagnerini, @cmsbuild, @davidlange6, @fabiocos, @ftenchini, @makortel, @mandrenguyen, @miquork, @smuzaffar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

makortel · 2025-12-18T21:44:21Z

test parameters:

workflows = 11834.99,12634.99,13034.99,250202.171,250202.181,250202.184,250202.185,250202.1,250202.3,250202.5,250206.181,250402.171,250402.181,250402.1,29834.9921,29834.999,29834.99,1041.0,1046.0

makortel · 2025-12-18T21:44:29Z

@cmsbuild, please test

makortel · 2025-12-18T22:41:53Z

I guess we don't have any workflows in runTheMatrix.py that would exercise the classical mixing with MinBias files generated in the workflow? (i.e. we can't test the RNTuple customization on MixingModule)

cmsbuild · 2025-12-18T22:49:32Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 60KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d8b69c/50354/summary.html
COMMIT: ac30b3e
CMSSW: CMSSW_16_0_X_2025-12-18-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49674/50354/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

ValueError: Undefined workflows: 29834.999, 29834.99, 29834.9921

makortel · 2025-12-18T23:13:44Z

test parameters:

workflows = 11834.99,12634.99,13034.99,250202.171,250202.181,250202.184,250202.185,250202.1,250202.3,250202.5,250206.181,250402.171,250402.181,250402.1,1041.0,1046.0

makortel · 2025-12-18T23:13:49Z

@cmsbuild, please test

cmsbuild · 2025-12-19T01:26:03Z

+1

Size: This PR adds an extra 60KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d8b69c/50358/summary.html
COMMIT: ac30b3e
CMSSW: CMSSW_16_0_X_2025-12-18-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49674/50358/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 5 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 1 differences found in the comparisons
Reco comparison had 4 failed jobs
DQMHistoTests: Total files compared: 68
DQMHistoTests: Total histograms compared: 5516729
DQMHistoTests: Total failures: 64
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 5516639
DQMHistoTests: Total skipped: 26
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 67 files compared)
Checked 294 log files, 259 edm output root files, 68 DQM output files
TriggerResults: no differences found

makortel · 2026-01-12T19:45:05Z

@cmsbuild, please test

cmsbuild · 2026-01-12T21:27:46Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d8b69c/50542/summary.html
COMMIT: d673b82
CMSSW: CMSSW_16_1_X_2026-01-12-1100/el8_amd64_gcc13
Additional Tests: RNTUPLE
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49674/50542/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
250202.181 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 14 lines to the logs
Reco comparison results: 3 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4167425
DQMHistoTests: Total failures: 40
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4167365
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 198 edm output root files, 53 DQM output files
TriggerResults: no differences found

makortel · 2026-01-12T21:42:00Z

Comparison differences (on standard workflows, not rntuple) are related to #47071

makortel · 2026-01-12T21:42:03Z

+core

makortel · 2026-01-12T21:42:17Z

This PR seems to be good to go in.

makortel · 2026-01-13T21:03:03Z

@cms-sw/pdmv-l2 Could you review and sign, please? Thanks!

AdrianoDee · 2026-01-14T10:26:25Z

+pdmv

cmsbuild · 2026-01-14T10:26:45Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-01-15T19:06:32Z

+1

cmsbuild added this to the CMSSW_16_1_X milestone Dec 18, 2025

cmsbuild added core-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending labels Dec 18, 2025

makortel mentioned this pull request Dec 18, 2025

Continuation of RNTuple testing in IBs #49640

Open

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 18, 2025

cmsbuild added tests-started and removed tests-pending labels Dec 18, 2025

cmsbuild added tests-rejected and removed tests-started labels Dec 18, 2025

cmsbuild added tests-started and removed tests-rejected labels Dec 18, 2025

cmsbuild added operations-approved and removed operations-pending tests-started labels Dec 19, 2025

cmsbuild added operations-pending tests-started and removed operations-approved tests-approved labels Jan 12, 2026

cmsbuild added operations-approved tests-approved and removed operations-pending tests-started labels Jan 12, 2026

cmsbuild added core-approved and removed core-pending labels Jan 12, 2026

makortel mentioned this pull request Jan 13, 2026

Revert "RNTuple requires MEtoEDM<long> have a dictionary" #49804

Merged

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Jan 14, 2026

cmsbuild added orp-approved and removed orp-pending labels Jan 15, 2026

cmsbuild merged commit 2f117e7 into cms-sw:master Jan 15, 2026
12 checks passed

This was referenced Jan 16, 2026

Avoid test name clashes #49844

Merged

[ROOT639] Updated root to tip of branch master cms-sw/cmsdist#10274

Merged

makortel deleted the rntupleSecondaryFileIn branch January 20, 2026 22:25

Make --secondfilein and on-the-fly pileup workflows to use RNTuple when --use-rntuple is specified #49674

Make --secondfilein and on-the-fly pileup workflows to use RNTuple when --use-rntuple is specified #49674

Uh oh!

Conversation

makortel commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Uh oh!

cmsbuild commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

makortel commented Dec 18, 2025

Uh oh!

makortel commented Dec 18, 2025

Uh oh!

makortel commented Dec 18, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Failed RelVals

Uh oh!

makortel commented Dec 18, 2025

Uh oh!

makortel commented Dec 18, 2025

Uh oh!

cmsbuild commented Dec 19, 2025

Comparison Summary

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

cmsbuild commented Jan 12, 2026

Comparison Summary

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

makortel commented Jan 12, 2026

Uh oh!

makortel commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdrianoDee commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

mandrenguyen commented Jan 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Make `--secondfilein` and on-the-fly pileup workflows to use RNTuple when `--use-rntuple` is specified #49674

Make `--secondfilein` and on-the-fly pileup workflows to use RNTuple when `--use-rntuple` is specified #49674

makortel commented Dec 18, 2025 •

edited

Loading

cmsbuild commented Dec 18, 2025 •

edited

Loading

makortel commented Jan 13, 2026 •

edited

Loading