Skip to content

Conversation

@ariostas
Copy link
Contributor

@ariostas ariostas commented Dec 4, 2025

This is a port of #48071 and #48818 (which where on the RNTUPLE_X branch) to master. It fixes the NanoAODRNTupleOutputModule so that it is compatible with the new RNTuple API (most of which is stable now). It is successfully able to output RNTuple NanoAODs with a nicer organization than before, but the implementation is not finalized, as delayed flags are not implemented yet and field projections will probably be needed for backwards compatibility.

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 4, 2025

cms-bot internal usage

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 4, 2025

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49551/47067

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 4, 2025

A new Pull Request was created by @ariostas for master.

It involves the following packages:

  • PhysicsTools/NanoAOD (xpog)

@battibass, @cmsbuild, @ftorrresd can you please review it and eventually sign? Thanks.
@gpetruc, @missirol, @mmusich this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@ariostas
Copy link
Contributor Author

ariostas commented Dec 4, 2025

@makortel fyi

@makortel
Copy link
Contributor

makortel commented Dec 4, 2025

@cmsbuild, please test

@makortel
Copy link
Contributor

makortel commented Dec 4, 2025

Thanks @ariostas!

In a wider context we ported the EDM components already in #49532

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 4, 2025

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b937d3/49808/summary.html
COMMIT: b60a926
CMSSW: CMSSW_16_0_X_2025-12-04-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49551/49808/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • Reco comparison results: 4 differences found in the comparisons
  • Reco comparison had 4 failed jobs
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4273241
  • DQMHistoTests: Total failures: 58
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4273163
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 227 log files, 198 edm output root files, 53 DQM output files
  • TriggerResults: no differences found

@ftorrresd
Copy link
Contributor

enable nano

@ftorrresd
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 6, 2025

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b937d3/49827/summary.html
COMMIT: b60a926
CMSSW: CMSSW_16_0_X_2025-12-05-2300/el8_amd64_gcc13
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49551/49827/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially removed 4 lines from the logs
  • Reco comparison results: 10 differences found in the comparisons
  • Reco comparison had 4 failed jobs
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4273241
  • DQMHistoTests: Total failures: 50
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4273171
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 227 log files, 198 edm output root files, 53 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

workflow id kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
NANOmc106Xul16v2 2500.0001 3.085 3.085 0.000 ( +0.0% ) 3.58 3.79 -5.6% 2.709 2.587
NANOmc106Xul16APVv2 2500.0002 3.041 3.041 0.000 ( +0.0% ) 3.60 3.82 -5.7% 2.689 2.612
NANOmc106Xul17v2 2500.0003 3.144 3.144 0.000 ( +0.0% ) 3.38 3.52 -4.1% 2.748 3.018
NANOmc106Xul18v2 2500.0004 3.097 3.097 0.000 ( +0.0% ) 3.41 3.58 -4.8% 2.735 3.036
NANOdata106Xul16v2 2500.0101 1.539 1.539 0.000 ( +0.0% ) 5.98 6.33 -5.6% 2.525 2.709
NANOdata106Xul17v2 2500.0102 1.926 1.926 0.000 ( +0.0% ) 4.02 4.19 -4.1% 2.579 2.934
NANOdata106Xul18v2 2500.0103 1.890 1.890 0.000 ( +0.0% ) 4.83 5.04 -4.3% 2.592 2.767
NANOmcUL16APVreMINI 2500.0201 0.022 0.022 0.000 ( +0.0% ) 1.76 1.87 -5.7% 2.930 2.545
NANOmcUL16reMINI 2500.0202 0.022 0.022 0.000 ( +0.0% ) 1.71 1.83 -6.4% 2.943 2.542
NANOmcUL17reMINI 2500.0203 0.022 0.022 0.000 ( +0.0% ) 1.60 1.64 -2.6% 2.972 2.674
NANOmcUL18reMINI 2500.0204 0.022 0.022 0.000 ( +0.0% ) 1.49 1.42 +5.0% 3.001 2.697
NANOdataUL16APVreMINI 2500.0301 0.035 0.035 0.000 ( +0.0% ) 1.56 1.65 -5.5% 2.886 2.667
NANOdataUL16reMINI 2500.0302 0.036 0.036 0.000 ( +0.0% ) 1.57 1.64 -4.2% 2.897 2.590
NANOdataUL17reMINI 2500.0303 0.037 0.037 0.000 ( +0.0% ) 1.52 1.54 -1.1% 2.906 2.699
NANOdataUL18reMINI 2500.0304 0.036 0.036 0.000 ( +0.0% ) 1.48 1.56 -5.1% 2.876 2.663
NANOmcUL16APVMini2Mini 2500.0401 0.023 0.023 0.000 ( +0.0% ) 2.56 2.49 +2.7% 2.227 2.239
NANOmcUL16Mini2Mini 2500.0402 0.023 0.023 0.000 ( +0.0% ) 2.41 2.57 -6.2% 2.262 2.158
NANOmcUL17Mini2Mini 2500.0403 0.023 0.023 0.000 ( +0.0% ) 2.38 2.42 -1.7% 2.271 2.181
NANOmcUL18Mini2Mini 2500.0404 0.023 0.023 0.000 ( +0.0% ) 2.33 2.45 -5.1% 2.269 2.190
NANOdataUL16Mini2Mini 2500.0501 0.039 0.039 0.000 ( +0.0% ) 3.46 3.57 -3.1% 2.167 2.012
NANOdataUL17Mini2Mini 2500.0502 0.040 0.040 0.000 ( +0.0% ) 2.58 2.67 -3.4% 2.226 2.076
NANOdataUL18Mini2Mini 2500.0503 0.039 0.039 0.000 ( +0.0% ) 3.04 3.16 -4.0% 2.214 2.074
NANOmc130X 2500.1001 3.220 3.220 0.000 ( +0.0% ) 4.03 4.25 -5.1% 3.184 2.910
NANOdata130Xrun3 2500.1101 1.790 1.790 0.000 ( +0.0% ) 6.06 6.48 -6.6% 2.842 2.618
NANOmc23Mini2Mini 2500.1201 0.022 0.022 0.000 ( +0.0% ) 2.65 2.78 -4.8% 2.190 2.021
NANOdata23Mini2Mini 2500.1301 0.041 0.041 0.000 ( +0.0% ) 3.56 3.67 -3.0% 2.472 2.399
ScoutingNANOdata124Xrun3 2500.1501 0.772 0.772 0.000 ( +0.0% ) 33.01 34.68 -4.8% 1.652 1.506
ScoutingNANOmonitordata124Xrun3 2500.1502 0.862 0.862 0.000 ( +0.0% ) 13.27 13.99 -5.1% 1.488 1.420
ScoutingNANOdata130Xrun3 2500.1503 1.251 1.251 0.000 ( +0.0% ) 24.38 25.39 -4.0% 1.621 1.476
ScoutingNANOmonitordata130Xrun3 2500.1504 1.297 1.297 0.000 ( +0.0% ) 10.70 10.43 +2.6% 1.472 1.431
muPOGNANOmc140X 2500.2201 2.065 2.065 0.000 ( +0.0% ) 5.53 5.78 -4.2% 2.226 2.022
EGMNANOmc140X 2500.2202 3.849 3.849 0.000 ( +0.0% ) 3.82 4.05 -5.7% 2.621 2.344
BTVNANOmc140X 2500.2203 10.466 10.466 0.000 ( +0.0% ) 2.50 2.59 -3.6% 2.841 2.536
jmeNANOmc140X 2500.2204 6.630 6.630 0.000 ( +0.0% ) 1.09 1.15 -5.5% 2.671 2.398
lepTrackInfoNANOmc140X 2500.2206 3.543 3.543 0.000 ( +0.0% ) 3.82 4.04 -5.4% 2.624 2.349
ScoutingNANOmc140X 2500.2207 1.488 1.488 0.000 ( +0.0% ) 13.61 13.94 -2.3% 1.927 1.801
ScoutingNANOwithPromptmc140X 2500.2208 4.452 4.452 0.000 ( +0.0% ) 3.10 3.22 -3.7% 2.875 2.510
muPOGNANO140Xrun3 2500.2301 1.469 1.469 0.000 ( +0.0% ) 7.75 8.11 -4.5% 2.560 1.965
EGMNANOdata140Xrun3 2500.2302 2.623 2.623 0.000 ( +0.0% ) 5.51 5.82 -5.4% 2.954 2.623
BTVNANOdata140Xrun3 2500.2303 5.612 5.612 0.000 ( +0.0% ) 3.59 3.75 -4.4% 3.172 2.792
jmeNANOdata140Xrun3 2500.2304 3.976 3.976 0.000 ( +0.0% ) 1.41 1.49 -5.4% 2.686 2.716
lepTrackInfoNANOdata140Xrun3 2500.2306 2.413 2.413 0.000 ( +0.0% ) 5.55 5.88 -5.6% 2.623 2.630
ScoutingNANOdata140Xrun3 2500.2307 1.101 1.101 0.000 ( +0.0% ) 29.26 30.46 -4.0% 1.580 1.547
ScoutingNANOmonitordata140Xrun3 2500.2308 2.784 2.784 0.000 ( +0.0% ) 4.93 5.08 -3.0% 2.807 2.813
ScoutingNANOmonitorWithPromptdata140Xrun3 2500.2309 2.784 2.784 0.000 ( +0.0% ) 4.93 5.10 -3.3% 2.834 2.788
L1ScoutingSelectionNANOdata140Xrun3 2500.2311 0.081 0.081 0.000 ( +0.0% ) 44.47 46.58 -4.5% 0.835 0.673
L1ScoutingNANOdata140Xrun3 2500.231 0.082 0.082 0.000 ( +0.0% ) 3.89 4.00 -2.8% 0.942 0.857
l1DPGNANO140Xrun3 2500.2401 9.404 9.404 0.000 ( +0.0% ) 7.26 6.28 +15.6% 1.870 1.741
muDPGNANO140Xrun3 2500.2402 10.377 10.377 0.000 ( +0.0% ) 1.59 1.60 -0.8% 1.719 1.613
muDPGNANOBkg140Xrun3 2500.2403 6.401 6.401 0.000 ( +0.0% ) 16.26 12.25 +32.8% 1.232 1.186
hcalDPGNANO140Xrun3 2500.2404 486.015 486.015 0.000 ( +0.0% ) 1.15 1.18 -2.0% 1.689 1.623
hcalDPGCalibNANO140Xrun3 2500.2405 826.418 826.418 0.000 ( +0.0% ) 1.50 1.54 -2.6% 1.679 1.552
hcalDPGMCNANO140Xrun3 2500.2501 645.333 645.333 0.000 ( +0.0% ) 1.65 1.67 -1.3% 1.790 1.636
NANOmc2024reMINI 2500.2601 0.021 0.021 0.000 ( +0.0% ) 1.88 1.92 -2.0% 3.059 2.618
NANOdata2024reMINI 2500.2701 0.036 0.036 0.000 ( +0.0% ) 1.88 1.97 -4.5% 2.844 2.568
NANOmc150X 2500.3001 3.217 3.217 0.000 ( +0.0% ) 17.07 18.05 -5.4% 2.734 2.590
NANOdata150X 2500.3101 2.144 2.144 0.000 ( +0.0% ) 32.22 35.31 -8.8% 2.471 2.234
NANOdata150X 2500.3102 2.630 2.630 0.000 ( +0.0% ) 27.46 29.16 -5.8% 2.562 2.287
muPOGNANOmc150X 2500.3201 2.255 2.255 0.000 ( +0.0% ) 16.50 16.87 -2.2% 2.584 2.366
EGMNANOmc150X 2500.3202 4.111 4.111 0.000 ( +0.0% ) 14.32 14.84 -3.5% 2.724 2.443
BTVNANOmc150X 2500.3203 10.552 10.552 0.000 ( +0.0% ) 4.77 5.01 -4.9% 2.946 2.649
jmeNANOmc150X 2500.3204 6.595 6.595 0.000 ( +0.0% ) 1.32 1.35 -2.1% 2.916 2.663
jmeNANOrePuppimc150X 2500.3205 6.598 6.598 0.000 ( +0.0% ) 1.25 1.26 -0.7% 3.190 2.938
lepTrackInfoNANOmc150X 2500.3206 3.721 3.721 0.000 ( +0.0% ) 14.88 15.30 -2.7% 2.720 2.432
ScoutingNANOmc150X 2500.3207 1.563 1.563 0.000 ( +0.0% ) 19.76 20.04 -1.4% 2.454 2.305
ScoutingNANOwithPromptmc150X 2500.3208 4.775 4.775 0.000 ( +0.0% ) 10.50 10.57 -0.7% 2.956 2.839
BPHNANOmc150X 2500.3209 4.127 4.127 0.000 ( +0.0% ) 16.69 16.74 -0.4% 2.795 2.537
muPOGNANO150Xrun3 2500.3301 1.125 1.125 0.000 ( +0.0% ) 32.68 33.08 -1.2% 2.347 2.037
EGMNANOdata150Xrun3 2500.3302 2.985 2.985 0.000 ( +0.0% ) 26.59 26.99 -1.5% 2.466 2.156
BTVNANOdata150Xrun3 2500.3303 7.971 7.971 0.000 ( +0.0% ) 5.95 6.06 -1.7% 2.696 2.397
jmeNANOdata150Xrun3 2500.3304 4.349 4.349 0.000 ( +0.0% ) 1.64 1.63 +0.8% 2.635 2.314
jmeNANOrePuppidata150Xrun3 2500.3305 4.332 4.332 0.000 ( +0.0% ) 1.43 1.42 +0.8% 2.909 2.572
lepTrackInfoNANOdata150Xrun3 2500.3306 2.625 2.625 0.000 ( +0.0% ) 28.28 28.36 -0.3% 2.459 2.125
ScoutingNANOdata150Xrun3 2500.3307 1.738 1.738 0.000 ( +0.0% ) 31.35 31.84 -1.6% 1.640 1.597
ScoutingNANOmonitordata150Xrun3 2500.3308 1.400 1.400 0.000 ( +0.0% ) 26.85 27.15 -1.1% 2.335 2.175
ScoutingNANOmonitorWithPromptdata150Xrun3 2500.3309 2.602 2.602 0.000 ( +0.0% ) 17.81 17.94 -0.7% 2.741 2.494
BPHNANOdata150Xrun3 2500.331 2.632 2.632 0.000 ( +0.0% ) 32.74 32.74 +0.0% 2.527 2.248
TTbarMINIAOD10.6_UL18v2 2500.9001 1.819 1.819 0.000 ( +0.0% ) 35.67 36.22 -1.5% 1.858 1.856
TTbarMINIAOD14.0 2500.9002 1.665 1.665 0.000 ( +0.0% ) 33.95 33.92 +0.1% 1.446 1.446
DYToLL_M-50_13TeV_pythia8 2500.9101 15.163 15.163 0.000 ( +0.0% ) 7.50 7.11 +5.5% 1.077 1.078
DYToll01234Jets_5f_LO_MLM_Madgraph_LHE_13TeV 2500.9102 0.310 0.150 0.161 ( +107.4% ) 2.37 2.18 +8.4% 0.842 0.841
TTbar_Pow_LHE_13TeV 2500.9103 0.110 0.110 0.000 ( +0.0% ) 2.37 2.50 -5.2% 0.846 0.846

NANO Comparison Summary

Summary:

  • You potentially added 534 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 32
  • DQMHistoTests: Total histograms compared: 41718
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 41718
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
  • Checked 181 log files, 106 edm output root files, 32 DQM output files
  • TriggerResults: no differences found

@makortel
Copy link
Contributor

@cms-sw/xpog-l2 It would be great to have this PR in 16_0_0_pre4 (scheduled to be built next week)

@battibass
Copy link

+xpog

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @ftenchini, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

@battibass
Copy link

@cms-sw/xpog-l2 It would be great to have this PR in 16_0_0_pre4 (scheduled to be built next week)

@makortel, done, as agreed on Thursday.

@battibass
Copy link

Also adding here a couple notes, as they were not mentioned in the PR description:

  • the code from this PR does not affect cmsDriver.py commands, unless activated explicitly;
  • when developments will be close to completion they are surely worth a report in a future x-pog meeting.

@makortel
Copy link
Contributor

Thanks!

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit ff1c182 into cms-sw:master Dec 15, 2025
12 checks passed
@smuzaffar
Copy link
Contributor

@ariostas , we now have conflicts in cmssw master and RNTUPLE_X branch [a]. Can you please provide a PR for RNTUPLE_X branch to resolve this conflict ?

[a] https://cmssdt.cern.ch/jenkins/job/auto-forward-port-branch/63402/console

diff --cc PhysicsTools/NanoAOD/plugins/rntuple/NanoAODRNTupleOutputModule.cc
index 03cbb67fb57,26f8aca1ee2..00000000000
--- a/PhysicsTools/NanoAOD/plugins/rntuple/NanoAODRNTupleOutputModule.cc
+++ b/PhysicsTools/NanoAOD/plugins/rntuple/NanoAODRNTupleOutputModule.cc
@@@ -255,24 -203,14 +255,34 @@@ void NanoAODRNTupleOutputModule::initia
    }
    m_evstrings.createFields(*model);
  
++<<<<<<< HEAD
 +  if (m_noSplitFields.size() == 1 and m_noSplitFields[0] == "all") {
 +    for (auto const& topName : model->GetFieldNames()) {
 +      auto& field = model->GetMutableField(topName);
 +      noSplitField(field);
 +      applyNoSplitToSubFields(field);
 +    }
 +  } else {
 +    for (auto const& name : m_noSplitFields) {
 +      auto& field = model->GetMutableField(name);
 +      noSplitField(field);
 +    }
 +  }
 +
++=======
++>>>>>>> origin/CMSSW_16_0_X
    // Model needs to be frozen before we bind buffers
    model->Freeze();
  
    m_tables.bindBuffers(*model);
++<<<<<<< HEAD
 +  m_ntuple = RNTupleWriter::Append(std::move(model), "Events", *m_file, m_writeOptions);
++=======
+ 
+   RNTupleWriteOptions options;
+   options.SetCompression(m_file->GetCompressionSettings());
+   m_ntuple = RNTupleWriter::Append(std::move(model), "Events", *m_file, options);
++>>>>>>> origin/CMSSW_16_0_X
  }
  
  void NanoAODRNTupleOutputModule::write(edm::EventForOutput const& iEvent) {'

@ariostas
Copy link
Contributor Author

@smuzaffar yeah, I'll do that. I'm a bit confused why there are conflicts since I just cherry-picked the commits, but I'll look into it. Thanks!

@ariostas
Copy link
Contributor Author

Oh sorry, I missed #49239 in here. @Dr15Jones do you want to open a PR to port it to master, or should I do it?

@Dr15Jones
Copy link
Contributor

please go ahead and do the port to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants