enable dqm comparison job for rntuple #2651

smuzaffar · 2026-01-14T09:26:48Z

This change enables running PR DQM/Reco comparison for RNTuple tests.

smuzaffar · 2026-01-14T09:27:03Z

enable rntuple

cmsbuild · 2026-01-14T09:27:21Z

A new Pull Request was created by @smuzaffar for branch master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-01-14T09:27:22Z

cms-bot internal usage

smuzaffar · 2026-01-14T09:28:00Z

please test

though RNTuple comparison job will fail but lets see what works

cmsbuild · 2026-01-14T10:23:35Z

Pull request #2651 was updated.

cmsbuild · 2026-01-14T11:14:01Z

Pull request #2651 was updated.

cmsbuild · 2026-01-14T11:15:31Z

Pull request #2651 was updated.

cmsbuild · 2026-01-14T11:58:52Z

Pull request #2651 was updated.

cmsbuild · 2026-01-15T10:48:33Z

Pull request #2651 was updated.

cmsbuild · 2026-01-15T11:11:06Z

Pull request #2651 was updated.

cmsbuild · 2026-01-15T14:14:51Z

Pull request #2651 was updated.

smuzaffar · 2026-01-15T14:25:20Z

please test

cmsbuild · 2026-01-15T16:32:25Z

-1

Failed Tests: RelVals-RNTUPLE
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50657/summary.html
COMMIT: 4b994f2
CMSSW: CMSSW_16_1_X_2026-01-15-1100/el8_amd64_gcc13
Additional Tests: RNTUPLE
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2651/50657/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-RNTUPLE

4.224.22_RunCosmics2011A/step4_RunCosmics2011A.log
1000.01000.0_RunMinBias2011A/step3_RunMinBias2011A.log
250202.181250202.181_TTbar13TeVPUppmx2018/step3_TTbar13TeVPUppmx2018.log

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025513
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

RNTUPLE Comparison Summary

Summary:

You potentially removed 209 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 3821985
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3821966
DQMHistoTests: Total skipped: 16
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 217 log files, 0 edm output root files, 52 DQM output files

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 132 workflow step(s) with memory usage exceeding the error threshold:

Error: Workflow 1000.0_RunMinBias2011A step2 max memory diff 486.3 exceeds +/- 90.0 MiB
Error: Workflow 1001.0_RunMinBias2011A step2 max memory diff 608.0 exceeds +/- 90.0 MiB
Error: Workflow 1001.0_RunMinBias2011A step3 max memory diff 328.7 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step1 max memory diff 164.6 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step2 max memory diff 501.5 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step3 max memory diff 888.2 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step1 max memory diff 167.8 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step2 max memory diff 516.8 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step3 max memory diff 565.9 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step5 max memory diff 136.0 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step1 max memory diff 171.0 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step2 max memory diff 522.4 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step3 max memory diff 630.3 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step5 max memory diff 135.7 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step1 max memory diff 164.0 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step2 max memory diff 516.0 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step3 max memory diff 558.8 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step5 max memory diff 134.8 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step1 max memory diff 163.4 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step2 max memory diff 497.2 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step3 max memory diff 525.4 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step5 max memory diff 124.9 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step1 max memory diff 164.0 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step2 max memory diff 411.1 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step3 max memory diff 866.2 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step1 max memory diff 140.7 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step2 max memory diff 445.8 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step3 max memory diff 705.3 exceeds +/- 90.0 MiB
Error: Workflow 13234.0_TTbar_14TeV+2022FS step1 max memory diff 146.0 exceeds +/- 90.0 MiB
Error: Workflow 13234.0_TTbar_14TeV+2022FS step2 max memory diff 220.4 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step1 max memory diff 145.1 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step2 max memory diff 441.0 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step3 max memory diff 621.6 exceeds +/- 90.0 MiB
Error: Workflow 135.4_ZEEFS_13 step1 max memory diff 336.1 exceeds +/- 90.0 MiB
Error: Workflow 135.4_ZEEFS_13 step3 max memory diff 298.4 exceeds +/- 90.0 MiB
Error: Workflow 136.731_RunSinglePh2016B step2 max memory diff 598.6 exceeds +/- 90.0 MiB
Error: Workflow 136.731_RunSinglePh2016B step3 max memory diff 828.4 exceeds +/- 90.0 MiB
Error: Workflow 136.793_RunDoubleEG2017C step2 max memory diff 622.7 exceeds +/- 90.0 MiB
Error: Workflow 136.793_RunDoubleEG2017C step3 max memory diff 1246.4 exceeds +/- 90.0 MiB
Error: Workflow 136.874_RunEGamma2018C step2 max memory diff 649.5 exceeds +/- 90.0 MiB
Error: Workflow 136.874_RunEGamma2018C step3 max memory diff 904.9 exceeds +/- 90.0 MiB
Error: Workflow 139.001_RunMinimumBias2021 step2 max memory diff 465.8 exceeds +/- 90.0 MiB
Error: Workflow 139.001_RunMinimumBias2021 step3 max memory diff 664.2 exceeds +/- 90.0 MiB
Error: Workflow 140.56_RunHI2018 step2 max memory diff 358.3 exceeds +/- 90.0 MiB
Error: Workflow 14034.0_TTbar_14TeV+2023FS step1 max memory diff 146.0 exceeds +/- 90.0 MiB
Error: Workflow 14034.0_TTbar_14TeV+2023FS step2 max memory diff 132.4 exceeds +/- 90.0 MiB
Error: Workflow 14234.0_TTbar_14TeV+2023FSPU step1 max memory diff 146.0 exceeds +/- 90.0 MiB
Error: Workflow 14234.0_TTbar_14TeV+2023FSPU step2 max memory diff 227.8 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step1 max memory diff 166.8 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step2 max memory diff 520.7 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step3 max memory diff 549.4 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step5 max memory diff 135.0 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step1 max memory diff 166.8 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step2 max memory diff 382.8 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step3 max memory diff 850.8 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step1 max memory diff 166.8 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step2 max memory diff 2269.9 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step3 max memory diff 2731.7 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step5 max memory diff 267.6 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step1 max memory diff 166.8 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step2 max memory diff 2232.5 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step3 max memory diff 3095.3 exceeds +/- 90.0 MiB
Error: Workflow 2022.0030001_RunBTagMu2022E_10k step2 max memory diff 606.7 exceeds +/- 90.0 MiB
Error: Workflow 2022.0030001_RunBTagMu2022E_10k step3 max memory diff 1178.4 exceeds +/- 90.0 MiB
Error: Workflow 2023.0020001_RunJetMET02023D_10k step2 max memory diff 619.2 exceeds +/- 90.0 MiB
Error: Workflow 2023.0020001_RunJetMET02023D_10k step3 max memory diff 1155.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0000001_RunZeroBias2024B_10k step2 max memory diff 569.4 exceeds +/- 90.0 MiB
Error: Workflow 2024.0000001_RunZeroBias2024B_10k step3 max memory diff 1230.8 exceeds +/- 90.0 MiB
Error: Workflow 2024.0010001_RunJetMET02024C_10k step2 max memory diff 575.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0010001_RunJetMET02024C_10k step3 max memory diff 1245.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0020001_RunEGamma02024D_10k step2 max memory diff 571.4 exceeds +/- 90.0 MiB
Error: Workflow 2024.0020001_RunEGamma02024D_10k step3 max memory diff 1183.9 exceeds +/- 90.0 MiB
Error: Workflow 2024.0030001_RunDisplacedJet2024E_10k step2 max memory diff 569.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0030001_RunDisplacedJet2024E_10k step3 max memory diff 1314.0 exceeds +/- 90.0 MiB
Error: Workflow 2024.0040001_RunPark2MuonLowMass02024F_10k step2 max memory diff 613.7 exceeds +/- 90.0 MiB
Error: Workflow 2024.0040001_RunPark2MuonLowMass02024F_10k step3 max memory diff 1400.9 exceeds +/- 90.0 MiB
Error: Workflow 2024.0050001_RunBTagMu2024G_10k step2 max memory diff 566.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0050001_RunBTagMu2024G_10k step3 max memory diff 1293.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0060001_RunMuon02024H_10k step2 max memory diff 570.1 exceeds +/- 90.0 MiB
Error: Workflow 2024.0060001_RunMuon02024H_10k step3 max memory diff 1140.9 exceeds +/- 90.0 MiB
Error: Workflow 2024.0070001_RunTau2024I_10k step2 max memory diff 563.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0070001_RunTau2024I_10k step3 max memory diff 1187.0 exceeds +/- 90.0 MiB
Error: Workflow 2025.0000001_RunZeroBias2025B_10k step2 max memory diff 2566.9 exceeds +/- 90.0 MiB
Error: Workflow 2025.0000001_RunZeroBias2025B_10k step3 max memory diff 4239.4 exceeds +/- 90.0 MiB
Error: Workflow 2025.0010001_RunJetMET02025C_10k step2 max memory diff 2697.2 exceeds +/- 90.0 MiB
Error: Workflow 2025.0010001_RunJetMET02025C_10k step3 max memory diff 4552.9 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step1 max memory diff 165.8 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step2 max memory diff 491.0 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step3 max memory diff 474.7 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step5 max memory diff 431.0 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step1 max memory diff 164.3 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step3 max memory diff -2035.1 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step1 max memory diff 154.4 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step2 max memory diff 500.0 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step3 max memory diff 772.8 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step1 max memory diff 163.6 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step2 max memory diff 444.9 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step3 max memory diff 630.6 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step1 max memory diff 171.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 461.5 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step3 max memory diff 1470.4 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step5 max memory diff 1013.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step1 max memory diff 171.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 335.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step1 max memory diff 170.2 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 461.9 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step3 max memory diff 1413.7 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step5 max memory diff 1016.0 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step1 max memory diff 148.8 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 420.9 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step3 max memory diff 1349.0 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step5 max memory diff 994.0 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step1 max memory diff 146.3 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 415.3 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step3 max memory diff 1292.5 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step5 max memory diff 988.5 exceeds +/- 90.0 MiB
Error: Workflow 4.22_RunCosmics2011A step2 max memory diff 611.1 exceeds +/- 90.0 MiB
Error: Workflow 4.22_RunCosmics2011A step3 max memory diff 431.1 exceeds +/- 90.0 MiB
Error: Workflow 4.53_RunPhoton2012B step2 max memory diff 492.8 exceeds +/- 90.0 MiB
Error: Workflow 4.53_RunPhoton2012B step3 max memory diff 1057.6 exceeds +/- 90.0 MiB
Error: Workflow 5.1_TTbarFS step1 max memory diff 351.9 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step1 max memory diff 141.3 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step2 max memory diff 310.3 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step3 max memory diff 641.0 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step4 max memory diff 269.9 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step1 max memory diff 141.3 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step2 max memory diff 311.3 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step3 max memory diff 644.8 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step4 max memory diff 490.6 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step1 max memory diff 163.3 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step2 max memory diff 485.1 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step3 max memory diff 654.9 exceeds +/- 90.0 MiB

makortel · 2026-01-15T18:49:02Z

(not related to this PR)

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 132 workflow step(s) with memory usage exceeding the error threshold:

Error: Workflow 1000.0_RunMinBias2011A step2 max memory diff 486.3 exceeds +/- 90.0 MiB

Error: Workflow 1001.0_RunMinBias2011A step2 max memory diff 608.0 exceeds +/- 90.0 MiB

I assume this report is for comparing RNTuple RelVals to default RelVals. Large increase is presently expected, but maybe there is some value in keeping them and recording the values in GitHub comments?

If the default RelVals would issue a warning as well, would we have two Max Memory Comparisons exceeding threshold sections in the summary message above?

Couple of further notes:

It would be nice if the order of workflows in the summary message #2651 (comment) would be the same as in the tables https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisonsRNTUPLE/CMSSW_16_1_X_2026-01-15-1100+96a0b5/72771/maxmem-comparison/maxmem_summary.html

In the summary page https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50657/summary.html I see

Could the "max memory" entries be ordered in the following way

Unit tests
max memory used comparison
max memory used RNTUPLE comparison failed
?

@gartung Would you be able look into polishing these aspects? (we should open a separate task issue for the work)

gartung · 2026-01-15T19:32:02Z

Unit tests

max memory used comparison

max memory used RNTUPLE comparison failed
?

@gartung Would you be able look into polishing these aspects? (we should open a separate task issue for the work)

I think the order is determined by when the tests complete so it is not deterministic. RNTUPLE is a flavor of test and might have completed first.

gartung · 2026-01-15T20:09:37Z

It might be alphabetical order in which case 2653 should fix it.

smuzaffar · 2026-01-16T08:41:56Z

I assume this report is for comparing RNTuple RelVals to default RelVals. Large increase is presently expected, but maybe there is some value in keeping them and recording the values in GitHub comments?

@makortel , yes this is for RNTuple Relvals comparison with default IBs relvals (non-rntuple). I personally think that we should only have a short message e.g. @cms-sw/core-l2 , I found 132 workflow step(s) with memory usage exceeding the error threshold and may be first 3 relvals numbers ( just like what we do not relvals/unit test failures).

If the default RelVals would issue a warning as well, would we have two Max Memory Comparisons exceeding threshold sections in the summary message above?

yes, each comparison job will add an extra Max Memory Comparison section. For now we are running comparison for default and rntuple only but if in future we enabled it for GPUs then each GPU can add this section.

@gartung , order of the items on the summary page is alphabetical, so if you change these items to start with Max memory used comparison and Max memory used comparison RNTuple then these will be next to each other

cmsbuild · 2026-01-17T15:18:14Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50703/summary.html
COMMIT: bfcd3e7
CMSSW: CMSSW_16_1_X_2026-01-17-1100/el8_amd64_gcc13
Additional Tests: RNTUPLE
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2651/50703/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025510
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

RNTUPLE Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025510
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 4 edm output root files, 52 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold RNTUPLE

@cms-sw/core-l2 , I found 134 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 4.22_RunCosmics2011A step4 max memory diff 424.9 exceeds +/- 90.0 MiB
Error: Workflow 4.53_RunPhoton2012B step3 max memory diff 1122.5 exceeds +/- 90.0 MiB
Error: Workflow 4.53_RunPhoton2012B step2 max memory diff 492.2 exceeds +/- 90.0 MiB
Error: Workflow 5.1_TTbarFS step1 max memory diff 351.9 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step2 max memory diff 308.2 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step3 max memory diff 639.5 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step4 max memory diff 269.9 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step1 max memory diff 140.8 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step2 max memory diff 311.3 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step4 max memory diff 490.7 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step3 max memory diff 643.7 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step1 max memory diff 140.8 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step3 max memory diff 707.7 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step1 max memory diff 162.8 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step2 max memory diff 484.8 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step3 max memory diff 532.5 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step2 max memory diff 490.8 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step5 max memory diff 431.0 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step1 max memory diff 165.2 exceeds +/- 90.0 MiB
Error: Workflow 135.4_ZEEFS_13 step1 max memory diff 336.1 exceeds +/- 90.0 MiB
Error: Workflow 135.4_ZEEFS_13 step3 max memory diff 295.3 exceeds +/- 90.0 MiB
Error: Workflow 136.731_RunSinglePh2016B step3 max memory diff 750.1 exceeds +/- 90.0 MiB
Error: Workflow 136.731_RunSinglePh2016B step2 max memory diff 596.6 exceeds +/- 90.0 MiB
Error: Workflow 136.793_RunDoubleEG2017C step2 max memory diff 620.7 exceeds +/- 90.0 MiB
Error: Workflow 136.793_RunDoubleEG2017C step3 max memory diff 1234.2 exceeds +/- 90.0 MiB
Error: Workflow 136.874_RunEGamma2018C step3 max memory diff 968.8 exceeds +/- 90.0 MiB
Error: Workflow 136.874_RunEGamma2018C step2 max memory diff 647.5 exceeds +/- 90.0 MiB
Error: Workflow 139.001_RunMinimumBias2021 step2 max memory diff 463.8 exceeds +/- 90.0 MiB
Error: Workflow 139.001_RunMinimumBias2021 step3 max memory diff 730.2 exceeds +/- 90.0 MiB
Error: Workflow 140.56_RunHI2018 step2 max memory diff 358.3 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step3 max memory diff 626.2 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step2 max memory diff 444.5 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step1 max memory diff 163.1 exceeds +/- 90.0 MiB
Error: Workflow 1000.0_RunMinBias2011A step3 max memory diff 382.0 exceeds +/- 90.0 MiB
Error: Workflow 1000.0_RunMinBias2011A step5 max memory diff -654.6 exceeds +/- 90.0 MiB
Error: Workflow 1001.0_RunMinBias2011A step2 max memory diff 608.1 exceeds +/- 90.0 MiB
Error: Workflow 1001.0_RunMinBias2011A step3 max memory diff 328.7 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step1 max memory diff 140.2 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step3 max memory diff 641.5 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step2 max memory diff 445.7 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step3 max memory diff 686.3 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step1 max memory diff 144.5 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step2 max memory diff 440.9 exceeds +/- 90.0 MiB
Error: Workflow 2022.0030001_RunBTagMu2022E_10k step2 max memory diff 604.7 exceeds +/- 90.0 MiB
Error: Workflow 2022.0030001_RunBTagMu2022E_10k step3 max memory diff 1178.3 exceeds +/- 90.0 MiB
Error: Workflow 2023.0020001_RunJetMET02023D_10k step2 max memory diff 617.1 exceeds +/- 90.0 MiB
Error: Workflow 2023.0020001_RunJetMET02023D_10k step3 max memory diff 1219.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0000001_RunZeroBias2024B_10k step3 max memory diff 1157.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0000001_RunZeroBias2024B_10k step2 max memory diff 567.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0010001_RunJetMET02024C_10k step3 max memory diff 1313.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0010001_RunJetMET02024C_10k step2 max memory diff 573.4 exceeds +/- 90.0 MiB
Error: Workflow 2024.0020001_RunEGamma02024D_10k step3 max memory diff 1167.1 exceeds +/- 90.0 MiB
Error: Workflow 2024.0020001_RunEGamma02024D_10k step2 max memory diff 571.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0030001_RunDisplacedJet2024E_10k step3 max memory diff 1194.7 exceeds +/- 90.0 MiB
Error: Workflow 2024.0030001_RunDisplacedJet2024E_10k step2 max memory diff 566.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0040001_RunPark2MuonLowMass02024F_10k step3 max memory diff 1223.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0040001_RunPark2MuonLowMass02024F_10k step2 max memory diff 575.0 exceeds +/- 90.0 MiB
Error: Workflow 2024.0050001_RunBTagMu2024G_10k step3 max memory diff 1295.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0050001_RunBTagMu2024G_10k step2 max memory diff 564.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0060001_RunMuon02024H_10k step3 max memory diff 1272.8 exceeds +/- 90.0 MiB
Error: Workflow 2024.0060001_RunMuon02024H_10k step2 max memory diff 568.1 exceeds +/- 90.0 MiB
Error: Workflow 2024.0070001_RunTau2024I_10k step3 max memory diff 1230.0 exceeds +/- 90.0 MiB
Error: Workflow 2024.0070001_RunTau2024I_10k step2 max memory diff 566.9 exceeds +/- 90.0 MiB
Error: Workflow 2025.0000001_RunZeroBias2025B_10k step2 max memory diff 2573.1 exceeds +/- 90.0 MiB
Error: Workflow 2025.0000001_RunZeroBias2025B_10k step3 max memory diff 4270.4 exceeds +/- 90.0 MiB
Error: Workflow 2025.0010001_RunJetMET02025C_10k step3 max memory diff 4538.4 exceeds +/- 90.0 MiB
Error: Workflow 2025.0010001_RunJetMET02025C_10k step2 max memory diff 2650.7 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step3 max memory diff 886.7 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step2 max memory diff 489.0 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step1 max memory diff 164.1 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step2 max memory diff 514.7 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step3 max memory diff 565.9 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step5 max memory diff 136.0 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step1 max memory diff 167.3 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step2 max memory diff 520.4 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step3 max memory diff 566.2 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step1 max memory diff 170.5 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step5 max memory diff 135.8 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step5 max memory diff 134.8 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step2 max memory diff 513.9 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step1 max memory diff 163.5 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step3 max memory diff 556.6 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step1 max memory diff 162.8 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step5 max memory diff 124.9 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step3 max memory diff 527.4 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step2 max memory diff 495.1 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step1 max memory diff 163.5 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step3 max memory diff 867.1 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step2 max memory diff 409.2 exceeds +/- 90.0 MiB
Error: Workflow 13234.0_TTbar_14TeV+2022FS step1 max memory diff 145.5 exceeds +/- 90.0 MiB
Error: Workflow 13234.0_TTbar_14TeV+2022FS step2 max memory diff 286.3 exceeds +/- 90.0 MiB
Error: Workflow 14034.0_TTbar_14TeV+2023FS step2 max memory diff 150.9 exceeds +/- 90.0 MiB
Error: Workflow 14034.0_TTbar_14TeV+2023FS step1 max memory diff 145.5 exceeds +/- 90.0 MiB
Error: Workflow 14234.0_TTbar_14TeV+2023FSPU step2 max memory diff 166.1 exceeds +/- 90.0 MiB
Error: Workflow 14234.0_TTbar_14TeV+2023FSPU step1 max memory diff 145.5 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step2 max memory diff 518.6 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step5 max memory diff 135.0 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step3 max memory diff 502.9 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step3 max memory diff 778.9 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step2 max memory diff 380.7 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step2 max memory diff 2270.2 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step3 max memory diff 2719.5 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step5 max memory diff 267.6 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step2 max memory diff 2230.3 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step3 max memory diff 3095.4 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step1 max memory diff 153.8 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step3 max memory diff 781.1 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step2 max memory diff 500.0 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 461.4 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step1 max memory diff 170.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step5 max memory diff 1013.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step3 max memory diff 1466.2 exceeds +/- 90.0 MiB
Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 331.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step1 max memory diff 170.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step5 max memory diff 1015.9 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 461.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step3 max memory diff 1401.2 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step1 max memory diff 169.7 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step5 max memory diff 994.0 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 420.8 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step3 max memory diff 1340.7 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step1 max memory diff 148.3 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 415.2 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step3 max memory diff 1350.3 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step5 max memory diff 988.5 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step1 max memory diff 145.8 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step4 max memory diff 1013.9 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step3 max memory diff 634.4 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step1 max memory diff 163.8 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step2 max memory diff 97.3 exceeds +/- 90.0 MiB

cmsbuild · 2026-01-17T16:12:40Z

Pull request #2651 was updated.

smuzaffar · 2026-01-21T16:21:08Z

please test

cmsbuild · 2026-01-21T16:21:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50705/summary.html
COMMIT: 3a403b4
CMSSW: CMSSW_16_1_X_2026-01-17-1100/el8_amd64_gcc13
Additional Tests: RNTUPLE
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2651/50705/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 16 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025504
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

RNTUPLE Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025510
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 4 edm output root files, 52 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold RNTUPLE

@cms-sw/core-l2 , I found 133 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 4.22_RunCosmics2011A step4 max memory diff 424.9 exceeds +/- 90.0 MiB
Error: Workflow 4.53_RunPhoton2012B step3 max memory diff 1122.5 exceeds +/- 90.0 MiB
Error: Workflow 4.53_RunPhoton2012B step2 max memory diff 492.2 exceeds +/- 90.0 MiB
Error: Workflow 5.1_TTbarFS step1 max memory diff 351.9 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step2 max memory diff 308.2 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step3 max memory diff 639.5 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step1 max memory diff 140.8 exceeds +/- 90.0 MiB
Error: Workflow 7.3_CosmicsSPLoose2018 step4 max memory diff 269.9 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step2 max memory diff 311.3 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step4 max memory diff 490.7 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step3 max memory diff 643.7 exceeds +/- 90.0 MiB
Error: Workflow 8.0_BeamHalo step1 max memory diff 140.8 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step1 max memory diff 162.8 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step2 max memory diff 484.8 exceeds +/- 90.0 MiB
Error: Workflow 9.0_Higgs200ChargedTaus step3 max memory diff 699.4 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step2 max memory diff 490.8 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step5 max memory diff 431.0 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step1 max memory diff 165.2 exceeds +/- 90.0 MiB
Error: Workflow 25.0_TTbar step3 max memory diff 466.5 exceeds +/- 90.0 MiB
Error: Workflow 135.4_ZEEFS_13 step3 max memory diff 356.1 exceeds +/- 90.0 MiB
Error: Workflow 135.4_ZEEFS_13 step1 max memory diff 336.1 exceeds +/- 90.0 MiB
Error: Workflow 136.731_RunSinglePh2016B step3 max memory diff 750.1 exceeds +/- 90.0 MiB
Error: Workflow 136.731_RunSinglePh2016B step2 max memory diff 596.6 exceeds +/- 90.0 MiB
Error: Workflow 136.793_RunDoubleEG2017C step3 max memory diff 1250.6 exceeds +/- 90.0 MiB
Error: Workflow 136.793_RunDoubleEG2017C step2 max memory diff 620.7 exceeds +/- 90.0 MiB
Error: Workflow 136.874_RunEGamma2018C step3 max memory diff 919.3 exceeds +/- 90.0 MiB
Error: Workflow 136.874_RunEGamma2018C step2 max memory diff 647.5 exceeds +/- 90.0 MiB
Error: Workflow 139.001_RunMinimumBias2021 step3 max memory diff 662.1 exceeds +/- 90.0 MiB
Error: Workflow 139.001_RunMinimumBias2021 step2 max memory diff 463.8 exceeds +/- 90.0 MiB
Error: Workflow 140.56_RunHI2018 step2 max memory diff 358.3 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step3 max memory diff 626.2 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step2 max memory diff 444.5 exceeds +/- 90.0 MiB
Error: Workflow 312.0_Pyquen_ZeemumuJets_pt10_2760GeV_2022 step1 max memory diff 163.1 exceeds +/- 90.0 MiB
Error: Workflow 1000.0_RunMinBias2011A step3 max memory diff 382.0 exceeds +/- 90.0 MiB
Error: Workflow 1000.0_RunMinBias2011A step5 max memory diff -654.6 exceeds +/- 90.0 MiB
Error: Workflow 1001.0_RunMinBias2011A step2 max memory diff 608.1 exceeds +/- 90.0 MiB
Error: Workflow 1001.0_RunMinBias2011A step3 max memory diff 328.7 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step1 max memory diff 140.2 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step3 max memory diff 703.4 exceeds +/- 90.0 MiB
Error: Workflow 1306.0_SingleMuPt1_UP15 step2 max memory diff 445.7 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step1 max memory diff 144.5 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step3 max memory diff 616.1 exceeds +/- 90.0 MiB
Error: Workflow 1330.0_ZMM_13 step2 max memory diff 440.9 exceeds +/- 90.0 MiB
Error: Workflow 2022.0030001_RunBTagMu2022E_10k step3 max memory diff 1244.3 exceeds +/- 90.0 MiB
Error: Workflow 2022.0030001_RunBTagMu2022E_10k step2 max memory diff 604.7 exceeds +/- 90.0 MiB
Error: Workflow 2023.0020001_RunJetMET02023D_10k step3 max memory diff 1157.3 exceeds +/- 90.0 MiB
Error: Workflow 2023.0020001_RunJetMET02023D_10k step2 max memory diff 617.1 exceeds +/- 90.0 MiB
Error: Workflow 2024.0000001_RunZeroBias2024B_10k step3 max memory diff 1224.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0000001_RunZeroBias2024B_10k step2 max memory diff 567.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0010001_RunJetMET02024C_10k step3 max memory diff 1313.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0010001_RunJetMET02024C_10k step2 max memory diff 573.4 exceeds +/- 90.0 MiB
Error: Workflow 2024.0020001_RunEGamma02024D_10k step3 max memory diff 1167.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0020001_RunEGamma02024D_10k step2 max memory diff 571.5 exceeds +/- 90.0 MiB
Error: Workflow 2024.0030001_RunDisplacedJet2024E_10k step2 max memory diff 566.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0030001_RunDisplacedJet2024E_10k step3 max memory diff 1195.7 exceeds +/- 90.0 MiB
Error: Workflow 2024.0040001_RunPark2MuonLowMass02024F_10k step3 max memory diff 1283.3 exceeds +/- 90.0 MiB
Error: Workflow 2024.0040001_RunPark2MuonLowMass02024F_10k step2 max memory diff 575.0 exceeds +/- 90.0 MiB
Error: Workflow 2024.0050001_RunBTagMu2024G_10k step3 max memory diff 1295.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0050001_RunBTagMu2024G_10k step2 max memory diff 564.2 exceeds +/- 90.0 MiB
Error: Workflow 2024.0060001_RunMuon02024H_10k step3 max memory diff 1206.8 exceeds +/- 90.0 MiB
Error: Workflow 2024.0060001_RunMuon02024H_10k step2 max memory diff 568.1 exceeds +/- 90.0 MiB
Error: Workflow 2024.0070001_RunTau2024I_10k step2 max memory diff 566.9 exceeds +/- 90.0 MiB
Error: Workflow 2024.0070001_RunTau2024I_10k step3 max memory diff 1296.0 exceeds +/- 90.0 MiB
Error: Workflow 2025.0000001_RunZeroBias2025B_10k step3 max memory diff 4191.9 exceeds +/- 90.0 MiB
Error: Workflow 2025.0000001_RunZeroBias2025B_10k step2 max memory diff 2575.6 exceeds +/- 90.0 MiB
Error: Workflow 2025.0010001_RunJetMET02025C_10k step2 max memory diff 2664.2 exceeds +/- 90.0 MiB
Error: Workflow 2025.0010001_RunJetMET02025C_10k step3 max memory diff 4510.8 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step2 max memory diff 488.9 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step3 max memory diff 885.7 exceeds +/- 90.0 MiB
Error: Workflow 10224.0_TTbar_13+2017PU step1 max memory diff 164.1 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step2 max memory diff 514.7 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step3 max memory diff 631.9 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step5 max memory diff 136.0 exceeds +/- 90.0 MiB
Error: Workflow 11634.0_TTbar_14TeV+2022 step1 max memory diff 167.3 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step3 max memory diff 508.5 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step2 max memory diff 520.4 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step1 max memory diff 170.5 exceeds +/- 90.0 MiB
Error: Workflow 12434.0_TTbar_14TeV+2023 step5 max memory diff 135.8 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step5 max memory diff 134.8 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step2 max memory diff 513.9 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step1 max memory diff 163.5 exceeds +/- 90.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step3 max memory diff 556.6 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step3 max memory diff 528.5 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step1 max memory diff 162.8 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step5 max memory diff 124.9 exceeds +/- 90.0 MiB
Error: Workflow 12846.0_ZEE_14+2024 step2 max memory diff 495.1 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step1 max memory diff 163.5 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step2 max memory diff 409.2 exceeds +/- 90.0 MiB
Error: Workflow 13034.0_TTbar_14TeV+2024PU step3 max memory diff 864.8 exceeds +/- 90.0 MiB
Error: Workflow 13234.0_TTbar_14TeV+2022FS step2 max memory diff 212.1 exceeds +/- 90.0 MiB
Error: Workflow 13234.0_TTbar_14TeV+2022FS step1 max memory diff 145.5 exceeds +/- 90.0 MiB
Error: Workflow 14034.0_TTbar_14TeV+2023FS step2 max memory diff 206.5 exceeds +/- 90.0 MiB
Error: Workflow 14034.0_TTbar_14TeV+2023FS step1 max memory diff 145.5 exceeds +/- 90.0 MiB
Error: Workflow 14234.0_TTbar_14TeV+2023FSPU step2 max memory diff 166.4 exceeds +/- 90.0 MiB
Error: Workflow 14234.0_TTbar_14TeV+2023FSPU step1 max memory diff 145.5 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step2 max memory diff 518.6 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step5 max memory diff 135.0 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step3 max memory diff 485.4 exceeds +/- 90.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step2 max memory diff 380.6 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step3 max memory diff 778.9 exceeds +/- 90.0 MiB
Error: Workflow 17034.0_TTbar_14TeV+2025PU step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step2 max memory diff 2270.2 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step3 max memory diff 2723.3 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step5 max memory diff 267.6 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step3 max memory diff 3099.5 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step1 max memory diff 166.3 exceeds +/- 90.0 MiB
Error: Workflow 18634.0_TTbar_14TeV+2026PU step2 max memory diff 2234.9 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step1 max memory diff 153.8 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step3 max memory diff 779.9 exceeds +/- 90.0 MiB
Error: Workflow 25202.0_TTbar_13 step2 max memory diff 500.0 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 461.4 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step3 max memory diff 1404.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step1 max memory diff 170.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step5 max memory diff 1013.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 331.3 exceeds +/- 90.0 MiB
Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step1 max memory diff 170.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step5 max memory diff 1015.9 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 461.8 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step3 max memory diff 1401.2 exceeds +/- 90.0 MiB
Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step1 max memory diff 169.7 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step5 max memory diff 994.0 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 420.8 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step1 max memory diff 148.3 exceeds +/- 90.0 MiB
Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step3 max memory diff 1275.8 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 415.2 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step3 max memory diff 1416.3 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step5 max memory diff 988.5 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step1 max memory diff 145.8 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step3 max memory diff 634.4 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step1 max memory diff 163.8 exceeds +/- 90.0 MiB
Error: Workflow 250202.181_TTbar13TeVPUppmx2018 step4 max memory diff 1079.9 exceeds +/- 90.0 MiB

smuzaffar · 2026-01-27T23:33:05Z

please test with cms-sw/cmsdist#10212

cmsbuild · 2026-01-28T10:30:15Z

-1

Failed Tests: UnitTests RelVals-RNTUPLE
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50944/summary.html
COMMIT: 3a403b4
CMSSW: CMSSW_16_1_X_2026-01-27-1100/el8_amd64_gcc13
Additional Tests: RNTUPLE
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2651/50944/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50944/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50944/git-merge-result

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testDiMuonBiasesPlotting had ERRORS

Failed RelVals-RNTUPLE

The relvals timed out after 4 hours.

Comparison Summary

Summary:

You potentially removed 76 lines from the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 49
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025467
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

smuzaffar · 2026-01-28T22:42:01Z

@makortel , should I kill the hanging jobs to let the comparison job finish (and have the build node used by other jobs) or you still want to debug more? I can restart the job tomorrow if needed?

makortel · 2026-01-28T23:08:57Z

Feel free to kill them (sorry, I could have said that earlier).

cmsbuild · 2026-01-29T00:42:10Z

-1

Failed Tests: UnitTests RelVals-RNTUPLE
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50944/summary.html
COMMIT: 3a403b4
CMSSW: CMSSW_16_1_X_2026-01-27-1100/el8_amd64_gcc13
Additional Tests: RNTUPLE
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2651/50944/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50944/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-96a0b5/50944/git-merge-result

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testDiMuonBiasesPlotting had ERRORS

Failed RelVals-RNTUPLE

The relvals timed out after 4 hours.

Comparison Summary

Summary:

You potentially removed 76 lines from the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 49
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025467
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

RNTUPLE Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 3652166
DQMHistoTests: Total failures: 39
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3652107
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 219 log files, 4 edm output root files, 52 DQM output files
TriggerResults: no differences found

cmsbuild added the orp-pending label Jan 14, 2026

cmsbuild added pending-signatures externals-pending tests-pending labels Jan 14, 2026

cmsbuild added tests-started tests-pending and removed tests-pending tests-started labels Jan 14, 2026

cmsbuild added tests-started and removed tests-pending labels Jan 15, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 15, 2026

gartung mentioned this pull request Jan 15, 2026

List maxmem tests with FLAVORS later. #2653

Merged

cmsbuild added tests-started requires-external labels Jan 17, 2026

smuzaffar mentioned this pull request Jan 17, 2026

Continuation of RNTuple testing in IBs cms-sw/cmssw#49640

Open

cmsbuild added tests-approved and removed tests-started labels Jan 17, 2026

Update run-pr-comparisons

3a403b4

cmsbuild added tests-pending and removed tests-approved requires-external labels Jan 17, 2026

cmsbuild added tests-approved tests-started and removed tests-pending tests-approved labels Jan 21, 2026

smuzaffar mentioned this pull request Jan 23, 2026

Speed up reading *Auxiliary fields in RNTupleTempSource cms-sw/cmssw#49902

Merged

cmsbuild added the requires-external label Jan 27, 2026

cmsbuild added tests-rejected tests-started and removed tests-started tests-rejected labels Jan 28, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 29, 2026

enable dqm comparison job for rntuple #2651

Are you sure you want to change the base?

enable dqm comparison job for rntuple #2651

Uh oh!

Conversation

smuzaffar commented Jan 14, 2026

Uh oh!

smuzaffar commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 15, 2026

Uh oh!

cmsbuild commented Jan 15, 2026

Uh oh!

cmsbuild commented Jan 15, 2026

Uh oh!

smuzaffar commented Jan 15, 2026

Uh oh!

cmsbuild commented Jan 15, 2026

Failed RelVals-RNTUPLE

Comparison Summary

RNTUPLE Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

makortel commented Jan 15, 2026

Max Memory Comparisons exceeding threshold

Uh oh!

gartung commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gartung commented Jan 15, 2026

Uh oh!

smuzaffar commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 17, 2026

Comparison Summary

RNTUPLE Comparison Summary

Max Memory Comparisons exceeding threshold RNTUPLE

Uh oh!

cmsbuild commented Jan 17, 2026

Uh oh!

smuzaffar commented Jan 21, 2026

Uh oh!

cmsbuild commented Jan 21, 2026

Comparison Summary

RNTUPLE Comparison Summary

Max Memory Comparisons exceeding threshold RNTUPLE

Uh oh!

smuzaffar commented Jan 27, 2026

Uh oh!

cmsbuild commented Jan 28, 2026

Failed Unit Tests

Failed RelVals-RNTUPLE

Comparison Summary

Uh oh!

smuzaffar commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

makortel commented Jan 28, 2026

Uh oh!

cmsbuild commented Jan 29, 2026

Failed Unit Tests

Failed RelVals-RNTUPLE

Comparison Summary

RNTUPLE Comparison Summary

cmsbuild commented Jan 14, 2026 •

edited

Loading

gartung commented Jan 15, 2026 •

edited

Loading

smuzaffar commented Jan 16, 2026 •

edited

Loading

smuzaffar commented Jan 28, 2026 •

edited

Loading