Skip to content

Conversation

@vlimant
Copy link
Contributor

@vlimant vlimant commented Oct 8, 2025

PR description:

this PR builds on top of #48988 with the addition of the cmsDriver integration

the mini-from-mini is configured with:
cmsDriver.py step2 -s PAT:Configuration/StandardSequences/REMINI_cff.patAlgosToolsTask --era Run3 -n 100 --process REMINI --conditions auto:phase1_2024_realistic --mc --scenario pp --eventcontent MINIAODSIM --datatier MINIAODSIM --filein /store/mc/Run3Summer23MiniAODv4/TTtoLNu2Q_TuneCP5_13p6TeV_powheg-pythia8/MINIAODSIM/130X_mcRun3_2023_realistic_v14-v2/2520000/373b1353-c6b4-4ec0-bc13-b5c0eff8c189.root --fileout file:step2.root

suggestions to a different syntax are welcome, and I'll update the PR

PR validation:

the limited matrix ran to completion, with some unrelated errors related to file access.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backports to 15.1 and 15.0 are expected

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 8, 2025

cms-bot internal usage

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 8, 2025

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49098/46335

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 8, 2025

A new Pull Request was created by @vlimant for master.

It involves the following packages:

  • Configuration/Applications (operations)
  • Configuration/StandardSequences (operations)
  • DQMOffline/JetMET (dqm)
  • DataFormats/PatCandidates (reconstruction, xpog)
  • PhysicsTools/PatAlgos (reconstruction, xpog)

@cmsbuild, @ctarricone, @davidlange6, @fabiocos, @ftenchini, @ftorrresd, @gabrielmscampos, @hqucms, @jfernan2, @mandrenguyen, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@AlexDeMoor, @GiacomoSguazzoni, @Martin-Grunewald, @Ming-Yan, @Senphy, @VinInn, @VourMa, @ahinzmann, @andrzejnovak, @azotz, @castaned, @dgulhan, @elusian, @fabiocos, @felicepantaleo, @gkasieczka, @gouskos, @gpetruc, @hatakeyamak, @jdamgov, @jdolen, @makortel, @mariadalfonso, @mbluj, @missirol, @mmarionncern, @mmasciov, @mmusich, @mtosi, @nhanvtran, @pavlo-kashko, @philippgadow, @rappoccio, @rociovilar, @rovere, @sameasy, @schoef, @seemasharmafnal, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@vlimant
Copy link
Contributor Author

vlimant commented Oct 8, 2025

enable nano

@vlimant
Copy link
Contributor Author

vlimant commented Oct 8, 2025

please test

@smuzaffar
Copy link
Contributor

@vlimant , bot das cache for 3 failing relvals is now updated. As I wrote earlier bot only force update its cache after 7 days if it already has a valid result (non-empty) . Anyway, I have forced run das-query jenkins job to update cache for these 3 relvals.

I have now restarted failing jobs to use the newer cache, hopefully things should work this time

@vlimant
Copy link
Contributor Author

vlimant commented Oct 15, 2025

thanks @smuzaffar

@AdrianoDee
Copy link
Contributor

AdrianoDee commented Oct 15, 2025

As I wrote earlier bot only force update its cache after 7 days if it already has a valid result (non-empty) .

@smuzaffar, thanks!

Since we are here, something I don't understand yet is how the original DAS cached result was non-empty in the first place, given the site=T2_CH_CERN in the query and the fact that the samples were only on tape.

(I tested an analogous das query locally, and it would give no result)

@smuzaffar
Copy link
Contributor

smuzaffar commented Oct 15, 2025

@AdrianoDee , In past (many many years ago) we wanted IB's relvals to access only data at CERN, so we had added site=T2_CH_CERN to das queries and explicitly requested the dataset/selected block to be present at CERN. Then came xrootd AAA service which we enabled for IB/PR tests so that we can access data from anywhere and cache the root files which are not in ibeos cache at CERN. This way we do not have to request the full dataset/block at CERN. The assumption was that if das returns files then they can be accessed via xrootd AAA redirector.

das-query cache script works on the above assumption. So if it can not find any data with site=T2_CH_CERN (which is present in all das queries run y runTheMatrix) then it drops the site=T2_CH_CERN requirement and try to get the files for a dataset and cache those results.

So for issue we have seen here was that though das was returning a list of files but they were not available on disk or not accessible via AAA. So cmsRun was not able to read those files and bot was not able to cache the root files.

@smuzaffar
Copy link
Contributor

by the way, all 3 failing workflows now passed

@AdrianoDee
Copy link
Contributor

Clear, thanks a lot.

@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-43dfc2/48682/summary.html
COMMIT: 8e9096d
CMSSW: CMSSW_16_0_X_2025-10-14-1100/el8_amd64_gcc13
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49098/48682/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 1147 lines to the logs
  • Reco comparison results: 820 differences found in the comparisons
  • DQMHistoTests: Total files compared: 51
  • DQMHistoTests: Total histograms compared: 3940073
  • DQMHistoTests: Total failures: 3
  • DQMHistoTests: Total nulls: 10
  • DQMHistoTests: Total successes: 3940040
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.5920000000000004 KiB( 50 files compared)
  • DQMHistoSizes: changed ( 10224.0,... ): 0.008 KiB MessageLogger/Errors
  • DQMHistoSizes: changed ( 10224.0,... ): 0.008 KiB MessageLogger/Warnings
  • Checked 218 log files, 188 edm output root files, 51 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

  • Nano ERROR: Missing ref/2500.0002_NANOmc106Xul16APVv2-size.json
  • Nano ERROR: Missing ref/2500.0002_NANOmc106Xul16APVv2-size.json
  • Nano ERROR: Missing ref/2500.0003_NANOmc106Xul17v2-size.json
  • Nano ERROR: Missing ref/2500.0003_NANOmc106Xul17v2-size.json
  • Nano ERROR: Missing ref/2500.0004_NANOmc106Xul18v2-size.json
  • Nano ERROR: Missing ref/2500.0004_NANOmc106Xul18v2-size.json
  • Nano ERROR: Missing ref/2500.0401_NANOmcUL16APVMini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0401_NANOmcUL16APVMini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0402_NANOmcUL16Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0402_NANOmcUL16Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0403_NANOmcUL17Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0403_NANOmcUL17Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0404_NANOmcUL18Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0404_NANOmcUL18Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0501_NANOdataUL16Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0501_NANOdataUL16Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0502_NANOdataUL17Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0502_NANOdataUL17Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0503_NANOdataUL18Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.0503_NANOdataUL18Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.1201_NANOmc23Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.1201_NANOmc23Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.1301_NANOdata23Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.1301_NANOdata23Mini2Mini-size.json
  • Nano ERROR: Missing ref/2500.1501_ScoutingNANOdata124Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1501_ScoutingNANOdata124Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1502_ScoutingNANOmonitordata124Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1502_ScoutingNANOmonitordata124Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1503_ScoutingNANOdata130Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1503_ScoutingNANOdata130Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1504_ScoutingNANOmonitordata130Xrun3-size.json
  • Nano ERROR: Missing ref/2500.1504_ScoutingNANOmonitordata130Xrun3-size.json
workflow id kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
NANOmc106Xul16v2 2500.0001 3.085 3.085 0.000 ( +0.0% ) 3.85 3.77 +2.1% 2.619 2.614
NANOdata106Xul16v2 2500.0101 1.539 1.539 0.000 ( +0.0% ) 6.34 6.40 -0.9% 2.679 2.687
NANOdata106Xul17v2 2500.0102 1.926 1.926 0.000 ( +0.0% ) 4.21 4.19 +0.5% 2.884 2.836
NANOdata106Xul18v2 2500.0103 1.890 1.890 0.000 ( +0.0% ) 5.08 5.06 +0.4% 2.748 2.725
NANOmcUL16APVreMINI 2500.0201 0.022 0.022 0.001 ( +2.9% ) 1.84 1.83 +0.7% 2.575 2.611
NANOmcUL16reMINI 2500.0202 0.022 0.022 0.000 ( +2.0% ) 1.79 1.76 +1.7% 2.589 2.596
NANOmcUL17reMINI 2500.0203 0.022 0.022 0.000 ( +2.0% ) 1.60 1.58 +1.2% 2.716 2.568
NANOmcUL18reMINI 2500.0204 0.022 0.022 0.001 ( +2.7% ) 1.57 1.42 +10.2% 2.737 2.710
NANOdataUL16APVreMINI 2500.0301 0.035 0.035 0.000 ( +0.0% ) 1.60 1.64 -2.3% 2.679 2.685
NANOdataUL16reMINI 2500.0302 0.036 0.036 0.000 ( +0.0% ) 1.60 1.63 -1.5% 2.598 2.615
NANOdataUL17reMINI 2500.0303 0.037 0.037 0.000 ( +0.0% ) 1.53 1.55 -1.4% 2.696 2.709
NANOdataUL18reMINI 2500.0304 0.036 0.036 0.000 ( +0.0% ) 1.55 1.57 -1.1% 2.688 2.703
NANOmc130X 2500.1001 3.220 3.220 0.000 ( +0.0% ) 4.38 4.37 +0.2% 3.019 3.073
NANOdata130Xrun3 2500.1101 1.799 1.799 0.000 ( +0.0% ) 6.48 6.61 -2.0% 2.747 2.800
muPOGNANOmc140X 2500.2201 2.065 2.065 0.000 ( +0.0% ) 5.63 5.87 -4.1% 2.033 2.235
EGMNANOmc140X 2500.2202 3.876 3.876 0.000 ( +0.0% ) 4.09 4.17 -1.9% 2.299 2.433
BTVNANOmc140X 2500.2203 10.607 10.607 0.000 ( +0.0% ) 2.58 2.60 -0.9% 2.490 2.572
jmeNANOmc140X 2500.2204 6.667 6.667 0.000 ( +0.0% ) 1.15 1.16 -1.6% 2.177 2.384
lepTrackInfoNANOmc140X 2500.2206 3.570 3.570 0.000 ( +0.0% ) 4.09 4.13 -0.9% 2.336 2.407
ScoutingNANOmc140X 2500.2207 1.488 1.488 0.000 ( +0.0% ) 16.72 16.52 +1.2% 1.879 1.885
ScoutingNANOwithPromptmc140X 2500.2208 4.476 4.476 0.000 ( +0.0% ) 3.42 3.36 +1.8% 2.650 2.567
muPOGNANO140Xrun3 2500.2301 1.469 1.469 0.000 ( +0.0% ) 8.16 8.28 -1.4% 2.305 2.237
EGMNANOdata140Xrun3 2500.2302 2.623 2.623 0.000 ( +0.0% ) 5.92 5.88 +0.7% 2.290 2.618
BTVNANOdata140Xrun3 2500.2303 5.612 5.612 0.000 ( +0.0% ) 3.75 3.84 -2.3% 2.434 2.534
jmeNANOdata140Xrun3 2500.2304 3.976 3.976 0.000 ( +0.0% ) 1.48 1.53 -3.2% 2.274 2.396
lepTrackInfoNANOdata140Xrun3 2500.2306 2.413 2.413 0.000 ( +0.0% ) 5.84 5.98 -2.4% 2.726 2.616
ScoutingNANOdata140Xrun3 2500.2307 1.101 1.101 0.000 ( +0.0% ) 30.03 30.50 -1.5% 1.541 1.463
ScoutingNANOmonitordata140Xrun3 2500.2308 2.784 2.784 0.000 ( +0.0% ) 5.17 5.12 +1.0% 2.962 3.070
ScoutingNANOmonitorWithPromptdata140Xrun3 2500.2309 2.784 2.784 0.000 ( +0.0% ) 5.17 5.03 +2.8% 2.926 3.080
L1ScoutingSelectionNANOdata140Xrun3 2500.2311 0.081 0.081 0.000 ( +0.0% ) 45.67 46.27 -1.3% 0.716 0.718
L1ScoutingNANOdata140Xrun3 2500.231 0.082 0.082 0.000 ( +0.0% ) 3.97 4.01 -1.1% 0.946 0.837
l1DPGNANO140Xrun3 2500.2401 9.404 9.404 0.000 ( +0.0% ) 6.57 6.56 +0.1% 1.869 1.826
muDPGNANO140Xrun3 2500.2402 10.377 10.377 0.000 ( +0.0% ) 1.67 1.66 +0.6% 1.717 1.666
muDPGNANOBkg140Xrun3 2500.2403 6.401 6.401 0.000 ( +0.0% ) 15.74 16.20 -2.8% 1.189 1.237
hcalDPGNANO140Xrun3 2500.2404 486.015 486.015 0.000 ( +0.0% ) 1.16 1.17 -1.2% 1.580 1.618
hcalDPGCalibNANO140Xrun3 2500.2405 826.418 826.418 0.000 ( +0.0% ) 1.52 1.51 +0.6% 1.635 1.554
hcalDPGMCNANO140Xrun3 2500.2501 645.333 645.333 0.000 ( +0.0% ) 1.67 1.68 -0.5% 1.703 1.656
NANOmc2024reMINI 2500.2601 0.021 0.021 0.000 ( +0.0% ) 1.89 1.88 +0.5% 2.694 2.639
NANOdata2024reMINI 2500.2701 0.036 0.036 0.000 ( +0.0% ) 1.94 1.95 -0.5% 2.615 2.565
NANOmc150X 2500.3001 3.217 3.217 0.000 ( +0.0% ) 19.39 18.67 +3.9% 2.706 2.701
NANOdata150X 2500.3101 2.144 2.144 0.000 ( +0.0% ) 34.80 36.11 -3.6% 2.302 2.220
NANOdata150X 2500.3102 2.630 2.630 0.000 ( +0.0% ) 31.02 29.62 +4.7% 2.230 2.435
muPOGNANOmc150X 2500.3201 2.255 2.255 0.000 ( +0.0% ) 18.16 17.13 +6.0% 2.526 2.579
EGMNANOmc150X 2500.3202 4.111 4.111 0.000 ( +0.0% ) 16.37 15.46 +5.9% 2.523 2.690
BTVNANOmc150X 2500.3203 10.552 10.552 0.000 ( +0.0% ) 5.16 5.04 +2.4% 2.684 2.864
jmeNANOmc150X 2500.3204 6.760 6.760 0.000 ( +0.0% ) 1.39 1.34 +4.0% 2.660 2.806
jmeNANOrePuppimc150X 2500.3205 6.763 6.763 0.000 ( +0.0% ) 1.33 1.24 +7.4% 3.149 2.915
lepTrackInfoNANOmc150X 2500.3206 3.721 3.721 0.000 ( +0.0% ) 17.05 16.19 +5.3% 2.417 2.421
ScoutingNANOmc150X 2500.3207 1.563 1.563 0.000 ( +0.0% ) 20.59 19.84 +3.8% 2.395 2.333
ScoutingNANOwithPromptmc150X 2500.3208 4.775 4.775 0.000 ( +0.0% ) 11.04 10.67 +3.5% 2.744 2.711
BPHNANOmc150X 2500.3209 4.127 4.127 0.000 ( +0.0% ) 18.28 17.44 +4.9% 2.575 2.695
muPOGNANO150Xrun3 2500.3301 1.125 1.125 0.000 ( +0.0% ) 33.33 31.16 +7.0% 2.329 2.353
EGMNANOdata150Xrun3 2500.3302 2.985 2.985 0.000 ( +0.0% ) 28.52 26.43 +7.9% 2.170 2.268
BTVNANOdata150Xrun3 2500.3303 7.971 7.971 0.000 ( +0.0% ) 6.36 5.99 +6.2% 2.660 2.548
jmeNANOdata150Xrun3 2500.3304 4.349 4.349 0.000 ( +0.0% ) 1.71 1.61 +6.2% 2.326 2.328
jmeNANOrePuppidata150Xrun3 2500.3305 4.332 4.332 0.000 ( +0.0% ) 1.50 1.38 +8.6% 2.827 2.555
lepTrackInfoNANOdata150Xrun3 2500.3306 2.625 2.625 0.000 ( +0.0% ) 30.62 27.58 +11.0% 2.103 2.404
ScoutingNANOdata150Xrun3 2500.3307 1.738 1.738 0.000 ( +0.0% ) 32.33 30.88 +4.7% 1.490 1.518
ScoutingNANOmonitordata150Xrun3 2500.3308 1.400 1.400 0.000 ( +0.0% ) 27.44 25.73 +6.6% 2.137 2.321
ScoutingNANOmonitorWithPromptdata150Xrun3 2500.3309 2.602 2.602 0.000 ( +0.0% ) 18.93 18.01 +5.1% 2.687 2.481
BPHNANOdata150Xrun3 2500.331 2.632 2.632 0.000 ( +0.0% ) 36.71 32.67 +12.4% 2.251 2.500
TTbarMINIAOD10.6_UL18v2 2500.9001 1.819 1.819 0.000 ( +0.0% ) 37.42 33.71 +11.0% 1.874 1.865
TTbarMINIAOD14.0 2500.9002 1.665 1.665 0.000 ( +0.0% ) 35.98 33.43 +7.6% 1.460 1.454
DYToLL_M-50_13TeV_pythia8 2500.9101 15.005 15.005 0.000 ( +0.0% ) 6.46 6.59 -2.0% 1.089 1.079
DYToll01234Jets_5f_LO_MLM_Madgraph_LHE_13TeV 2500.9102 0.150 0.310 -0.160 ( -51.6% ) 3.39 2.90 +16.9% 0.853 0.847
TTbar_Pow_LHE_13TeV 2500.9103 0.110 0.110 0.000 ( +0.0% ) 2.30 2.29 +0.3% 0.858 0.849

NANO Comparison Summary

The workflows 2500.0204, 2500.0203, 2500.0202, 2500.0201, 2500.0003, 2500.0002 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

  • You potentially added 1652 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 13747 differences found in the comparisons
  • DQMHistoTests: Total files compared: 22
  • DQMHistoTests: Total histograms compared: 11554
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 11554
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -111211.56599999999 KiB( 19 files compared)
  • DQMHistoSizes: changed ( 2500.0201,... ): -5306.582 KiB JetMET/JetValidation
  • DQMHistoSizes: changed ( 2500.0201,... ): -2583.706 KiB Muons_miniAOD/MuonRecoAnalyzer
  • DQMHistoSizes: changed ( 2500.0201,... ): -995.618 KiB ParticleFlow/Offset
  • DQMHistoSizes: changed ( 2500.0201,... ): -640.710 KiB JetMET/PFCandidates
  • DQMHistoSizes: changed ( 2500.0201,... ): -559.571 KiB RecoTauV/miniAODValidation
  • DQMHistoSizes: changed ( 2500.0201,... ): -448.898 KiB ParticleFlow/slimmedJetValidation
  • DQMHistoSizes: changed ( 2500.0201,... ): -300.393 KiB ParticleFlow/JetResponse
  • DQMHistoSizes: changed ( 2500.0201,... ): -279.518 KiB JetMET/METValidation
  • DQMHistoSizes: changed ( 2500.0201,... ): -263.359 KiB ParticleFlow/packedGenParticlesValidation
  • DQMHistoSizes: changed ( 2500.0201,... ): -263.359 KiB ParticleFlow/slimmedElectronValidation
  • DQMHistoSizes: changed ( 2500.0201 ): ...
  • Checked 136 log files, 81 edm output root files, 22 DQM output files
  • TriggerResults: no differences found

@vlimant
Copy link
Contributor Author

vlimant commented Oct 17, 2025

@cms-sw/xpog-l2 @cms-sw/pdmv-l2 : please go ahead and review/sign so that we get it in the 16.0 dev cycle, and add work on top to fix the nano vs nano discrepancies

@Moanwar
Copy link
Contributor

Moanwar commented Oct 17, 2025

+Upgrade

@AdrianoDee
Copy link
Contributor

+pdmv

@ftorrresd
Copy link
Contributor

+xpog

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit c923b8f into cms-sw:master Oct 17, 2025
12 checks passed
@mbluj mbluj mentioned this pull request Nov 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.