Fix replacing extensions in WorkFlowRunner.py

[iarspider]

PR description:

Fixes #48360:

• Handle any number of files passed to --filein
• Avoid search space explosion by rewriting regex

PR validation:

Bot tests

[cmsbuild]

cms-bot internal usage

[iarspider]

please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48366/45251

[cmsbuild]

A new Pull Request was created by @iarspider for CMSSW_15_1_RNTUPLE_X.

It involves the following packages:

• Configuration/PyReleaseValidation (upgrade, pdmv)

@AdrianoDee, @Moanwar, @DickyChant, @miquork, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @slomeo this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[iarspider]

test parameters:

• workflow = 1041.0, 1046.0

[iarspider]

please test

[iarspider]

please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48366/45253

[cmsbuild]

Pull request #48366 was updated. @AdrianoDee, @Moanwar, @DickyChant, @miquork, @srimanob, @subirsarkar can you please check and sign again.

[cmsbuild]

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d0e925/46826/summary.html
COMMIT: f2b0062
CMSSW: CMSSW_15_1_RNTUPLE_X_2025-06-17-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48366/46826/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @iarspider Do not access itr->second if itr is invalid #48348
• @hqucms Fix JMEnano workflow 11634.15 #48349
• @llunerti Protect SoftMuonMvaRun3Estimator from returning nan #48352
• @smuzaffar [GCC14]Included header algorithm for std::max_element #48344
• @BenjaminRS [L1T] Add Displaced Vertices into the default running of the P2 L1T Emulator (Phase 2) #48340
• @bsunanda Run3-alca253 Provide run range in the IsoTrackAnalyzer code and provide cfg's for ACaReco and AlCaRaw o/p's #48341
• @mmusich remove obsolete parameter transactionId from CondDBESSource #48353
• @iarspider [DBG_X] Fix trackerMaterialAnalysisPlots unit test failure #48338
• @civanch [SIM] Update handling of G4Regions, improved printouts, added Phase2 stacking action  #48337
• @BlancoFS Filter infinite loops in muon trajectory reconstruction [CMSHLT-3557] #48318
• @SegmentLinking Vectorize data pre-processing when training track embeddings in LST #48335
• @cerminar [L1T] Restructure handling of Calorimeter input to Correlator Layer-1 + new TkElectrons in barrel (Phase-2) #48002
• @fabiocos MTD geometry: update defaults test scenario for MTD to D121 #48333
• @BenjaminRS [L1T] Adding displaced vertices into L1Nano (Phase 2) #48326

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d0e925/46826/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d0e925/46826/git-merge-result

Unit Tests

I found 2 errors in the following unit tests:

---> test GeometryMTDCommonDataTestDriver had ERRORS
---> test SimG4CMSForwardTestDriver had ERRORS

RelVals

• 25202.0

----- Begin Fatal Exception 19-Jun-2025 01:24:45 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type RNTupleSource
   [2] Calling StorageFactory::open()
   [3] Calling File::sysopen()
Exception Message:
Failed to open the file 'step3_inMINIAODSIM..rntpl'
   Additional Info:
      [a] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

• 25.0

----- Begin Fatal Exception 19-Jun-2025 01:13:22 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type RNTupleSource
   [2] Calling StorageFactory::open()
   [3] Calling File::sysopen()
Exception Message:
Failed to open the file 'step3..rntpl'
   Additional Info:
      [a] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

• 135.4

----- Begin Fatal Exception 19-Jun-2025 01:09:39 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type RNTupleSource
   [2] Calling StorageFactory::open()
   [3] Calling File::sysopen()
Exception Message:
Failed to open the file 'step1..rntpl'
   Additional Info:
      [a] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

• 24834.911
• 29700.0
• 1330.0
• 250202.181
• 1000.0
• 1041.0
• 1046.0
• 29834.999
• 1001.0
• 29634.75
• 4.22
• 29696.0
• 10224.0
• 29634.911
• 29634.0

[iarspider]

please test

[iarspider]

test parameters

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48366/45255

[cmsbuild]

Pull request #48366 was updated. @AdrianoDee, @Moanwar, @cmsbuild, @DickyChant, @miquork, @srimanob, @subirsarkar can you please check and sign again.

[iarspider]

please test

[iarspider]

Command runTheMatrix.py --maxSteps=0 -l 1041.0 --useInput all --job-reports --command "--customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix 'python2 /data/cmsbld/jenkins/workspace/ib-run-relvals/cms-bot/monitor_workflow.py timeout --signal SIGTERM 9000 '" -t 4 no longer hangs.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d0e925/46829/summary.html
COMMIT: a03f6b3
CMSSW: CMSSW_15_1_RNTUPLE_X_2025-06-18-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48366/46829/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 616 lines to the logs
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4050825
• DQMHistoTests: Total failures: 965
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4049840
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 8 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[iarspider]

@cms-sw/pdmv-l2 @cms-sw/upgrade-l2 could you please review and sign this? Thanks!

[Moanwar]

+Upgrade

[iarspider]

urgent
to fix running RelVals in RNTUPLE_X

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_15_1_RNTUPLE_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_15_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1