Fix inconsistent behaviour of --dump_python

[pkalbhor]

PR description:

Bug fix for cmsDriver.py which when used with --dump_python was giving inconsistent/incorrect output. Issue was raised few times already Issue#37976, Issue#38686

P.S. My first PR to cmssw :)

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39127/31684

• This PR adds an extra 12KB to repository

[cmsbuild]

A new Pull Request was created by @pkalbhor (Pritam Kalbhor) for master.

It involves the following packages:

• Configuration/Applications (operations)

@perrotta, @rappoccio, @cmsbuild, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@makortel, @missirol, @fabiocos, @Martin-Grunewald this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[qliphy]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7632c8/26969/summary.html
COMMIT: ffdbb35
CMSSW: CMSSW_12_5_X_2022-08-21-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39127/26969/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3693040
• DQMHistoTests: Total failures: 13
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3693004
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 212 log files, 49 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[perrotta]

assign core
(I am not sure that this is the correct assignment for this PR: I would like however that some expert can review it at some point...)

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

I am not sure that this is the correct assignment for this PR

It is not :)

Anyway, I think it is not worse than the current implementation (and certainly works around #37976). Using edmConfigDump to read and write the same file smells a bit, but after looking the implementation it is not worse than what cmsDriver.py does already, i.e. keep the read file handle open while opening the file again for writing (while truncating it).

I think the cmsDriver.py should check the return value of edmConfigDump, and if non-zero, terminate the program immediately with the same return value.

[rappoccio]

@makortel can you confirm with the usual "+1" if you're okay with this (as discussed in ORP meeting)?

[makortel]

I'd like to see

I think the cmsDriver.py should check the return value of edmConfigDump, and if non-zero, terminate the program immediately with the same return value.

addressed first.

[srimanob]

@pkalbhor
Could you please provide the fix following the suggestion of @makortel here
Thanks.

[cmsbuild]

Pull request #39127 was updated. @perrotta, @smuzaffar, @rappoccio, @Dr15Jones, @makortel, @cmsbuild, @fabiocos, @davidlange6 can you please check and sign again.

[makortel]

@cmsbuild, please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7632c8/27204/summary.html
COMMIT: 32dda69
CMSSW: CMSSW_12_6_X_2022-08-30-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39127/27204/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @nhduongvn Add new data catalogs from Rucio storage description (RucioCatalog) and use it by default instead of trivial data catalogs (TrivialCatalog) #37278
• @bsunanda Run3-gex138C Try to avoid getByLabel and instead use getByToken in a few modules of Calibration #39216
• @waredjeb TICL version 4: CLUE3D as default Pattern Recognition, Linking #38880
• @guitargeek Merge .h and .cc files of plugins in RecoLocalCalo/EcalRecProducers #39171
• @bsunanda Run3-gex138E Use getByToken instead of getByLabel in Calibration/EcalCalibAlgos #39241
• @lathomas Compute/store unprefirable event information by default in MINIAOD  #39169
• @dpilipov Adds FED ID 1405 for uGT Testcrate for L1T Ntuples #39104
• @bsunanda Run3-gex138D Use getByToken instead of getByLabel in CaloOnlineTools/EcalTools #39226

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7632c8/27204/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7632c8/27204/git-merge-result

Comparison Summary

Summary:

• No significant changes to the logs found
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 9955 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3595258
• DQMHistoTests: Total failures: 22336
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3572900
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -80769.815 KiB( 50 files compared)
• DQMHistoSizes: changed ( 23234.0,... ): -10466.235 KiB HGCAL/HGCalValidator
• DQMHistoSizes: changed ( 39434.0,... ): -9874.222 KiB HGCAL/HGCalValidator
• Checked 212 log files, 49 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[makortel]

@cmsbuild, please test

To clean up the comparisons

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7632c8/27208/summary.html
COMMIT: 32dda69
CMSSW: CMSSW_12_6_X_2022-08-30-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39127/27208/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3620448
• DQMHistoTests: Total failures: 14
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3620412
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 212 log files, 49 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[makortel]

+core

(with the disclaimer that this package is not under core responsibility)

[perrotta]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.