Move FWCore and Configuration python scripts to python3 #34352

davidlange6 · 2021-07-06T14:29:00Z

No description provided.

cmsbuild · 2021-07-06T14:36:07Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34352/23730

This PR adds an extra 148KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/Applications/python/cmsDriverOptions.py modified in PR(s): Fix isData when neither --data nor --mc is used #34302
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): [DD4hep] Add DD4hep Run 3 workflow that reads geometry from DB #34343, Add GPUParams to runTheMatrix workflows #33538, Bianca pinolini ecal reco phase2 commit unimib #34212

cmsbuild · 2021-07-06T14:36:29Z

A new Pull Request was created by @davidlange6 (David Lange) for master.

It involves the following packages:

Configuration/Applications
Configuration/DataProcessing
Configuration/PyReleaseValidation
FWCore/ParameterSet
FWCore/Skeletons

@smuzaffar, @Dr15Jones, @jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @makortel, @srimanob, @silviodonato, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @slomeo, @wddgit, @fabiocos this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Dr15Jones · 2021-07-06T14:40:18Z

@davidlange6 did you try running the mk* scripts to be sure they work under python3?

davidlange6 · 2021-07-06T14:48:46Z

I ran at least mkesprod and mkedprod. any of the others follow a different pattern that I should test out?

davidlange6 · 2021-07-06T14:49:01Z

please test

Dr15Jones · 2021-07-06T14:51:20Z

@davidlange6 mkskel is the only outlier I can think of.

davidlange6 · 2021-07-06T15:02:38Z

that runs ok as well

cmsbuild · 2021-07-06T23:22:24Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4b5303/16501/summary.html
COMMIT: deec5ec
CMSSW: CMSSW_12_0_X_2021-07-06-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34352/16501/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 58 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2785924
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2785899
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

makortel · 2021-07-07T01:08:04Z

+1

Reco comparison differences look like the order Paths has changed between the baseline and py3 version of these files, presumably order of dictionary keys in ConfigBuilder or something similar. Py3 build has anyway been validated by PdmV.

srimanob · 2021-07-07T01:53:31Z

FWCore/Skeletons/python/utils.py

        print("\n### generated code\n")
        print(func)
    # compile python code as exec statement
+    print(func)


Why do we need this line also outside debug? If need, should we move it out from debug (line 102, above)?

good question. I'm not sure how I added that, but it was not intended. Fixing that.

srimanob · 2021-07-07T11:35:56Z

Please test

cmsbuild · 2021-07-07T14:40:36Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4b5303/16542/summary.html
COMMIT: 0d6c18d
CMSSW: CMSSW_12_0_X_2021-07-06-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34352/16542/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestConfigDP had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 61 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2785924
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2785893
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

davidlange6 · 2021-07-07T14:46:23Z

this failed test hit a timeout

makortel · 2021-07-07T15:12:57Z

+1

srimanob · 2021-07-07T15:41:02Z

+Upgrade

chayanit · 2021-07-07T15:42:27Z

+1

qliphy · 2021-07-07T15:49:41Z

+operations

cmsbuild · 2021-07-07T15:50:02Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-07-07T15:50:27Z

+1

qliphy · 2021-07-07T15:51:15Z

merge

davidlange6 added 3 commits July 6, 2021 15:04

move to python3

a30ec46

move to python3

c134b92

futher stringio cleaning

deec5ec

cmsbuild added this to the CMSSW_12_0_X milestone Jul 6, 2021

cmsbuild added code-checks-pending core-pending operations-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Jul 6, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 6, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 6, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 6, 2021

cmsbuild added core-approved and removed core-pending labels Jul 7, 2021

srimanob reviewed Jul 7, 2021

View reviewed changes

cmsbuild added the tests-rejected label Jul 7, 2021

cmsbuild added tests-started and removed tests-rejected labels Jul 7, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 7, 2021

cmsbuild added core-approved and removed core-pending labels Jul 7, 2021

cmsbuild added upgrade-approved and removed upgrade-pending labels Jul 7, 2021

cmsbuild added pdmv-approved and removed pdmv-pending labels Jul 7, 2021

cmsbuild added fully-signed operations-approved and removed operations-pending pending-signatures labels Jul 7, 2021

cmsbuild added orp-approved and removed orp-pending labels Jul 7, 2021

cmsbuild merged commit e2e55a4 into cms-sw:master Jul 7, 2021

This was referenced Jul 7, 2021

Remove duplicate service from T25&T26 #34388

Merged

cleanup python2; use rpm macros for python versions cms-sw/cmsdist#7106

Merged

Move FWCore and Configuration python scripts to python3 #34352

Move FWCore and Configuration python scripts to python3 #34352

Uh oh!

Conversation

davidlange6 commented Jul 6, 2021

Uh oh!

cmsbuild commented Jul 6, 2021

Uh oh!

cmsbuild commented Jul 6, 2021

Uh oh!

Dr15Jones commented Jul 6, 2021

Uh oh!

davidlange6 commented Jul 6, 2021

Uh oh!

davidlange6 commented Jul 6, 2021

Uh oh!

Dr15Jones commented Jul 6, 2021

Uh oh!

davidlange6 commented Jul 6, 2021

Uh oh!

cmsbuild commented Jul 6, 2021

Comparison Summary

Uh oh!

makortel commented Jul 7, 2021

Uh oh!

srimanob Jul 7, 2021

Choose a reason for hiding this comment

Uh oh!

davidlange6 Jul 7, 2021

Choose a reason for hiding this comment

Uh oh!

srimanob commented Jul 7, 2021

Uh oh!

cmsbuild commented Jul 7, 2021

Unit Tests

Comparison Summary

Uh oh!

davidlange6 commented Jul 7, 2021

Uh oh!

makortel commented Jul 7, 2021

Uh oh!

srimanob commented Jul 7, 2021

Uh oh!

chayanit commented Jul 7, 2021

Uh oh!

qliphy commented Jul 7, 2021

Uh oh!

cmsbuild commented Jul 7, 2021

Uh oh!

qliphy commented Jul 7, 2021

Uh oh!

qliphy commented Jul 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants