Update setuptools to 58.2.0; drop nose, hyperas, hyperopt #7400

iarspider · 2021-10-20T08:09:13Z

No description provided.

iarspider · 2021-10-20T08:09:21Z

cmsbuild · 2021-10-20T08:09:33Z

A new Pull Request was created by @iarspider for branch IB/CMSSW_12_1_X/master.

@smuzaffar, @iarspider, @ddaina can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2021-10-20T08:26:22Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/19758/summary.html
COMMIT: f315ec3
CMSSW: CMSSW_12_1_X_2021-10-19-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7400/19758/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

byte-compiling /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/55f1a5e40197a75f8e5c4746a70a5217/opt/cmssw/slc7_amd64_gcc900/external/py3-setuptools/58.2.0-55f1a5e40197a75f8e5c4746a70a5217/lib/python3.9/site-packages/setuptools/command/upload_docs.py to upload_docs.cpython-39.pyc
byte-compiling /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/55f1a5e40197a75f8e5c4746a70a5217/opt/cmssw/slc7_amd64_gcc900/external/py3-setuptools/58.2.0-55f1a5e40197a75f8e5c4746a70a5217/lib/python3.9/site-packages/setuptools/extern/__init__.py to __init__.cpython-39.pyc
running install_egg_info
Writing /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/55f1a5e40197a75f8e5c4746a70a5217/opt/cmssw/slc7_amd64_gcc900/external/py3-setuptools/58.2.0-55f1a5e40197a75f8e5c4746a70a5217/lib/python3.9/site-packages/setuptools-58.2.0-py3.9.egg-info
running install_scripts
error: cannot copy tree 'build/scripts-3.9': not a directory
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.p456T9 (%install)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope

iarspider · 2021-10-20T10:20:21Z

Not sure why it fails in cmsdist, but works in spack. Here is what Spack does:

python3 '-s' 'setup.py' '--no-user-cfg' 'build'
python3 '-s' 'setup.py' '--no-user-cfg' 'install' '--prefix=%{i}' '--single-version-externally-managed' '--root=%{i}' '--install-purelib=lib/python3.9/site-packages' '--install-platlib=lib/python3.9/site-packages' '--install-scripts=bin' '--install-data=' '--install-headers=include/python3.9'

(here the full installation path was replaced with %{i} for clarity)

And here is what cmsdist does:

python3 setup.py build
python3 setup.py install --single-version-externally-managed --record=/dev/null --skip-build --prefix=%{i}

iarspider · 2021-10-20T10:25:38Z

Spack log fragment:

byte-compiling %{i}/lib/python3.9/site-packages/_distutils_hack/override.py to override.cpython-39.pyc
byte-compiling %{i}/lib/python3.9/site-packages/_distutils_hack/__init__.py to __init__.cpython-39.pyc
running install_egg_info
running egg_info
writing setuptools.egg-info/PKG-INFO
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
writing requirements to setuptools.egg-info/requires.txt
writing top-level names to setuptools.egg-info/top_level.txt
reading manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.txt' under directory 'docs'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
adding license file 'LICENSE'
writing manifest file 'setuptools.egg-info/SOURCES.txt'
Copying setuptools.egg-info to %{i}/lib/python3.9/site-packages/setuptools-58.2.0-py3.9.egg-info
running install_scripts
creating %{i}/lib/python3.9/site-packages/distutils-precedence.pth

cmsdist log:

byte-compiling %{i}/lib/python3.9/site-packages/_distutils_hack/__init__.py to __init__.cpython-39.pyc
byte-compiling %{i}/lib/python3.9/site-packages/_distutils_hack/override.py to override.cpython-39.pyc
running install_egg_info
Writing %{i}/lib/python3.9/site-packages/setuptools-58.2.0-py3.9.egg-info
running install_scripts
running build_scripts
error: cannot copy tree 'build/scripts-3.9': not a directory

iarspider · 2021-10-20T10:48:18Z

If I run python3 setup.py egg_info (as suggested in one of the issues on setuptools github) before python3 setup.py install, the build goes through, but bin directory is not created (it is also not created with Spack build), so the patching part of the spec breaks. The file is now in site-packages/command dir:

$ find ${i} -name 'easy*'
.../lib/python3.9/site-packages/setuptools/command/__pycache__/easy_install.cpython-39.pyc
.../lib/python3.9/site-packages/setuptools/command/easy_install.py

But easy_install is a deprecated tool (replaced with pip), I guess I can just patch the files at their new location, and not create %{i}/bin/easy_install, right?

iarspider · 2021-10-20T10:53:43Z

diff --git a/py3-setuptools.spec b/py3-setuptools.spec
index 016a8a2..2fad229 100644
--- a/py3-setuptools.spec
+++ b/py3-setuptools.spec
@@ -11,12 +11,12 @@ Requires: python3
 %build
 python3 bootstrap.py
 python3 setup.py build
+python3 setup.py egg_info

 %install
 python3 setup.py install --single-version-externally-managed --record=/dev/null --skip-build --prefix=%{i}
 sed -i 's|#!.*python.*|#!/usr/bin/env python3|' \
- %{i}/bin/* \
  %{i}/${PYTHON3_LIB_SITE_PACKAGES}/setuptools/command/easy_install.py \
  %{i}/${PYTHON3_LIB_SITE_PACKAGES}/pkg_resources/_vendor/appdirs.py
-mv %{i}/bin/easy_install %{i}/bin/easy_install3

+mv %{i}/${PYTHON3_LIB_SITE_PACKAGES}/setuptools/command/easy_install.py %{i}/${PYTHON3_LIB_SITE_PACKAGES}/setuptools/command/easy_install3.py

smuzaffar · 2021-10-20T14:14:36Z

go ahead and apply the change and restart the tests

iarspider · 2021-10-20T14:15:38Z

@cmsbuild please test

cmsbuild · 2021-10-20T14:16:01Z

Pull request #7400 was updated.

cmsbuild · 2021-10-20T14:37:52Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/19769/summary.html
COMMIT: 8988a97
CMSSW: CMSSW_12_1_X_2021-10-19-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7400/19769/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/python3/3.9.6/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/py3-setuptools/58.2.0-67b20c002e1e875b7b73d537896e7bd0/lib/python3.9/site-packages/setuptools/command/install_scripts.py", line 18, in run
import setuptools.command.easy_install as ei
ModuleNotFoundError: No module named 'setuptools.command.easy_install'
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Fz5eh1 (%install)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Fz5eh1 (%install)

That one changes was not needed

cmsbuild · 2021-10-20T14:45:58Z

Pull request #7400 was updated.

iarspider · 2021-10-20T14:46:55Z

@cmsbuild please test

cmsbuild · 2021-10-29T03:58:45Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/20023/summary.html
COMMIT: 9dde93c
CMSSW: CMSSW_12_1_X_2021-10-27-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7400/20023/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/20023/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/20023/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS
---> test import-hyperas had ERRORS
---> test import-hyperopt had ERRORS
---> test import-nose had ERRORS
and more ...

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901406
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-10-29T07:19:27Z

please test with cms-sw/cmssw#35878

cmsbuild · 2021-10-29T18:33:38Z

-1

Failed Tests: UnitTests RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/20036/summary.html
COMMIT: 9dde93c
CMSSW: CMSSW_12_1_X_2021-10-28-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7400/20036/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

RelVals-INPUT

138.3138.3_RunMinimumBias2021Splash+RunMinimumBias2021Splash+RECODR3Splash+HARVESTDR3/step2_RunMinimumBias2021Splash+RunMinimumBias2021Splash+RECODR3Splash+HARVESTDR3.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901411
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

iarspider · 2021-10-29T18:34:52Z

@smuzaffar this RelVal fails in other PRs, so I think it's safe to merge this one.

iarspider · 2021-10-29T18:35:08Z

merge

smuzaffar · 2021-12-07T19:13:55Z

please test

cmsbuild · 2021-12-07T19:14:02Z

Pull request #7400 was updated.

cmsbuild · 2021-12-08T14:54:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-90434f/21066/summary.html
COMMIT: dc5d031
CMSSW: CMSSW_12_3_X_2021-12-07-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7400/21066/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 3250608
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3250574
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 41 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-03-18T05:59:09Z

closing as it has been integrated via #7696

Update setuptools to 58.2.0

f315ec3

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Oct 20, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 20, 2021

iarspider requested a review from smuzaffar October 20, 2021 12:36

Update recipe

8988a97

cmsbuild added tests-started and removed tests-rejected labels Oct 20, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 20, 2021

Update py3-setuptools.spec

c6f4b28

That one changes was not needed

cmsbuild added tests-pending and removed tests-rejected labels Oct 20, 2021

cmsbuild removed the tests-pending label Oct 20, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 28, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 29, 2021

smuzaffar changed the base branch from IB/CMSSW_12_1_X/master to IB/CMSSW_12_2_X/master October 29, 2021 06:22

cmsbuild added requires-external tests-started and removed tests-rejected labels Oct 29, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 29, 2021

smuzaffar changed the base branch from IB/CMSSW_12_2_X/master to IB/CMSSW_12_3_X/master December 6, 2021 21:31

Merge branch 'IB/CMSSW_12_3_X/master' into update-setuptools

dc5d031

cmsbuild added tests-started and removed tests-rejected requires-external labels Dec 7, 2021

cmsbuild added tests-approved and removed tests-started labels Dec 8, 2021

smuzaffar changed the base branch from IB/CMSSW_12_3_X/master to IB/CMSSW_12_4_X/master March 11, 2022 08:15

iarspider mentioned this pull request Mar 16, 2022

Update various low-level packages e.g. pip, setuptools etc. #7696

Merged

smuzaffar closed this Mar 18, 2022

smuzaffar deleted the update-setuptools branch March 18, 2022 05:59

Update setuptools to 58.2.0; drop nose, hyperas, hyperopt #7400

Update setuptools to 58.2.0; drop nose, hyperas, hyperopt #7400

Uh oh!

Conversation

iarspider commented Oct 20, 2021

Uh oh!

iarspider commented Oct 20, 2021

Uh oh!

cmsbuild commented Oct 20, 2021

Uh oh!

cmsbuild commented Oct 20, 2021

External Build

Uh oh!

iarspider commented Oct 20, 2021

Uh oh!

iarspider commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iarspider commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iarspider commented Oct 20, 2021

Uh oh!

smuzaffar commented Oct 20, 2021

Uh oh!

iarspider commented Oct 20, 2021

Uh oh!

cmsbuild commented Oct 20, 2021

Uh oh!

cmsbuild commented Oct 20, 2021

External Build

Uh oh!

cmsbuild commented Oct 20, 2021

Uh oh!

iarspider commented Oct 20, 2021

Uh oh!

cmsbuild commented Oct 29, 2021

Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented Oct 29, 2021

Uh oh!

cmsbuild commented Oct 29, 2021

Unit Tests

RelVals-INPUT

Comparison Summary

Uh oh!

iarspider commented Oct 29, 2021

Uh oh!

iarspider commented Oct 29, 2021

Uh oh!

smuzaffar commented Dec 7, 2021

Uh oh!

cmsbuild commented Dec 7, 2021

Uh oh!

cmsbuild commented Dec 8, 2021

Comparison Summary

Uh oh!

smuzaffar commented Mar 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iarspider commented Oct 20, 2021 •

edited

Loading

iarspider commented Oct 20, 2021 •

edited

Loading