Dev / 25.0 container image not building

[afgane]

Describe the bug
Now that 25.0 branch exists, I’m trying to build an image for 25.0 However, I see about 2 weeks ago images quit building: https://github.com/galaxyproject/galaxy/actions/workflows/build_container_image.yaml?page=6
I don’t see anything in the related commit that would cause this (00ac536), so it’s likely something independent. My local build is complaining about the same issue:

...
Running command git fetch -q https://github.com/nsoranzo/isa-rwval.git 3d989181058d2765a93cb0e7ca85d6955e0eb6ef\n  error: subprocess-exited-with-error\n  \n  × Getting requirements to build wheel did not run successfully.\n  │ exit code: 1\n  ╰─> [20 lines of output]\n      Traceback (most recent call last):\n        File \"/galaxy/server/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 389, in <module>\n          main()\n        File \"/galaxy/server/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 373, in main\n          json_out[\"return_val\"] = hook(**hook_input[\"kwargs\"])\n                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n        File \"/galaxy/server/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 143, in get_requires_for_build_wheel\n          return hook(config_settings)\n                 ^^^^^^^^^^^^^^^^^^^^^\n        File \"/tmp/pip-build-env-5gp3b45c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py\", line 331, in get_requires_for_build_wheel\n          return self._get_build_requires(config_settings, requirements=[])\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n        File \"/tmp/pip-build-env-5gp3b45c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py\", line 301, in _get_build_requires\n          self.run_setup()\n        File \"/tmp/pip-build-env-5gp3b45c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py\", line 512, in run_setup\n          super().run_setup(setup_script=setup_script)\n        File \"/tmp/pip-build-env-5gp3b45c/overlay/lib/python3.12/site-packages/setuptools/build_meta.py\", line 317, in run_setup\n          exec(code, locals())\n        File \"<string>\", line 11, in <module>\n      ModuleNotFoundError: No module named 'setuptools.package_index'

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 25.0
Commit: 3df9658

Browser and Operating System
Operating System: Linux, macOS

To Reproduce
Steps to reproduce the behavior:

1. Go to Container Build Action
2. or, build locally with docker build -f .k8s_ci.Dockerfile -t quay.io/galaxyproject/galaxy-min:25.0 . --platform linux/amd64

The build fails about about 2 minutes with the above exception.

[ksuderman]

This is likely due to setuptools.package_index being removed on May 3rd (See: https://setuptools.pypa.io/en/stable/history.html), which is the error I see when I try to build the image locally:

ModuleNotFoundError: No module named 'setuptools.package_index'

[afgane]

@nsoranzo Is this perhaps something that can be addressed in your isa-rwval repo?

[ksuderman]

This is absolutely caused by pypa/setuptools#4963

I've created nsoranzo/isa-rwval#1 that pins the setuptools version and that fixes this instance of the problem. But it just occurs again later in the build with firecloud, and possibly other packages after that.

[nsoranzo]

Happy to help! Do you have a way to reproduce the issue outside of the image building? I wasn't able to.

Also isa-rwval doesn't use the removed package_index from setuptools, so I don't understand how it comes into play.

[ksuderman]

Possibly one of the dependencies? I'll take a look at firecloud and see if there is anything in common.

[mvdbeek]

Might be a python 3.12 issue ?

[ksuderman]

I've tried 3.13 with the same results. I haven't tried older Python versions. However, the CI build broke the same day setuptools 80.3.0 was released and pinning setuptools < 80 solves the problem. It seems to be an issue with third party libraries (eg. isa-rwval and firecloud) and/or the third party libraries they depend on.

[afgane]

We've been using the base Python 3.12 image for nearly a year and, looking at the CI tests, this just started happening ~2 weeks ago.

[mvdbeek]

Any chance you could run a git bisect to see at what commit this broke @ksuderman ?

[ksuderman]

@mvdbeek no need for a bisect, it was 8940538 that was part of pypa/setuptools#4963