[BUG] packages of the form package.something renamed to package_something in 75.8.1

[krpatter-intc]

setuptools version

setuptools==75.8.1

Python version

python3.10

OS

windows

Additional environment information

Description

We have packages using pyproject.toml of the form:

something.module that use implicit namespaces for something

The latest 75.8.1 seems to rename the something.module-1.2.3.dist-info to be instead: something_module-1.2.3.dist-info

rolling back to 75.8.0 this problem does not exist.

This breaks folks using something of the form pkg_resources.require('something.module') as it now only works if they have pkg_resources.require('something-module')

Expected behavior

packages should continue to have the something.module form

How to Reproduce

git clone https://github.com/krpatter-intc/setuptools-example.git
cd setuptools-example
python -m build

And you'll note the naming difference in the wheel (and the .dist-info file)

You can then run:

import pkg_resources
pkg_resources.require("something.module") # this now throws an exception

Output

I'm aware pkg_resources is on its way out, but this still feels bad that the package has a new name. I'm not sure what the new equivalent/replacement for pkg_resources.require is.

[krpatter-intc]

I realize from the release notes that the wheel renaming was intentional:

#4766

But the it's not clear that renaming the .dist-info and the actual package name was indeed intended.

[keenangraham]

This issue sounds relevant to what we're experiencing.

We started getting Docker image runtime failures today (very unusual), where as far as we can tell the only difference is setuptools version 75.8.0 versus 75.8.1.

...
File "/opt/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 410, in get_context
return loadcontext(object_type, name,
File "/opt/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 299, in loadcontext
return _loaders[scheme](
File "/opt/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 334, in _loadegg
return loader.get_context(object_type, name, global_conf)
File "/opt/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 624, in get_context
entry_point, protocol, ep_name = self.find_egg_entry_point(
File "/opt/venv/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 645, in find_egg_entry_point
pkg_resources.require(self.spec)
File "/opt/venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1061, in require
needed = self.resolve(parse_requirements(requirements))
File "/opt/venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 888, in resolve
dist = self._resolve_dist(
File "/opt/venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 929, in _resolve_dist
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zope.interface==5.3.0' distribution was not found and is required...

Also:
pkg_resources.DistributionNotFound: The 'repoze.debug==1.1' distribution was not found

So seems related to packages with . in name, and #3777 actually mentions zope.interface so guessing it's related.

Our solution was to bump PasteDeploy package, which apparently dropped setuptools dependency and started using importlib.metadata instead: https://docs.pylonsproject.org/projects/pastedeploy/en/latest/news.html

Not sure there is fix needed here, but commenting since this broke us in prod.

[di]

But the it's not clear that renaming the .dist-info and the actual package name was indeed intended.

It is, this brings setuptools in alignment with the specification for both wheel filenames and the .dist-info directory.

This breaks folks using something of the form pkg_resources.require('something.module') as it now only works if they have pkg_resources.require('something-module')

It seems like this is probably a bug in pkg_resources.require in that it should handle all forms of normalized names?

[krpatter-intc]

Well...the spec itself is not consistent in this regard:

This is equivalent to regular name normalization followed by replacing - with _. Tools consuming wheels must be prepared to accept . (FULL STOP)

It implies . should still be supported (even though admittedly the example regular expression provided implies the . should be changed).

One other thought...doesn't this now mean these two tools will alias? does something like pypi.org prevent both of these packages from existing?

• something.module
• something_module

If this is the new standard and it already works with all the rest of the eco-system, then we'll work to code up a new equivalent of the pkg_resources.require. There's not a direct replacement, but the tools exist in importlib to code it up again.

[hansingt]

We have the exact same issue for our tooling.

We do use namespaces extensively in our packages and the're now all broken. 🫤

[abravalheri]

I think the best way forward here is to create a patch for pkg_resources to restore compatibility. If anyone in the community is interested in giving it a try, please submit a PR.

I would also like to emphasize that is important for the community to start moving away from pkg_resources. I understand it is complicated, but it is being many years that it is advertised as deprecated, so it is reasonable to expect that the users have started to work at least a little bit on that.

[krpatter-intc]

I would also like to emphasize that is important for the community to start moving away from pkg_resources. I understand it is complicated, but it is being many years that it is advertised as deprecated, so it is reasonable to expect that the users have started to work at least a little bit on that.

No disgareements on the need to move away, but it's not scheduled for EOL till python3.13. I thought we had till then to remove it from our infrastructure. It effectively is 3.12 now (or freeze to old setuptools -- which we'll have to do for a bit)

[gilamsalem]

Same issue here. All our jobs are broken since yesterday because of this change.
Any workaround for that?
What is weird is that I tried to pin setuptools to older versions, and it doesn't help :-(

[fabiSwDevBeg]

Same issue here, all the docker images build today are not starting because of collective.volto.formsupport's wheel being created in collective_volto_formsupport instead. I just found out about this issue, pinning setuptools==75.8.0 doesen't fix it. (only collective.volto.formsupport is being written to the wrong path, idk why)
Building wheel for collective.volto.formsupport (pyproject.toml): started Building wheel for collective.volto.formsupport (pyproject.toml): finished with status 'done' Created wheel for collective.volto.formsupport: filename=collective_volto_formsupport-3.2.1-py3-none-any.whl size=150261 sha256=22163ba4c9108d639f2937790102bf17ee840c7a80d01af615555ea5a2d680a7 Building wheel for collective.tiles.collection (setup.py): started Building wheel for collective.tiles.collection (setup.py): finished with status 'done' Created wheel for collective.tiles.collection: filename=collective.tiles.collection-2.1.1-py3-none-any.whl size=24459 sha256=09d1d2c5b05666c97de079aaef41ce0278ebec756e7a6c0f49d6fa9ffbeb021f
I will try to pin the wheel version too

[krpatter-intc]

In case it helps: make sure you are pinning in the [build-system] otherwise the build-isolation will just grab the latest, regardless of what is in your current venv.

[di]

To be clear, there are two separate issues being discussed here:

1. pkg_resources.require and pkg_resources.get_distribution doesn't correctly normalize project names with periods;
2. wheel filenames now have underscores in project names instead of periods.

The first one is a bug and I've prepared a PR to fix it: #4855.

If you're having the second issue, setuptools is now producing the correct filename going forward and you should update your builds accordingly (or pin back setuptools<75.8.1 until you can).

[gilamsalem]

Hi @di
Thanks for handling that.
Assuming that my issue is indeed the second one, what can be the reason that pinning the setuptools version doesn't solve our issue?

...
Collecting setuptools==75.8.0
...
E   pkg_resources.DistributionNotFound: The 'booking.experiment.tracking' distribution was not found and is required by the application

Here is my build system config:

[build-system]
requires = [
    "setuptools <= 75.8.0",
]
build-backend = "setuptools.build_meta"

[di]

@gilamsalem The pkg_resources.DistributionNotFound exception is a result of the first issue with pkg_resources, not the second issue. Without seeing more logs or the command you're running, it's hard for me to say what you would need to change here.

[gilamsalem]

Thanks @di for the quick replies.

So from our side anything we can do to at least temporary unblock us?

Also the code that throws the exception is:
version = pkg_resources.get_distribution("booking.experiment.tracking").version

[di]

Seems like your issue is a mix of both. The issue with using get_distribution will also be resolved by the fix for the first issue.

Downgrading the version of setuptools used to build "booking.experiment.tracking" and rebuilding would resolve this for now (not this may or may not be the same of the version of setuptools that is providing pkg_resources.get_distribution to you)

[abravalheri]

The CI is working now to release a new patched version of setuptools that should solve the problem with pkg_resources.require(...).

For people having problems with the exact name of the distribution file has changed a character to _, I recommend trying something more flexible like glob patterns/regexes or wildcards in general. Since Setuptools is required to comply with the PyPA standards, the change in the file name itself is not considered a bug.

[abravalheri]

I would also like to emphasize that is important for the community to start moving away from pkg_resources. I understand it is complicated, but it is being many years that it is advertised as deprecated, so it is reasonable to expect that the users have started to work at least a little bit on that.

No disgareements on the need to move away, but it's not scheduled for EOL till python3.13. I thought we had till then to remove it from our infrastructure. It effectively is 3.12 now (or freeze to old setuptools -- which we'll have to do for a bit)

Hi @krpatter-intc could you please point out where you found the information about pkg_resources and Python 3.13. I was not aware that setuptools maintainers committed to that timeline.

[krpatter-intc]

Apologies, I stated 3.13, and I probably confused with disutils. So while pkg_resources is giving a deprecation warning, there's no timeline (that I see) for when it plans to stop being supported (just this link warning of the deprecation https://setuptools.pypa.io/en/latest/pkg_resources.html). I see some other un-offical stack overflows/github issues mention 3.12, but I see no official python release or setuptools version mentioned from the actual setuptools team.

And indeed it seems like it is still being supported via some PRs so I'm very grateful for the help here.

#4855 (comment)
#4856

[valipod]

If you're having the second issue, setuptools is now producing the correct filename going forward and you should update your builds accordingly (or pin back setuptools<75.8.1 until you can).

So this is a 75.8.1 type of change?

[gilamsalem]

My problem is solved now with version 75.8.2. Thanks!

[Zoltag]

If you're having the second issue, setuptools is now producing the correct filename going forward and you should update your builds accordingly (or pin back setuptools<75.8.1 until you can).

Does this mean that implicit namespaces in Python 3.9 is no longer supported by setuptools, post v75.8.0? Or am I missing something obvious?

[abravalheri]

It does not mean that.

The (machine-readable standardised) name of the generated wheel/sdist file and associated .dist-info directories is something separated from its contents. And you can keep having implicit namespace inside of your whell/sdist files even if the names don't contain dots.

[Zoltag]

The (machine-readable standardised) name of the generated wheel/sdist file and associated .dist-info directories is something separated from its contents. And you can keep having implicit namespace inside of your whell/sdist files even if the names don't contain dots.

I'm sorry, I'm feeling a bit dense this morning. When I build my namespace package using setuptools >75.8.0, it builds fine, the resulting wheel installs fine, but when I try to import the package in Python 3.9, I get either PackageNotFoundError or ModuleNotFoundError. This appears to be because the dist-info directory has replaced . with _. A simple rename of this directory to restore the .'s allows Python 3.9 to successfully import the module again (this is what I'm trying to demonstrate in my previous screenshot)

Is this something that is meant to be fixed in Python 3.9, or am I missing something here?

[abravalheri]

From the point of view of setuptools we have not identified anything else so far that needs fixing in Python 3.9.

For a simple import statement the name of the dist-info directory should not change things. If an error is happening it might be related to the particular Python code written in the file that is being imported? Any chance it is tightly coupled to the directory name?

The standard recently implemented in setuptools does require the names of dist-info directory to have ., replaced by _, so that is not considered a bug.

[Zoltag]

For a simple import statement the name of the dist-info directory should not change things. If an error is happening it might be related to the particular Python code written in the file that is being imported? Any chance it is tightly coupled to the directory name?

This is helpful, thank you !

When I build a simple use-case, it works:

When I add basic version information, it breaks:

[abravalheri]

I suspect this problem may have already been fixed in the latest versions of importlib.metadata... it might be worth checking that. Otherwise we can report the problem with the distribution name normalisation.

(The trouble with stdlib is that the version of the libraries are fixed on the moment of the release, but you may be lucky with importlib-metadata backport on PyPI)

[Zoltag]

Thank you - I've tested this and with version 8.6.1 installed, the problem persists (added as a dependency in the package wheel)

[di]

I confirmed that this is a bug that exists in importlib.metadata provided in Python 3.9 (all the way up to 3.9.16) that is fixed in Python 3.10 (starting at 3.10.0).

Looks like this was fixed in https://pypi.org/project/importlib-metadata/2.1.0/ and https://pypi.org/project/importlib-metadata/3.1.0/. The entry at https://github.com/python/importlib_metadata/blob/main/NEWS.rst#v210 says:

#253: When querying for package metadata, the lookup now honors package normalization rules.

@Zoltag Note that if you're unable to upgrade your Python version and instead installing importlib_metadata from PyPI, you need to make the following change to your file:

- from importlib.metadata import version
+ from importlib_metadata import version