Add support for converting pyproject.toml-based Python3 packages. #1982

amdei · 2023-01-03T17:30:48Z

TL;DR:
It works! (With some limitations at the moment though, as this is work in progress currently)

python3 -m pip install -U importlib wheel setuptools packaging pkginfo

git clone ....
cd fpm
./bin/fpm --verbose -s python -t deb --python-install-lib=/usr/local/lib/python3.9/dist-packages/ --python-package-name-prefix python3 --python-bin=python3  typing_extensions

Limitations:

Does not support python2 in any way (obviously)

Temporary limitations (TODOs):

~~1. Do not support automatic dependencies extraction to the moment~~ - supports
~~2. Do not support license extraction to the moment~~ - supports
3. Checked only on Debian 11/12 amd64/arm64 only so far
~~4. Not sure about applicability to 'native' packages~~ - supports native package detection
~~5. A lot of @todo's in code~~ Only a few of them left...
~~6. Code quality is poor~~ made it better
~~7. Code-style is poor~~ tried to made it better

I had no previous experience with ruby, so any suggestions/comments/guidelines are welcome.

@ObjatieGroba - could you please check Python part of code, and give your suggestions on how to improve it?

Fixes #1780
Fixes #1873
Fixes #1860
Fixes magma/magma#13847

amdei · 2023-01-06T11:16:41Z

After some reflections I've had a second thought.

Standard importlib.metadata is better that backported importlib_metadata
importlib.metadata is still not good enough to obtain all required metadata of not-yet-installed python package
I don't want to build wheel out of downloaded python package
But seems there is no solution without building wheel out of python package prior to converting it to some other format
It is possible and it is easy to get ALL required metadata via wheel API from pkginfo package

Going to revise my solution accordingly.
Stay tuned.

amdei · 2023-02-12T06:48:16Z

It seems to be working a little bit better now.
At least in terms of metadata retrieval.

Things to do:

1. Apply proper arch attribute to generated deb-package. At the moment all toml-based packages has Architecture: all;
2. Instruct wheel to not download dependencies for build package wheel;
3. For some packages fpm now silently dies with zero exit code. Need to get it work again;
4. Deal with packages dependencies. Not easy at the first glance.
5. Speed-up execution on processing toml packages. Now it 3 times slower that setup.py-based;
6. Perform code de-duplication. As now it contains significant amount of redundant fragments.
7. Cleanup code.
8. File some issues to pip, wheel, importlib, etc.
9. Obey --[no-]python-obey-requirements-txt fpm's option

Any suggestions/comments/guidelines are still welcome.
As well as package samples, where things go wrong.

jordansissel · 2023-02-13T02:58:20Z

Thanks for your work on this! I’ll take a look as soon as I can :)

lib/fpm/package/pyfpm_wheel/get_metadata_wheel.py

amdei · 2023-02-14T04:38:51Z

Bottom line:

Please update python wheel to latest version before trying anything: python3 -m pip install -U wheel - there was a bug, affected fpm, fixed only very recently.
It is slow. Very slow. Significantly slower that setup.py-based packaging;
Wasn't able to made it with architecture - it looks like there is simply no way to get arch neither from *.whl file nor from sdist via any API. But it contains in *.whl. But I'm not brave enough to parse it in that way (yet).
There is no alternatives found for setup.py's --install-lib, --install-data, --install-scripts and build_scripts --executable. So all fpm's --python-install-* options will not work (and currently silently ignored).
Staging with pip is broken. For example see here: Unexpected --target behavior with/without --upgrade pypa/pip#8799 Or it is intended for something else.
Documentation for pip install is rather short: https://docs.python.org/3/installing/index.html#installing-index Significantly shorter than one would expect.

amdei · 2023-04-08T23:14:18Z

After a while, I hack the Wheel metadata to get information if package is pure or not.
https://peps.python.org/pep-0491/
As it seems to be impossible (at least yet) via pkginfo.Wheel API: https://bugs.launchpad.net/pkginfo/+bug/2015657

amdei · 2023-04-10T00:30:10Z

It reached "works for me" stage.
Let's collect feedback before starting to polish things..

lib/fpm/package/pyfpm/get_metadata.py

lib/fpm/package/python.rb

ObjatieGroba · 2023-04-12T20:24:45Z

lib/fpm/package/python.rb

+        "#{Shellwords.escape(toml_metadata_code)}"
+
+      # @todo FIXME!
+      #      if attributes[:python_obey_requirements_txt?]


Doesn't this break backward compatibility?

It can.
Do you know any cmd option for PEP517 builds to do the same thing as --load-requirements-txt does for setup.py?

ObjatieGroba · 2023-04-12T20:25:30Z

lib/fpm/package/python.rb

+    ::Dir.chdir(project_dir) do
+      flags = [ "--root", staging_path ]
+
+      # if !attributes[:python_install_lib].nil?


Lot's of commented code. Is it necessary?

Seems there is just no alternative for such option (--prefix) in setup.py for PEP517 builds...

JordanStopford · 2024-01-25T15:10:23Z

Amy update on this? A lot of packages no longer ship setup.py files so if we need to use fpm then we're stuck on old versions

cwegener · 2024-04-09T01:02:22Z

I don't understand what this does? It still requires a legacy setup.py file to be generated by the python project that is supposed to be converted. However, pretty much all PEP-517 build backends disable the support for generating setup.py legacy files by default, so they will not be able to be used by fpm. (some backends don't even offer an option at all to generate legacy setup.py files)

And if you have a source python project that has the option to generate a legeacy setup.py enabled, then such a project can already be used as a source in fpm without this change.

I am a bit confused what this change is supposed to do.

What is the intention of this change?

amdei · 2024-04-09T16:21:59Z

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented.
But if there is pyproject.toml file - it will be used instead.

cwegener · 2024-04-09T21:27:51Z

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented. But if there is pyproject.toml file - it will be used instead.

Thanks for the clarification. I'll need to read through the patch again then. 😁

amdei · 2024-04-17T00:43:36Z

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

cwegener · 2024-04-17T08:11:38Z

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

That's excellent. pip wheel will actually just invoke the Python build module which takes care of performing the right build process (legacy setup.py vs new PEP-517 build backends)

I will need to test out your PR and report the results.

Specifically, I will need to test with this PEP-517 project ... https://github.com/openai/openai-python

gmabey · 2024-05-08T14:08:22Z

Just an enthusiastic vote for more eyeballs on this merge request ... I would love ❤️ to see the project catch up to this shift in the python community.

cwegener · 2024-05-22T03:20:35Z

I will need to test out your PR and report the results.

Quick feedback from my initial testing of this patch:

Testing with

openai package from PyPI (which uses hatchling as the PEP 517 build backend)
Testing on Ubuntu 22.04 with built-in Python3.10

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel (and to a lesser degree pip download ... which is a separate story).

I think the main concern so far is the new dependency on the pkginfo Python library. E.g. on Ubuntu 22.04, the version of python3-pkginfo is a bit older and does not yet have support for version 2.3 Metadata file formats.

But a lot of wheels will now write 2.3 Metadata file formats.

Therefore, injecting or vendoring an up-to-date version of pkginfo is important.

amdei · 2024-07-14T20:20:22Z

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel

@cwegener could you please elaborate, what is your issue with openai package building?
Is it difficulty with determining build dependencies?
Or something else?

cwegener · 2024-07-14T21:31:28Z

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel

@cwegener could you please elaborate, what is your issue with openai package building? Is it difficulty with determining build dependencies? Or something else?

I think it's actually a different issue that is unrelated to the PEP-517 build support.

The issue is that all dependencies are being built from source due to a change introduced here: 77e0cf7

The --no-build :all: option forces all dependencies to be built from source, even though this is unnecessary.

The dependencies will be thrown away at the end. They don't end up in the released package that fpm creates. So there is really no benefit in building them from source.

Nevertheless, I still have not managed to build an openai package, but I think that might just be an issue on my side.

amdei · 2024-07-14T22:17:14Z

Nevertheless, I still have not managed to build an openai package, but I think that might just be an issue on my side.

Just checked - built seamlessly.

Could you please try to build it with more verbose output and share the result?
Something like this:
./bin/fpm --verbose --log debug --debug --debug-workspace -s python -t deb --python-bin=python3 --python-package-name-prefix python3 --python-install-lib=/usr/local/lib/python3.9/dist-packages/ openai

(you may want to adjust --python-install-lib option to reflect your python version)

cwegener · 2024-08-20T06:13:21Z

I had another look. And I think I figured out where the issue with this approach is.
pip download will perform unwanted build steps in order to generate metadata. Those unwanted build steps will fail quite often.

Example package: jiter

python3 -m pip download --no-clean --no-deps --no-binary :all: -d . jiter

This will try to immediately build a wheel from the jiter sdist, which will fail due to the build backend (maturin) not being present on my system.

This is a pip problem as per the following references.

References:
pypa/pip#7995
pypa/pip#1884

amdei · 2024-08-28T23:14:06Z

And I think I figured out where the issue with this approach is.

Could you please elaborate, do you able to offer different approach?
Yes, maturin, for example, requires Rust in order to compile its dependencies.

May be you know the way to download and use pre-build binary packages, for all possible supported OS/architectures?

gmabey · 2024-09-18T22:42:09Z

@amdei This branch worked for me on ubuntu 24.04!
I did have to install some packages that probably already should have been installed: python3-pkginfo, python3-packaging, python3-setuptools and python3-wheel. It looked to me like something was trying to automatically install some of them but that effort (whatever was doing it) produced errors. However, when I just installed the distro-provided versions, everything worked!

On ubuntu 22.04, ubuntu 20.04, rockylinux 8 and rockylinux 9, I had to both upgrade the distro-provided pip as well as upgrade the above modules before the creation of a .deb from python/pip would work.

So, I conclude that the core of your patch works great, but there may be some dependency auto-foo that is lacking?

Thank you for submitting this MR!

Fixes jordansissel#1780 Fixes jordansissel#1873 Fixes jordansissel#1860 Fixes magma/magma#13847

…and name'.

@ObjatieGroba

…l files (#1) Co-authored-by: Igr @ObjatieGroba

amdei · 2024-11-24T02:22:33Z

Rebased on top of main.
Squashed intermediate commits.

amdei · 2024-11-24T04:40:31Z

@cwegener Guess I finally get what you've meant here:

And I think I figured out where the issue with this approach is.

I asked:

May be you know the way to download and use pre-build binary packages, for all possible supported OS/architectures?

But the answer appears quite obvious: in wast majority of cases all these packages ARE on PyPi already.
And, instead of wasting CPU poser to compile source packages all over again, we may just download pre-built wheel, and do whatever we would like to do with it.
The only thing required to be taken into account - how to fallback to building wheel from sources when binary distributions is not available.

Will try to figure it out.

amdei · 2024-11-25T08:17:25Z

lib/fpm/package/pyfpm_wheel/get_metadata_wheel.py

@@ -0,0 +1,106 @@
+import os
+import sys
+import pkg_resources


The usage of pkg_resources is DEPRECATED:
https://setuptools.pypa.io/en/latest/pkg_resources.html

Users should refrain from new usage of pkg_resources and should work to port to importlib-based solutions.

amdei · 2024-12-01T23:51:51Z

The issue is that all dependencies are being built from source due to a change introduced here: 77e0cf7

@cwegener
Tried to overcome the issue with rebuilding the whole world all over again.

Do pip download with --prefer-binary flag;
If wheel was downloaded - do not attempt to rebuild package from sources;
If wheel was NOT downloaded - fallback to rebuilding package from sources (either via setup.py or pyproject.toml).

Can not properly check p. 3 though at the moment...

Could you please try it now?
Build time expected to be significantly lower.
Something like this:
./bin/fpm -s python -t deb --python-bin=python3 --python-package-name-prefix python3 numpy

CommandMC mentioned this pull request Jan 10, 2023

Update GH actions & specify Python version as dependency derrod/legendary#537

Merged

amdei force-pushed the python2deb-great-again branch from 419d841 to 9e16f11 Compare February 12, 2023 03:54

ObjatieGroba reviewed Feb 13, 2023

View reviewed changes

lib/fpm/package/pyfpm_wheel/get_metadata_wheel.py Show resolved Hide resolved

bugfood mentioned this pull request Mar 13, 2023

PEP518 - setup.py is deprecated #1987

Open

amdei marked this pull request as ready for review April 10, 2023 00:29

ObjatieGroba reviewed Apr 12, 2023

View reviewed changes

amdei force-pushed the python2deb-great-again branch from a9a1434 to 43fefae Compare September 22, 2023 22:49

amdei mentioned this pull request Nov 19, 2023

could not find setup.py in installable package #2029

Open

Stealthii mentioned this pull request Apr 24, 2024

PEP517 Support #2056

Open

Add support for converting pyproject.toml-based Python3 packages.

4f440c8

Fixes jordansissel#1780 Fixes jordansissel#1873 Fixes jordansissel#1860 Fixes magma/magma#13847

amdei and others added 3 commits November 24, 2024 05:14

Obtain package architecture via hackily hack.

f916189

Fix 'ERROR: The --python option must be placed before the pip subcomm…

cf5df8b

…and name'.

Handle local package creation via specifying path to setup.py or .tom…

c5af917

…l files (#1) Co-authored-by: Igr @ObjatieGroba

amdei force-pushed the python2deb-great-again branch from d722b81 to c5af917 Compare November 24, 2024 02:18

amdei requested a review from ObjatieGroba November 24, 2024 02:23

amdei added 2 commits November 24, 2024 07:33

Some code beautification's.

bf131b2

Preserve package author as 'Vendor'.

9d7991e

amdei commented Nov 25, 2024

View reviewed changes

amdei added 6 commits December 1, 2024 21:03

Fix crash on absence of package author.

1d1a1ab

Code deduplication.

d1d442b

Redo input() to prefer Wheel.

0f40b69

Proper code reformat.

3851171

Fix Unknown author apperance.

02f6e6c

Strip NYI, make messages better.

908224a

amdei added 2 commits December 2, 2024 02:54

Del extra loggind.

daf93a9

Some adjustments for pip wheel build

c86f43e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for converting pyproject.toml-based Python3 packages. #1982

Add support for converting pyproject.toml-based Python3 packages. #1982

amdei commented Jan 3, 2023 •

edited

Loading

amdei commented Jan 6, 2023 •

edited

Loading

amdei commented Feb 12, 2023 •

edited

Loading

jordansissel commented Feb 13, 2023

amdei commented Feb 14, 2023 •

edited

Loading

amdei commented Apr 8, 2023

amdei commented Apr 10, 2023

ObjatieGroba Apr 12, 2023

amdei Nov 24, 2024

ObjatieGroba Apr 12, 2023

amdei Nov 24, 2024

JordanStopford commented Jan 25, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 9, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 17, 2024 •

edited

Loading

cwegener commented Apr 17, 2024

gmabey commented May 8, 2024

cwegener commented May 22, 2024

amdei commented Jul 14, 2024

cwegener commented Jul 14, 2024

amdei commented Jul 14, 2024 •

edited

Loading

cwegener commented Aug 20, 2024

amdei commented Aug 28, 2024

gmabey commented Sep 18, 2024

amdei commented Nov 24, 2024

amdei commented Nov 24, 2024 •

edited

Loading

amdei Nov 25, 2024

amdei commented Dec 1, 2024 •

edited

Loading

Add support for converting pyproject.toml-based Python3 packages. #1982

Are you sure you want to change the base?

Add support for converting pyproject.toml-based Python3 packages. #1982

Conversation

amdei commented Jan 3, 2023 • edited Loading

amdei commented Jan 6, 2023 • edited Loading

amdei commented Feb 12, 2023 • edited Loading

jordansissel commented Feb 13, 2023

amdei commented Feb 14, 2023 • edited Loading

amdei commented Apr 8, 2023

amdei commented Apr 10, 2023

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

amdei Nov 24, 2024

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

amdei Nov 24, 2024

Choose a reason for hiding this comment

JordanStopford commented Jan 25, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 9, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 17, 2024 • edited Loading

cwegener commented Apr 17, 2024

gmabey commented May 8, 2024

cwegener commented May 22, 2024

amdei commented Jul 14, 2024

cwegener commented Jul 14, 2024

amdei commented Jul 14, 2024 • edited Loading

cwegener commented Aug 20, 2024

amdei commented Aug 28, 2024

gmabey commented Sep 18, 2024

amdei commented Nov 24, 2024

amdei commented Nov 24, 2024 • edited Loading

amdei Nov 25, 2024

Choose a reason for hiding this comment

amdei commented Dec 1, 2024 • edited Loading

amdei commented Jan 3, 2023 •

edited

Loading

amdei commented Jan 6, 2023 •

edited

Loading

amdei commented Feb 12, 2023 •

edited

Loading

amdei commented Feb 14, 2023 •

edited

Loading

amdei commented Apr 17, 2024 •

edited

Loading

amdei commented Jul 14, 2024 •

edited

Loading

amdei commented Nov 24, 2024 •

edited

Loading

amdei commented Dec 1, 2024 •

edited

Loading