-
Notifications
You must be signed in to change notification settings - Fork 36
Scikit-build-core build system rework #1988
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
I removed Another difference that I found is that the original |
I have noticed the following packaging behavior: When running |
Testing if Conda feedstock will work: |
Seems like conda packages were able to produce: |
I tested the shared object copying behavior similar to how I did for TileDB-VCF (TileDB-Inc/TileDB-VCF#717). I ran into 2 issues:
docker run --rm -it ubuntu:22.04
# Setup
apt-get update
apt-get install --yes git python-is-python3 python3 python3-pip python3-venv unzip wget
cd
mkdir downloads install-libtiledb
# Install nightly binary
the_date=2024-04-08
wget --quiet -P downloads/ \
https://github.com/jdblischak/centralized-tiledb-nightlies/releases/download/$the_date/libtiledb-$the_date.tar.gz
tar -C install-libtiledb -xzf downloads/libtiledb-$the_date.tar.gz
export LD_LIBRARY_PATH=$(pwd)/install-libtiledb/lib
# Clone TileDB-Py
git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py/
git checkout db/scikit-build-core
# Build and install TileDB-Py **with** bundled libtiledb
python -m pip install -v .
## -- Downloading TileDB default version ...
## -- [download 100% complete]
## -- Adding "libtiledb" into install group
##
## -- Installing: /tmp/tmppmr6xlbn/wheel/platlib/tiledb/libtiledb.so.2.24
##
## Successfully installed numpy-2.0.0 tiledb-0.30.3.dev109+gd18362b
ls /usr/local/lib/python3.10/dist-packages/tiledb/*.so*
## /usr/local/lib/python3.10/dist-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.so.2.24
## /usr/local/lib/python3.10/dist-packages/tiledb/main.cpython-310-x86_64-linux-gnu.so
ldd /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep libtiledb
## libtiledb.so.2.24 => /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.so.2.24 (0x00007f3ac2e8a000)
readelf -d /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep RUNPATH
## 0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN]
python -c "import tiledb; tiledb.libtiledb.version()"
## Traceback (most recent call last):
## File "<string>", line 1, in <module>
## File "/root/TileDB-Py/tiledb/__init__.py", line 22, in <module>
## from .array_schema import ArraySchema
## File "/root/TileDB-Py/tiledb/array_schema.py", line 10, in <module>
## from .attribute import Attr
## File "/root/TileDB-Py/tiledb/attribute.py", line 9, in <module>
## from .ctx import Ctx, CtxMixin
## File "/root/TileDB-Py/tiledb/ctx.py", line 16, in <module>
## class Config(lt.Config):
## AttributeError: module 'tiledb.cc' has no attribute 'Config'
python -m pip uninstall --yes tiledb
# Build and install TileDB-Py **without** bundled libtiledb
export TILEDB_PATH=${HOME}/install-libtiledb/
python -m pip install -v .
## -- Found external TileDB core library
## -- Setting RPATH for targets "main" and "libtiledb" to /root/install-libtiledb/lib
## -- Setting RPATH for target "cc" to /root/install-libtiledb/lib
##
## In file included from /tmp/tmp8dwduzb1/build/tiledb/libtiledb.cc:1248:
## /root/install-libtiledb/include/tiledb/tiledb.h:2973:34: note: declared here
## 2973 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_array_delete_fragments(
## | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## ninja: build stopped: subcommand failed.
##
## *** CMake build failed
## error: subprocess-exited-with-error
##
## × Building wheel for tiledb (pyproject.toml) did not run successfully.
## │ exit code: 1
## ╰─> See above for output.
##
## note: This error originates from a subprocess, and is likely not a problem with pip.
## full command: /usr/bin/python /usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpkbu_gaxq
## cwd: /root/TileDB-Py
## Building wheel for tiledb (pyproject.toml) ... error
## ERROR: Failed building wheel for tiledb
## Failed to build tiledb
## ERROR: Could not build wheels for tiledb, which is required to install pyproject.toml-based projects |
@ihnorton explained to me the source of the import error I reported above. It's because I ran it within the Git repo, and by default Python looks for The reprex below confirms that the import works fine outside of the Git repo: docker run --rm -it ubuntu:22.04
# Setup
apt-get update
apt-get install --yes git python-is-python3 python3 python3-pip python3-venv unzip wget
cd
# Clone TileDB-Py
git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py/
git checkout db/scikit-build-core
# Build and install TileDB-Py **with** bundled libtiledb
python -m pip install -v .
# Run import test outside of Git repo
cd
python -c "import tiledb; print(tiledb.libtiledb.version())"
## (2, 24, 0) |
Also, I switched to using the 2.24.2 release tarball, and I was able to compile TileDB-Py against the external libtiledb. The shared object isn't copied, and all the linking looks good. docker run --rm -it ubuntu:22.04
# Setup
apt-get update
apt-get install --yes git python-is-python3 python3 python3-pip python3-venv unzip wget
cd
mkdir downloads install-libtiledb
# Install 2.24.2 release binary
# https://github.com/TileDB-Inc/TileDB/releases/tag/2.24.2
wget --quiet -P downloads/ \
https://github.com/TileDB-Inc/TileDB/releases/download/2.24.2/tiledb-linux-x86_64-2.24.2-76cd03c.tar.gz
tar -C install-libtiledb -xzf downloads/tiledb-linux-x86_64-2.24.2-76cd03c.tar.gz
export LD_LIBRARY_PATH=$(pwd)/install-libtiledb/lib
# Clone TileDB-Py
git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py/
git checkout db/scikit-build-core
# Build and install TileDB-Py **without** bundled libtiledb
export TILEDB_PATH=${HOME}/install-libtiledb/
python -m pip install -vv .
## -- Found external TileDB core library
## -- Setting RPATH for targets "main" and "libtiledb" to /root/install-libtiledb/lib
## -- Setting RPATH for target "cc" to /root/install-libtiledb/lib
ls /usr/local/lib/python3.10/dist-packages/tiledb/*.so*
## /usr/local/lib/python3.10/dist-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/main.cpython-310-x86_64-linux-gnu.so
ldd /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep libtiledb
## libtiledb.so.2.24 => /root/install-libtiledb/lib/libtiledb.so.2.24 (0x00007f7dda6f6000)
readelf -d /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep RUNPATH
## 0x000000000000001d (RUNPATH) Library runpath: [/root/install-libtiledb/lib]
cd
python -c "import tiledb; print(tiledb.libtiledb.version())"
## (2, 24, 2) |
Lastly, I got this PR working in a branch of my centralized-tiledb-nightlies, so I'll be able to quickly migrate the nightly TileDB-Py build after this is merged. |
message(STATUS "Downloading TileDB default version ...") | ||
# Download latest release | ||
fetch_prebuilt_tiledb( | ||
VERSION 2.24.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow-up: we need to move this out at least to the top of the cmakelist if not the workflow file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I'm not sure if this is being validated correctly, because this is the hash for 2.24.2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nevermind, it is definitely being validated 😆 (I grabbed the wrong one)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about a variable named TILEDB_VERSION? This variable can set on the top of the file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to provide a way to override SETUPTOOLS_SCM_PRETEND_VERSION at the workflow level in order to avoid having to make temporary tags (to get properly named wheels from a QA branch).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I ran this branch here and noticed the manylinux wheels are ~20MB larger than the current ones on PyPI for the last release.
The wheel size is most probably due to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎉
This PR reworks build system for TileDB-Py to
scikit-build-core
+CMake
. To comply with our packaging requirements I made sure that theTILEDB_PATH
environment variable is properly handled. In all casesRPATH
is set to the location where TileDB core library is present.Additionally, Github workflows were added to create, test and upload PyPI wheels and sdist packages.
[sc-49832]