Scikit-build-core build system rework #1988

dudoslav · 2024-06-05T14:59:47Z

This PR reworks build system for TileDB-Py to scikit-build-core + CMake. To comply with our packaging requirements I made sure that the TILEDB_PATH environment variable is properly handled. In all cases RPATH is set to the location where TileDB core library is present.

Additionally, Github workflows were added to create, test and upload PyPI wheels and sdist packages.

[sc-49832]

dudoslav · 2024-07-04T10:17:04Z

Thanks

* we can exclude all of `doc` and `ci`

* seems like this is a local thing that should not have been included?

I removed doc and ci folders and made sure that I create wheel in a clean environment. This is the result right now:
https://www.diffchecker.com/9i4Ge9l2/

Another difference that I found is that the original setup.py approach puts the libtiledb.so in a different place. I did not notice any difference when running the wheel or testing it, but I want to mention it:

dudoslav · 2024-07-04T11:57:29Z

I have noticed the following packaging behavior:

When running pipx run build if TileDB core library is found in the system it gets picked up and used as the core TileDB library for the wheel. This however never happens in our CI/CD when deploying wheels, so it should be fine, but it can lead to incorrect versions of TileDB included inside wheel for packages produced manually, for example in a Conda environment with TileDB installed in it.

dudoslav · 2024-07-10T14:52:56Z

Testing if Conda feedstock will work:
conda-forge/tiledb-py-feedstock#224

dudoslav · 2024-07-11T10:59:53Z

Seems like conda packages were able to produce:
conda-forge/tiledb-py-feedstock#224

jdblischak · 2024-07-11T18:14:04Z

I tested the shared object copying behavior similar to how I did for TileDB-VCF (TileDB-Inc/TileDB-VCF#717).

I ran into 2 issues:

When bundling libtiledb.so into the installed Python package, I get the error AttributeError: module 'tiledb.cc' has no attribute 'Config' when running import tiledb
When using an external libtiledb.so, the compilation fails

docker run --rm -it ubuntu:22.04

# Setup
apt-get update
apt-get install --yes git python-is-python3 python3 python3-pip python3-venv unzip wget
cd
mkdir downloads install-libtiledb

# Install nightly binary
the_date=2024-04-08
wget --quiet -P downloads/ \
  https://github.com/jdblischak/centralized-tiledb-nightlies/releases/download/$the_date/libtiledb-$the_date.tar.gz
tar -C install-libtiledb -xzf downloads/libtiledb-$the_date.tar.gz
export LD_LIBRARY_PATH=$(pwd)/install-libtiledb/lib

# Clone TileDB-Py
git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py/
git checkout db/scikit-build-core

# Build and install TileDB-Py **with** bundled libtiledb
python -m pip install -v .
##   -- Downloading TileDB default version ...
##   -- [download 100% complete]
##   -- Adding "libtiledb" into install group
##
##   -- Installing: /tmp/tmppmr6xlbn/wheel/platlib/tiledb/libtiledb.so.2.24
##
## Successfully installed numpy-2.0.0 tiledb-0.30.3.dev109+gd18362b

ls /usr/local/lib/python3.10/dist-packages/tiledb/*.so*
## /usr/local/lib/python3.10/dist-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.so.2.24
## /usr/local/lib/python3.10/dist-packages/tiledb/main.cpython-310-x86_64-linux-gnu.so
ldd /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep libtiledb
## libtiledb.so.2.24 => /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.so.2.24 (0x00007f3ac2e8a000)
readelf -d /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep RUNPATH
## 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]
python -c "import tiledb; tiledb.libtiledb.version()"
## Traceback (most recent call last):
##   File "<string>", line 1, in <module>
##   File "/root/TileDB-Py/tiledb/__init__.py", line 22, in <module>
##     from .array_schema import ArraySchema
##   File "/root/TileDB-Py/tiledb/array_schema.py", line 10, in <module>
##     from .attribute import Attr
##   File "/root/TileDB-Py/tiledb/attribute.py", line 9, in <module>
##     from .ctx import Ctx, CtxMixin
##   File "/root/TileDB-Py/tiledb/ctx.py", line 16, in <module>
##     class Config(lt.Config):
## AttributeError: module 'tiledb.cc' has no attribute 'Config'

python -m pip uninstall --yes tiledb

# Build and install TileDB-Py **without** bundled libtiledb
export TILEDB_PATH=${HOME}/install-libtiledb/
python -m pip install -v .
## -- Found external TileDB core library
## -- Setting RPATH for targets "main" and "libtiledb" to /root/install-libtiledb/lib
## -- Setting RPATH for target "cc" to /root/install-libtiledb/lib
##
##   In file included from /tmp/tmp8dwduzb1/build/tiledb/libtiledb.cc:1248:
##   /root/install-libtiledb/include/tiledb/tiledb.h:2973:34: note: declared here
##    2973 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_array_delete_fragments(
##         |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
##   ninja: build stopped: subcommand failed.
##
##   *** CMake build failed
##   error: subprocess-exited-with-error
##
##   × Building wheel for tiledb (pyproject.toml) did not run successfully.
##   │ exit code: 1
##   ╰─> See above for output.
##
##   note: This error originates from a subprocess, and is likely not a problem with pip.
##   full command: /usr/bin/python /usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpkbu_gaxq
##   cwd: /root/TileDB-Py
##   Building wheel for tiledb (pyproject.toml) ... error
##   ERROR: Failed building wheel for tiledb
## Failed to build tiledb
## ERROR: Could not build wheels for tiledb, which is required to install pyproject.toml-based projects

jdblischak · 2024-07-12T18:52:20Z

@ihnorton explained to me the source of the import error I reported above. It's because I ran it within the Git repo, and by default Python looks for ./module.py and ./module/__init__.py. Thus it ran the local one instead of the one installed in site-packages. Thus the fix it to either run an editable install (-e) or move outside of the Git repo.

The reprex below confirms that the import works fine outside of the Git repo:

docker run --rm -it ubuntu:22.04

# Setup
apt-get update
apt-get install --yes git python-is-python3 python3 python3-pip python3-venv unzip wget
cd

# Clone TileDB-Py
git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py/
git checkout db/scikit-build-core

# Build and install TileDB-Py **with** bundled libtiledb
python -m pip install -v .

# Run import test outside of Git repo
cd
python -c "import tiledb; print(tiledb.libtiledb.version())"
## (2, 24, 0)

jdblischak · 2024-07-12T19:05:38Z

Also, I switched to using the 2.24.2 release tarball, and I was able to compile TileDB-Py against the external libtiledb. The shared object isn't copied, and all the linking looks good.

docker run --rm -it ubuntu:22.04

# Setup
apt-get update
apt-get install --yes git python-is-python3 python3 python3-pip python3-venv unzip wget
cd
mkdir downloads install-libtiledb

# Install 2.24.2 release binary
# https://github.com/TileDB-Inc/TileDB/releases/tag/2.24.2
wget --quiet -P downloads/ \
  https://github.com/TileDB-Inc/TileDB/releases/download/2.24.2/tiledb-linux-x86_64-2.24.2-76cd03c.tar.gz
tar -C install-libtiledb -xzf downloads/tiledb-linux-x86_64-2.24.2-76cd03c.tar.gz
export LD_LIBRARY_PATH=$(pwd)/install-libtiledb/lib

# Clone TileDB-Py
git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py/
git checkout db/scikit-build-core

# Build and install TileDB-Py **without** bundled libtiledb
export TILEDB_PATH=${HOME}/install-libtiledb/
python -m pip install -vv .
##  -- Found external TileDB core library
##  -- Setting RPATH for targets "main" and "libtiledb" to /root/install-libtiledb/lib
##  -- Setting RPATH for target "cc" to /root/install-libtiledb/lib

ls /usr/local/lib/python3.10/dist-packages/tiledb/*.so*
## /usr/local/lib/python3.10/dist-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so
## /usr/local/lib/python3.10/dist-packages/tiledb/main.cpython-310-x86_64-linux-gnu.so

ldd /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep libtiledb
## libtiledb.so.2.24 => /root/install-libtiledb/lib/libtiledb.so.2.24 (0x00007f7dda6f6000)

readelf -d /usr/local/lib/python3.10/dist-packages/tiledb/libtiledb.cpython-310-x86_64-linux-gnu.so | grep RUNPATH
##  0x000000000000001d (RUNPATH)            Library runpath: [/root/install-libtiledb/lib]

cd
python -c "import tiledb; print(tiledb.libtiledb.version())"
## (2, 24, 2)

jdblischak · 2024-07-12T19:07:32Z

Lastly, I got this PR working in a branch of my centralized-tiledb-nightlies, so I'll be able to quickly migrate the nightly TileDB-Py build after this is merged.

ihnorton · 2024-07-17T22:37:09Z

CMakeLists.txt

+        message(STATUS "Downloading TileDB default version ...")
+        # Download latest release
+        fetch_prebuilt_tiledb(
+                VERSION 2.24.0


Follow-up: we need to move this out at least to the top of the cmakelist if not the workflow file.

Also I'm not sure if this is being validated correctly, because this is the hash for 2.24.2

Nevermind, it is definitely being validated 😆 (I grabbed the wrong one)

What about a variable named TILEDB_VERSION? This variable can set on the top of the file

ihnorton

We need to provide a way to override SETUPTOOLS_SCM_PRETEND_VERSION at the workflow level in order to avoid having to make temporary tags (to get properly named wheels from a QA branch).

ihnorton

I ran this branch here and noticed the manylinux wheels are ~20MB larger than the current ones on PyPI for the last release.

dudoslav · 2024-07-22T12:49:21Z

The wheel size is most probably due to strip=False and build_type=RelWithDebInfo. Latest commit removed these options, lets see if the size will get smaller.

dudoslav · 2024-07-22T12:50:01Z

Latest wheels are about 15MB in size

ihnorton

🎉

TileDB-Inc/TileDB-Py#1988

dudoslav added 30 commits May 17, 2024 15:00

Initial rework into scikit-build-core

678f50d

Add auto downloading of TileDB

7c20a47

Add TILEDB_PATH env variable

6f8c182

Remove debug project name

b5d8bce

Add testing build-wheels ci

bc166f7

Add cxx_standard to 20

f3ec205

Fix incorrect flag set

51723fd

Enable CI/CD tests

cff8080

Fix pytest command

042f320

Run tests in tests folder

5fa6f72

Clean up code

76da30d

Remove __init__.py for tests

210a75e

Add test requirements

f73dcb8

Remove libtiledb logic

a456a6d

Disable python 39

cac2204

Improve pyproject.toml

4e145ba

Rework basic CI workflow

0b429c4

Install testing dependencies in CI

8ed0840

Remove unused CI/CD files

7156bdc

Merge remote-tracking branch 'origin/dev' into db/scikit-build-core

982d1e5

Fix format

dfb32b8

Use only one version on numpy

e160a89

Also enable python 3.9

56f7b76

Fix test issue with incorrect slashes in Win

7e7099f

Increment cibuildwheel version

7f1c3cd

Fix syntax error

9c13ff8

Format code

09ac7d6

Convert Path to str before search

2c1699e

Build arm macos wheels on macos-14

c9f5b73

Add basic CI/CD

b2f8f14

Remove doc and ci folders from wheel

de47a7e

Remove BUILD_STATE check

fb3d692

dudoslav added 2 commits July 9, 2024 11:55

Add extra logging

f621881

Merge remote-tracking branch 'origin/dev' into db/scikit-build-core

d18362b

dudoslav mentioned this pull request Jul 10, 2024

Test scikit-build-core tiledb-py conda-forge/tiledb-py-feedstock#224

Closed

5 tasks

jdblischak approved these changes Jul 12, 2024

View reviewed changes

ihnorton reviewed Jul 17, 2024

View reviewed changes

ihnorton requested changes Jul 18, 2024

View reviewed changes

Remove debug info and stripping

f705ef2

ihnorton approved these changes Jul 22, 2024

View reviewed changes

dudoslav merged commit 1ff19ef into dev Jul 22, 2024
33 checks passed

dudoslav deleted the db/scikit-build-core branch July 22, 2024 13:47

jdblischak mentioned this pull request Jul 22, 2024

Nightly feedstock build failed TileDB-Inc/conda-forge-nightly-controller#108

Closed

jdblischak added a commit to TileDB-Inc/centralized-tiledb-nightlies that referenced this pull request Jul 22, 2024

Return TileDB-Py to dev now that scikit-build-core PR was merged

f8771ad

TileDB-Inc/TileDB-Py#1988

This was referenced Jul 22, 2024

Migrate tiledb-py-feedstock to scikit-build-core TileDB-Inc/conda-forge-nightly-controller#114

Merged

Remove temporary code to migrate tiledb-py-feedstock to scikit-build-core TileDB-Inc/conda-forge-nightly-controller#116

Closed

kounelisagis mentioned this pull request Aug 6, 2024

Update requirements_dev.txt to improve contributor workflow #1766

Closed

jdblischak mentioned this pull request Aug 9, 2024

Migrate TileDB-Py to scikit-build-core and don't check for deprecations TileDB-Inc/TileDB-VCF#756

Merged

jdblischak mentioned this pull request Aug 19, 2024

The new scikit-build-core setup copies external shared objects into Python wheel TileDB-Inc/TileDB-VCF#717

Open

Scikit-build-core build system rework #1988

Scikit-build-core build system rework #1988

Uh oh!

Conversation

dudoslav commented Jun 5, 2024 • edited by kounelisagis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dudoslav commented Jul 4, 2024

Uh oh!

dudoslav commented Jul 4, 2024

Uh oh!

dudoslav commented Jul 10, 2024

Uh oh!

dudoslav commented Jul 11, 2024

Uh oh!

jdblischak commented Jul 11, 2024

Uh oh!

jdblischak commented Jul 12, 2024

Uh oh!

jdblischak commented Jul 12, 2024

Uh oh!

jdblischak commented Jul 12, 2024

Uh oh!

ihnorton Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

ihnorton Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ihnorton Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

dudoslav Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

ihnorton left a comment

Choose a reason for hiding this comment

Uh oh!

ihnorton left a comment

Choose a reason for hiding this comment

Uh oh!

dudoslav commented Jul 22, 2024

Uh oh!

dudoslav commented Jul 22, 2024

Uh oh!

ihnorton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dudoslav commented Jun 5, 2024 •

edited by kounelisagis

Loading

ihnorton Jul 17, 2024 •

edited

Loading