GH-49321: [C++][Python] Add ASAN / UBSAN pixi builds for Arrow and PyArrow#49849
GH-49321: [C++][Python] Add ASAN / UBSAN pixi builds for Arrow and PyArrow#49849raulcd wants to merge 22 commits into
Conversation
|
|
|
@github-actions crossbow submit test-pixi-cpp |
|
Revision: a27d06c Submitted crossbow builds: ursacomputing/crossbow @ actions-fd33b1f179
|
|
@github-actions crossbow submit test-pixi-* |
|
Revision: 17bb233 Submitted crossbow builds: ursacomputing/crossbow @ actions-c9de1cf29d
|
|
@github-actions crossbow submit test-pixi-* |
|
Revision: 3f7725b Submitted crossbow builds: ursacomputing/crossbow @ actions-3daa50a309
|
| preview = ["pixi-build"] | ||
|
|
||
| [environments] | ||
| default = [] |
|
@github-actions crossbow submit test-pixi-* |
|
Revision: fafcb99 Submitted crossbow builds: ursacomputing/crossbow @ actions-2d14fbba39
|
|
@github-actions crossbow submit test-pixi-* |
|
Revision: 4757b86 Submitted crossbow builds: ursacomputing/crossbow @ actions-d318220858
|
|
@github-actions crossbow submit test-pixi-python-sanitizer |
|
Revision: ad4a829 Submitted crossbow builds: ursacomputing/crossbow @ actions-c442e4bcac
|
|
@github-actions crossbow submit test-pixi-python-sanitizer |
|
Revision: 9d4dd89 Submitted crossbow builds: ursacomputing/crossbow @ actions-1fd6ac9933
|
|
@github-actions crossbow submit test-pixi-python-sanitizer |
|
Revision: aa462b6 Submitted crossbow builds: ursacomputing/crossbow @ actions-c76613ce90
|
|
I think it works. I was expecting more failures :) |
|
@github-actions crossbow submit test-pixi-* |
|
Revision: 5917506 Submitted crossbow builds: ursacomputing/crossbow @ actions-c352190c2e
|
|
@github-actions crossbow submit test-pixi-* |
|
Revision: f142864 Submitted crossbow builds: ursacomputing/crossbow @ actions-9d972c1253
|
|
@github-actions crossbow submit test-pixi-* |
|
Revision: 1a9c8e7 Submitted crossbow builds: ursacomputing/crossbow @ actions-46cf962c89
|
…dd missing task for pixi-python-sanitizer
|
@github-actions crossbow submit test-pixi-* |
|
Revision: 5115f7d Submitted crossbow builds: ursacomputing/crossbow @ actions-b613038c28
|
There was a problem hiding this comment.
Pull request overview
Adds Pixi-based CI builds (including ASan/UBSan variants) for Arrow C++ and PyArrow, with supporting Docker/compose plumbing and a few PyArrow test-suite adjustments to avoid known sanitizer-unfriendly tests.
Changes:
- Introduces new Pixi manifests for building/testing Arrow C++ and PyArrow (default + sanitizer variants) and a dedicated Pixi Docker image.
- Wires new Pixi jobs into the CI task definitions and docker-compose hierarchy/volumes.
- Updates PyArrow build/test configuration for sanitizer runs (CMake sanitizer options + pytest marker to skip tests under ASan).
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| python/pyarrow/tests/test_memory.py | Marks known intentional OOB debug-memory-pool tests to be skipped under ASan. |
| python/pyarrow/tests/test_dlpack.py | Skips a bytes-allocation assertion test under ASan (plus comment). |
| python/pyarrow/conftest.py | Adds ASan detection and a noasan pytest group/marker defaulting to enabled when not under ASan. |
| python/CMakeLists.txt | Adds sanitizer CMake options and includes sanitizer flag configuration. |
| dev/tasks/tasks.yml | Adds CI docker-test tasks for the new Pixi images (incl. extended timeout for python sanitizer). |
| dev/release/utils-prepare.sh | Extends version-bump automation to update ci/pixi/*/*/pixi.toml package versions. |
| dev/release/post-10-bump-versions-test.rb | Extends release version bump test expectations to include Pixi manifests. |
| dev/release/01-prepare-test.rb | Extends pre-tag prepare test expectations to include Pixi manifests. |
| compose.yaml | Adds Pixi services, hierarchy entries, and build-cache volumes. |
| ci/pixi/default/python/pixi.toml | New Pixi manifest to build/test PyArrow against sibling Arrow C++ manifest. |
| ci/pixi/default/cpp/pixi.toml | New Pixi manifest to build/test Arrow C++ with pixi-build. |
| ci/pixi/asan/python/pixi.toml | New Pixi manifest to build/test PyArrow with ASan/UBSan + instrumented CPython/NumPy pins. |
| ci/pixi/asan/cpp/pixi.toml | New Pixi manifest to build/test Arrow C++ with ASan/UBSan. |
| ci/pixi/.gitignore | Ignores Pixi workdirs/lockfiles under ci/pixi. |
| ci/docker/pixi.dockerfile | New Dockerfile providing a Pixi-capable Ubuntu base for the new jobs. |
| option(ARROW_USE_ASAN "Enable Address Sanitizer checks" OFF) | ||
| option(ARROW_USE_UBSAN "Enable Undefined Behavior sanitizer checks" OFF) | ||
|
|
||
| include(SetupCxxFlags) | ||
| include(san-config) |
| # Asan tests are flacky on check_bytes_allocated assertion. | ||
| @pytest.mark.noasan |
|
FYI @raulcd I think prefix-dev/pixi#6519 contains the final fixes needed for Pixi to handle this use-case nicely. Now that we have https://conda.org/learn/ceps/cep-0045/, I think this can scale well to more and more variants as needed -- I have drafts at python/cpython#152385 numpy/numpy#31805 scipy/scipy#24066 which now make use of it. prefix-dev/pixi#2813 is the final thing that would improve scalability, allowing us to define all variants on a single |
|
@raulcd might you have time to return to this before the summit on Monday? |
@lucascolley I don't think I am going to have time for the next couple of days. Unfortunately I won't be able to make it to the summit either, I'll arrive to EuroPython on Tuesday. Hopefully we can catch up during the week, I'll try to follow the notes of the summit discussions on the HackMD. |
|
no worries, see you then! |
Rationale for this change
There has been discussions and appettite in the past to run Address and Undefined Behaviour Sanitizers on PyArrow.
In order to runn Sanitizers we have to instrument our build to build CPython and Numpy with Sanitizer flags too.
What changes are included in this PR?
This PR takes inspiration from other tools that have started intrumenting their sanitizer builds with Pixi like:
We add 4 new jobs and docker builds for:
For
pixi-python-sanitizer: Use CPython and Numpy pixi.toml(s) upstream and Arrow CPP with sanitizers pixi.toml to instrument everythin.Are these changes tested?
Yes, I've tested the changes and I created interim tests both on Python C++ side and Cython that where expected to run OOB and raise the sanitizer errors.
Are there any user-facing changes?
No