GH-49566: [Python] Skip header files when installing compiled Cython files and other Python release verification fixes by AlenkaF · Pull Request #49571 · apache/arrow

AlenkaF · 2026-03-20T10:20:16Z

Rationale for this change

Local import is broken when doing editable install with scikit-build-core, also nightly verification jobs are failing for the same reason.

What changes are included in this PR?

lib.h and lib_api.h are already installed separately so we skip them when installing Cython extensions into the output destination (side-packages/pyarrow).
Arrow library directory is added to the RPATH, in the case of not bundling ARROW_CPP, to fix linking issues on macos
test_pyarrow_include is updated due to compiled files now being moved to the site-packages
updates to the release verification script when building pyarrow
update of brew on the "Install System Dependencies" step in the rc verification job to fix an issue with protobuf mixed versions

Are these changes tested?

Yes, locally and via the extended verification builds.

Are there any user-facing changes?

No.

GitHub Issue: [Python] Cython API header files seem to break editable installs #49566

github-actions · 2026-03-20T10:21:29Z

⚠️ GitHub issue #49566 has been automatically assigned in GitHub to PR creator.

AlenkaF · 2026-03-20T10:22:52Z

@github-actions crossbow submit verify-rc-source-python-*

github-actions · 2026-03-20T10:25:23Z

Revision: 719eaf2

Submitted crossbow builds: ursacomputing/crossbow @ actions-3593c891fd

Task	Status
verify-rc-source-python-linux-almalinux-10-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-ubuntu-24.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

raulcd · 2026-03-20T10:47:12Z

macOS failure are unrelated, I am trying to find a fix, the issue tracking them is this one:

[CI][Python][C++] macos 14 fails with missing std::log2p1 #49569

AlenkaF · 2026-03-20T10:49:52Z

We have a different issue now:

_____________________________ test_pyarrow_include _____________________________

    def test_pyarrow_include():
        # We need to make sure that pyarrow/include is always
        # created. Either with PyArrow C++ header files or with
        # Arrow C++ and PyArrow C++ header files together
    
        source = os.path.dirname(os.path.abspath(__file__))
        pyarrow_dir = pjoin(source, '..')
        pyarrow_include = pjoin(pyarrow_dir, 'include')
        pyarrow_cpp_include = pjoin(pyarrow_include, 'arrow', 'python')
    
>       assert os.path.exists(pyarrow_include)
E       AssertionError: assert False
E        +  where False = <function exists at 0x7ff433ef9e80>('/arrow/python/pyarrow/tests/../include')
E        +    where <function exists at 0x7ff433ef9e80> = <module 'posixpath' (frozen)>.exists
E        +      where <module 'posixpath' (frozen)> = os.path

Investigating ...

AlenkaF · 2026-03-20T11:11:20Z

@raulcd I have updated the failing test as it was still checking the old location of the include folder which now moved to the site-packages. What do you think? The commit: 9ba4a76

AlenkaF · 2026-03-20T11:11:38Z

@github-actions crossbow submit verify-rc-source-python-*

github-actions · 2026-03-20T11:14:01Z

Revision: 9ba4a76

Submitted crossbow builds: ursacomputing/crossbow @ actions-7e817c381a

Task	Status
verify-rc-source-python-linux-almalinux-10-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-ubuntu-24.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

AlenkaF · 2026-03-20T11:45:54Z

Another issue I need to investigate:

arrow/dev/release/verify-release-candidate.sh: line 537: 94675 Bus error: 10              python -c "import pyarrow.flight"

AlenkaF · 2026-03-20T11:53:06Z

@github-actions crossbow submit verify-rc-source-python-macos-arm64

AlenkaF · 2026-03-20T11:53:25Z

Testing if it is a flaky macos issue ...

github-actions · 2026-03-20T11:55:20Z

Revision: 9ba4a76

Submitted crossbow builds: ursacomputing/crossbow @ actions-49b32e9b3a

Task	Status
verify-rc-source-python-macos-arm64

AlenkaF · 2026-03-20T15:33:59Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-20T15:36:26Z

Revision: 9ba4a76

Submitted crossbow builds: ursacomputing/crossbow @ actions-109f022a4d

Task	Status
verify-rc-source-python-macos-amd64

AlenkaF · 2026-03-23T07:29:42Z

verify-rc-source-python-macos-arm64 failure seems to be connected to #49569, but verify-rc-source-python-macos-amd64 is failing due to:

arrow/dev/release/verify-release-candidate.sh: line 537: 70649 Bus error: 10              python -c "import pyarrow.flight"

cc @raulcd for ideas? (can't test locally, my chip is arm)

…new one for the build

raulcd · 2026-03-24T10:18:38Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64 verify-rc-source-python-macos-arm64

raulcd · 2026-03-24T10:20:37Z

@AlenkaF I've rebased main (the macOS arm64 should be fixed there, just validating now) and have pushed a new commit because I think the verification wasn't correct. Is not using the built virtualenv/conda env. I am unsure what the macOS Intel failure is but the reason doesn't seem to be the same one as the one for the other failures so we might want to tackle that one on a separate issue.

github-actions · 2026-03-24T10:20:55Z

Revision: 3f0f110

Submitted crossbow builds: ursacomputing/crossbow @ actions-b16e795f8d

Task	Status
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64

AlenkaF · 2026-03-24T10:55:07Z

Python 3.14 verification on macos arm is now failing due to:

pip._vendor.pyproject_hooks._impl.BackendUnavailable: Cannot import '_build_backend'

update: both builds have the same, above, issue now 👍

github-actions · 2026-03-24T18:53:26Z

Revision: 5f8c18c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e5fbc461ef

Task	Status
verify-rc-source-python-macos-amd64

raulcd · 2026-03-24T19:44:35Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T19:46:02Z

502 Server Error
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23509011623

raulcd · 2026-03-24T20:35:55Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T20:38:14Z

Revision: a5ff56d

Submitted crossbow builds: ursacomputing/crossbow @ actions-0c439b7c1c

Task	Status
verify-rc-source-python-macos-amd64

raulcd · 2026-03-25T06:27:53Z

@github-actions crossbow submit verify-rc-source-python-*

github-actions · 2026-03-25T06:30:19Z

Revision: f9ca17b

Submitted crossbow builds: ursacomputing/crossbow @ actions-fd39c8b507

Task	Status
verify-rc-source-python-linux-almalinux-10-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-ubuntu-24.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

raulcd

@AlenkaF @kou can you take a look at the final changes?

raulcd · 2026-03-25T07:36:23Z


  # Build pyarrow
-  python -m pip install -e .
+  python -m pip install --no-build-isolation .


For the release verification of Python. I am unsure why we were testing with build isolation. We are setting up a virtualenv or a conda env and we should use that, that's why I am adding the --no-build-isolation. Also I am unsure why we were testing with editable builds. That doesn't make much sense for verification.

I agree with you, looks OK to me now.

AlenkaF · 2026-03-25T08:24:37Z

@raulcd the changes LGTM, thank you!
I will update the description to reflect all the extra changes in the PR. Please take a look when you can, so I don't miss anything important.

raulcd

From my side also 👍

kou

+1

kou · 2026-03-25T09:01:18Z


  # Build pyarrow
-  python -m pip install -e .
+  python -m pip install --no-build-isolation .


conbench-apache-arrow · 2026-03-25T16:18:38Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 2c2a61c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

…ython files and other Python release verification fixes (apache#49571) ### Rationale for this change Local import is broken when doing editable install with scikit-build-core, also nightly verification jobs are failing for the same reason. ### What changes are included in this PR? - `lib.h` and `lib_api.h` are already installed separately so we skip them when installing Cython extensions into the output destination (`side-packages/pyarrow`). - Arrow library directory is added to the `RPATH`, in the case of not bundling ARROW_CPP, to fix linking issues on macos - `test_pyarrow_include` is updated due to compiled files now being moved to the site-packages - updates to the release verification script when building pyarrow - update of `brew` on the "Install System Dependencies" step in the rc verification job to fix an issue with protobuf mixed versions ### Are these changes tested? Yes, locally and via the extended verification builds. ### Are there any user-facing changes? No. * GitHub Issue: apache#49566 Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: AlenkaF <frim.alenka@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

AlenkaF requested review from raulcd and rok as code owners March 20, 2026 10:20

github-actions Bot added Component: Python awaiting review Awaiting review labels Mar 20, 2026

AlenkaF and others added 3 commits March 24, 2026 11:17

Skip header files when installing compiled Cython files

41026d4

Fix old build include location test

efc1ae9

Use the virtualenv / conda env created instead of trying to create a …

3f0f110

…new one for the build

raulcd force-pushed the gh-49566-cython-headers branch from 9ba4a76 to 3f0f110 Compare March 24, 2026 10:18

raulcd requested review from assignUser, jonkeane and kou as code owners March 24, 2026 10:18

Fix protobug 33 and 34 on the same process

a5ff56d

kou reviewed Mar 24, 2026

View reviewed changes

Comment thread dev/release/verify-release-candidate.sh Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Mar 24, 2026

Remove debugging

f9ca17b

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 25, 2026

raulcd reviewed Mar 25, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 25, 2026

raulcd approved these changes Mar 25, 2026

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Mar 25, 2026

kou approved these changes Mar 25, 2026

View reviewed changes

raulcd changed the title ~~GH-49566: [Python] Skip header files when installing compiled Cython files~~ GH-49566: [Python] Skip header files when installing compiled Cython files and other Python release verification fixes Mar 25, 2026

raulcd merged commit 2c2a61c into apache:main Mar 25, 2026
25 of 26 checks passed

raulcd removed the awaiting merge Awaiting merge label Mar 25, 2026

raulcd mentioned this pull request Mar 25, 2026

[Python] Cython API header files seem to break editable installs #49566

Closed

AlenkaF deleted the gh-49566-cython-headers branch March 25, 2026 10:49

Uh oh!

Conversation

AlenkaF commented Mar 20, 2026 • edited by raulcd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

raulcd commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 23, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

AlenkaF commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

Uh oh!

raulcd commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

raulcd Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

AlenkaF Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

kou Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

AlenkaF commented Mar 25, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

AlenkaF commented Mar 20, 2026 •

edited by raulcd

Loading

raulcd commented Mar 20, 2026 •

edited

Loading

AlenkaF commented Mar 24, 2026 •

edited

Loading