WIP: Dummy PR to check maint-25.0.1 status - #50800
Draft
raulcd wants to merge 15 commits into
Draft
Conversation
…OS CRAN and wasm builds (#50297) ### Rationale for this change R build failures due CRAN toolchain ### What changes are included in this PR? Use version of functions available on CRAN toolchain ### Are these changes tested? By existing CI jobs, and additional unit tests. ### Are there any user-facing changes? No * GitHub Issue: #50295 Lead-authored-by: Nic Crane <thisisnic@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Nic Crane <thisisnic@gmail.com>
…#50328) ### Rationale for this change We need `libpng-dev` for the png R package. ### What changes are included in this PR? Install `libpng-dev`. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #50318 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
…erGetRunDecode (#50332) ### Rationale for this change Fix compiler error. ### What changes are included in this PR? Missing typename keyword. ### Are these changes tested? In CI. ### Are there any user-facing changes? No. * GitHub Issue: #50330 Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
…further steps and add tag to set_enabled (#50340) ### Rationale for this change Currently when check-labels is not a pull_request even the following steps are cancelled. Also we don't enable tags execution which is required for releases. ### What changes are included in this PR? Run check-labels for the specified events and enable jobs for tags. ### Are these changes tested? I've tested on my fork by pushing to main and creating tags, more details on the comment on the PR. ### Are there any user-facing changes? No * GitHub Issue: #50293 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
…se scripts (#50337) ### Rationale for this change The source verification and binary verification scripts comment to the Verification PR were failing due to the last updates to archery in order to use a single dependency. ### What changes are included in this PR? Add required base and head branches to be used when looking for a PR. Use create_issue_comment on PRs instead of create_comment which tries to create a review comment which requires more arguments. ### Are these changes tested? Yes, I've tested them locally in isolation in order to add comments on the verification PR. ### Are there any user-facing changes? No * GitHub Issue: #50336 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
…ment (#50384) ### Rationale for this change The Windows Wheels verification job is currently failing due to a change of API on the conda create call. ### What changes are included in this PR? Remove the deprecated `-f` flag. ### Are these changes tested? Yes, I have pushed the branch to the Apache Arrow repository instead of my fork to be able to manually trigger the workflow for the RC via workflow dispatch in order to validate the changes. ### Are there any user-facing changes? No * GitHub Issue: #50383 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change On macOS, two independently-linked mimalloc v3 instances built with default TLS settings can end up using the same hard-coded TLS slots and crash due to conflicting expectations. See upstream issue at microsoft/mimalloc#1327 This can manifest when PyArrow is loaded side-by-side with another Python extension module that bundles its own instance of mimalloc. ### What changes are included in this PR? 1. Bump mimalloc to 3.4.1, for the availability of the required CMake option. 2. Configure macOS to use C thread-local variables for thread-local storage, avoiding conflicting accesses to hard-coded TLS slots on macOS. 3. Also, unrelatedly, make sure the default malloc is not overriden by our mimalloc build on macOS. ### Are these changes tested? By existing CI jobs. ### Are there any user-facing changes? No, just a bugfix. * GitHub Issue: #50428 Lead-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Signed-off-by: Antoine Pitrou <antoine@python.org>
…nt Scalars in to_pylist (#50327) ### Rationale for this change `pa.Array.to_pylist()` converts one element at a time through `Array::GetScalar` plus a Python `Scalar` wrapper; for list types each row additionally allocates a Python `Array` wrapper for the row's values slice and a fresh generator before recursing per element. A `sample` profile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% in `GetScalar`, and only ~7% doing the useful work of creating the output objects — making `to_pylist` several times slower than converting via `to_pandas()` and back, and ~24x slower than `ndarray.tolist()` for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943). ### What changes are included in this PR? Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type `to_pylist` overrides: - `Array` gains `cdef object _getitem_py(self, int64_t i)`, returning `self[i]` as a Python object. The base implementation is `GetScalar` + `Scalar.as_py`, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...). - The baseline `Array.to_pylist` becomes a single loop over `_getitem_py`. `maps_as_pydicts != None` keeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics. - Specializations avoid all per-element Scalar and per-row Array-wrapper allocation: - integers and floats (a `type_id` switch on `NumericArray`; date/time/timestamp subclasses fall through to the exact base), - boolean, - string/binary and large variants (`GetValue` + `PyUnicode_DecodeUTF8` / `PyBytes_FromStringAndSize`, matching `str(buf, 'utf8')` / `to_pybytes()` exactly), - list/large_list/fixed_size_list (each row's list is built from the child's `_getitem_py` over the offset range; the wrapped child is cached on the parent array), - map (association list of key/value tuples, matching `MapScalar.as_py`), - struct (one dict per row; duplicate field names fall back to the Scalar path so they raise `ValueError` like `StructScalar.as_py`). Nested types compose without any per-row wrappers. `ChunkedArray.to_pylist`, `Table.to_pylist` and `ListScalar.as_py` delegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32. Benchmarks (macOS arm64, M4 Max): | benchmark | before | after | speedup | |---|---|---|---| | flat `int64` with nulls (4M) | 0.39 s | 0.028 s | 14x (~7 ns/element, on par with `ndarray.tolist`) | | flat `string` (4M) | 0.83 s | 0.06 s | 14x | | `list<string>` (2M rows) | 1.93 s | 0.46 s | 4.2x | | `list<list<int32>>` (1M rows) | 2.10 s | 0.40 s | 5.2x | | `struct<int64,string>` (1M rows) | 0.91 s | 0.07 s | 13x | | `map<string,int64>` (1M rows) | 2.77 s | 0.74 s | 3.8x | ### Are these changes tested? `test_to_pylist_bulk_paths` (added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against `[x.as_py() for x in arr]` with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-name `ValueError`), map (incl. strict-mode duplicate-key `KeyError`), dictionary/null fallbacks, sliced/chunked arrays, and both `maps_as_pydicts` modes — no differences. `pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py`: 1295 passed. ### Are there any user-facing changes? No behavior changes, only performance. * GitHub Issue: #50326 This pull request and its description were written by Isaac. Lead-authored-by: Liang-Chi Hsieh <liangchi.hsieh@databricks.com> Co-authored-by: Isaac Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change The SVE128 code path has conflict with the SVE256 that we do not yet manage properly. - There was first the ODR violation in GH-49921 - Now it seems that there may also be an issue with LTO Anyhow, after we fixed the inlining issue in Neon, the SVE128 had no clear advantages over Neon as expected, os this was due to be removed anyways. ### What changes are included in this PR? Remove SVE128 unpack ### Are these changes tested? In CI. ### Are there any user-facing changes? No * GitHub Issue: #50503 Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
…reResults (#50700) ### Rationale for this change Fixes a bug in the implementation of ODBC `GetMoreResults` in the FlightSQL ODBC driver. According to https://learn.microsoft.com/en-us/sql/odbc/reference/appendixes/statement-transitions?view=sql-server-ver17#sqlmoreresults, we should return `SQL_NO_DATA` for some states we previously were throwing another error in. This appears to be exposed by a behavior of only the Windows ODBC driver manager: `GetMoreResults` always gets called even for metadata queries. ### What changes are included in this PR? - Changed implementation and test: `GetMoreResults` now always returns `SQL_NO_DATA`. ### Are these changes tested? Yes, in CI. ### Are there any user-facing changes? No. * GitHub Issue: #50578 Authored-by: Bryce Mecum <petridish@gmail.com> Signed-off-by: Bryce Mecum <petridish@gmail.com>
|
Thanks for opening a pull request! This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format. If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or After updating the title, you can mark the pull request as ready for review. See also: |
github-actions
Bot
added
awaiting committer review
Member
Author
|
@github-actions crossbow submit --group packaging |
This comment was marked as outdated.
This comment was marked as outdated.
Member
Author
|
@github-actions crossbow submit --group verify-rc-source |
This comment was marked as outdated.
This comment was marked as outdated.
Member
Author
|
@github-actions crossbow submit verify-rc-sourceconda |
This comment was marked as outdated.
This comment was marked as outdated.
### Rationale for this change Follow-up to #49232 (comment) / #49590 ### What changes are included in this PR? Remove warnings from `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`. Warn **only** on writing with `version=1` / reading Feather V1 file. Update docs to V1-only and keep IPC migration guide. Additionally corrected notes like `deprecated as of 24.0.0` to `25.0.0` instead. `DeprecationWarning` in place of current `FutureWarning`. ### Are these changes tested? Yes, by CI. ### Are there any user-facing changes? Yes! Feather V2 APIs no longer emit deprecation warnings. Reading/writing the legacy Feather V1 format emits `DeprecationWarning` in place of `FutureWarning`. * GitHub Issue: #50808 Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Co-authored-by: tadeja <tadeja@users.noreply.github.com> Co-authored-by: Rok Mihevc <rok@mihevc.org> Signed-off-by: Rok Mihevc <rok@mihevc.org>
Member
Author
|
@github-actions crossbow submit --group packaging |
Member
Author
|
@github-actions crossbow submit --group verify-rc-source |
|
Revision: 849d463 Submitted crossbow builds: ursacomputing/crossbow @ actions-df71cdc2c0
|
|
Revision: 849d463 Submitted crossbow builds: ursacomputing/crossbow @ actions-1808d2e847
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caution
Do not merge this PR.
This PR is being used to test the status of the 25.0.1 release branch on CI and should not be merged.