GH-49644: [Python] Support converting list of multi-dimensional arrays to FixedShapeTensor

[aboderinsamuel]

Rationale for this change

Constructing a fixed-shape-tensor array from a list of individual ndarrays only
worked when each element was 1-D; ≥2-D elements failed with
ArrowInvalid: Can only convert 1-dimensional array values. The only workaround
was stacking the list into a single ndarray and using
FixedShapeTensorArray.from_numpy_ndarray.

What changes are included in this PR?

The C++ list converter PyListConverter::AppendNdarray now accepts
multi-dimensional ndarray elements for fixed-size lists (the storage of a
fixed-shape tensor) by flattening them in C order. The fixed-size-list builder
still validates that the flattened length matches the list width, so wrong sizes
error cleanly. Variable-sized lists remain restricted to 1-D values to avoid
ambiguity. As a side benefit, plain fixed_size_list also accepts
multi-dimensional ndarray elements now.

Are these changes tested?

Yes:

• test_tensor_array_from_list_of_ndarrays — construction from 2-D and 3-D
  ndarrays, null handling, storage parity with from_numpy_ndarray, and the
  size-mismatch error, across int8/int64/float32.
• test_fixed_size_list_from_multidim_ndarray — plain fixed_size_list from
  multi-dim arrays, plus a check that variable-sized lists still reject 2-D.

Are there any user-facing changes?

Yes — pa.array([multi-dim ndarrays], type=fixed_shape_tensor(...)) (and the
same for fixed_size_list) now works instead of raising. Existing 1-D behavior
and variable-sized-list behavior are unchanged.

Scoped to construction only; the reverse to_numpy shape-preservation also
raised in the issue is intentionally left as a separate follow-up.

• GitHub Issue: [Python] Support converting list of multi-dimensional arrays to FixedShapeTensor #49644

[github-actions]

⚠️ GitHub issue #49644 has been automatically assigned in GitHub to PR creator.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for constructing fixed-shape tensors and fixed-size lists from lists of multi-dimensional NumPy ndarrays by flattening values in C order (GH-49644).

Changes:

• Add tests covering tensor arrays built from lists of ndarrays (including nulls and shape mismatch).
• Add tests ensuring fixed-size lists accept multi-dimensional ndarray elements (and reject invalid cases).
• Update ndarray-to-list conversion to allow flattening for FIXED_SIZE_LIST while keeping variable-sized lists restricted to 1D.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
python/pyarrow/tests/test_extension_type.py	Adds coverage for building FixedShapeTensorArray from a list of ndarrays.
python/pyarrow/tests/test_array.py	Adds a regression test for fixed-size list conversion from multi-dimensional ndarrays.
python/pyarrow/src/arrow/python/python_to_arrow.cc	Implements multi-dimensional ndarray flattening for fixed-size lists during conversion.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[AlenkaF]

Thank you @aboderinsamuel!
Can you have a look at the linter error here: https://github.com/apache/arrow/actions/runs/27719407632/job/82001639136?pr=50203#step:5:98

Also, am curious how you decided to use PyArray_Ravel? Are there any other alternatives?

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[aboderinsamuel]

Hi @AlenkaF The macOS-intel failure is an intermittent dependency-install flake. Homebrew lock on aws-sdk-cpp, before the build/tests even run. It's hitting other PRs too (e.g. #50146), and #50195 is actively moving the macOS AWS-SDK dependency, so it looks like known infra rather than my change. Happy to keep re-running until it catches a clean runner. Everything else is green. @rok 🙂

[AlenkaF]

Repeating a question :)

[github-actions]

⚠️ GitHub issue #49644 has been automatically assigned in GitHub to PR creator.

[tadeja]

Hi all. I'd add two points for further review of current PR changes (thanks @aboderinsamuel !)
Example of wrong shape and example of permuted tensor currently produce results but might need to return errors instead:

import numpy as np
import pytest
import pyarrow as pa

np_dtype = np.dtype("int8")
tensor_type = pa.fixed_shape_tensor(pa.from_numpy_dtype(np_dtype), (2, 3))
pa.array([np.arange(6, dtype=np_dtype).reshape(3, 2)], type=tensor_type)
# As (3, 2) has the right number of elements (6), but the wrong shape for a (2, 3) tensor,
# shouldn't we get an error?

Currently returns:

<pyarrow.lib.FixedShapeTensorArray object at 0x12524ec20>
[
  [
    0,
    1,
    2,
    3,
    4,
    5
  ]
]

np_dtype2 = np.dtype("float32")
elements = [
  np.arange(6, dtype=np_dtype2).reshape(2, 3),
  np.arange(6, 12, dtype=np_dtype2).reshape(2, 3),
]
permuted_type = pa.fixed_shape_tensor(pa.from_numpy_dtype(np_dtype2), (2, 3),
                                    permutation=[1, 0])
pa.array(elements, type=permuted_type)
# For permuted tensor types built from multi-dim ndarrays, doesn't this store the wrong layout?

Currently returns:

<pyarrow.lib.FixedShapeTensorArray object at 0x104597ac0>
[
  [
    0,
    1,
    2,
    3,
    4,
    5
  ],
  [
    6,
    7,
    8,
    9,
    10,
    11
  ]
]

[aboderinsamuel]

@rok I'd keep PyArray_Ravel here. It's a zero-copy reshape of the already-C-contiguous array, and it gives the 1-D array that the dtype-mismatch fallback (value_converter_->Extend, just below) needs to iterate element-wise. Reading PyArray_DATA directly works for the typed fast path, but the fallback would still need a 1-D handle either way, so the ravel keeps a single shared path with no extra copy. Happy to switch if you'd prefer, but I'd lean to keeping it as-is.

[rok]

@aboderinsamuel - did you mean to push changes?

[aboderinsamuel]

yes, @rok. in a few minutes

[rok]

This looks good and I think we're ready to merge. Will just wait for @AlenkaF to chime in on the PyArray_Ravel comment.

[rok]

Maybe we could add an example of using the new constructor here? e.g.:

Suggested change

	Create an extension array from a list of multi-dimensional NumPy arrays.
	Each element is flattened in row-major (C) order, its shape must match
	the tensor shape.
	>>> import numpy as np
	>>> pa.array([np.array([[1, 2], [3, 4]], dtype=np.int32),
	... np.array([[10, 20], [30, 40]], dtype=np.int32)],
	... type=tensor_type)
	<pyarrow.lib.FixedShapeTensorArray object at ...>
	[
	[
	1,
	2,
	3,
	4
	],
	[
	10,
	20,
	30,
	40
	]
	]
	"""

[aboderinsamuel]

Done, I added the example to the FixedShapeTensorArray docstring with a runnable doctest. Verified it passes locally (with ELLIPSIS). Thanks!

[AlenkaF]

Yes, it seems the use of PyArray_Ravel is needed.

I think this PR is a great feature and agree with @rok that this should also be added to the docs. Thank you for the contribution @aboderinsamuel !

[aboderinsamuel]

Thank you @AlenkaF, @rok, and @tadeja, i'm grateful for the thorough review. @tadeja's catch on the wrong-shape and permutation cases especially made this a lot more correct. Added the docstring example as suggested. 🙏

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 85d0d9d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.