Skip to content

GH-33923: [Docs] Tensor canonical extension type specification #33925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AlenkaF merged 27 commits into from
Mar 15, 2023
Merged

GH-33923: [Docs] Tensor canonical extension type specification #33925

Changes from 11 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
af571cb
Add Fixed size tensor spec to canonical extensions list
AlenkaF Jan 30, 2023
8231150
Apply suggestions from code review
AlenkaF Jan 30, 2023
884d871
Remove implementation-specific metadata
AlenkaF Jan 30, 2023
83edd70
Change order with is_row_major
AlenkaF Jan 30, 2023
16ef6f1
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Jan 30, 2023
4f4ccce
Update metadata part
AlenkaF Jan 30, 2023
92fd7c6
Correct True to true in json
AlenkaF Jan 31, 2023
7873676
Change name from fixed_size_tensor to fixed_shape_tensor
AlenkaF Jan 31, 2023
a4219e3
Add description for ListType parameters
AlenkaF Jan 31, 2023
37e83db
Change the description for ListType parameters
AlenkaF Feb 1, 2023
5c92ff0
Remove is_row_major from the spec
AlenkaF Feb 2, 2023
cb5e2dd
Add dim_names and permutation to optional metadata
AlenkaF Feb 15, 2023
b562b8d
Add notes to the usage of dim_names and permutations metadata
AlenkaF Feb 15, 2023
c44101b
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Feb 15, 2023
24e7c28
Add dim_names and permutation to optional parameters
AlenkaF Feb 15, 2023
333ae67
Add explicit explanation of permutation indices
AlenkaF Feb 15, 2023
4086dfb
Change order with layout
AlenkaF Feb 15, 2023
bd2a515
Rephrase text about absent permutation param
AlenkaF Feb 15, 2023
bc07d7a
Apply suggestions from code review - Joris
AlenkaF Feb 15, 2023
68c6244
Remove redundant sentence in permutations explanation
AlenkaF Feb 16, 2023
3e2bb25
Update value_type description
AlenkaF Feb 22, 2023
a49f14f
Update parameters description
AlenkaF Feb 22, 2023
89d8042
Add a logical layout shape example in the desc of the serialization
AlenkaF Feb 22, 2023
4ff7a65
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Feb 22, 2023
1daf820
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Feb 28, 2023
70059d9
Add note about IPC tensor
AlenkaF Mar 9, 2023
6f44296
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Mar 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion docs/source/format/CanonicalExtensions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,30 @@ same rules as laid out above, and provide backwards compatibility guarantees.
Official List
=============

No canonical extension types have been standardized yet.
Fixed shape tensor
==================

* Extension name: `arrow.fixed_shape_tensor`.

* The storage type of the extension: ``FixedSizeList`` where:
Comment thread
jorisvandenbossche marked this conversation as resolved.

* **value_type** is the data type of individual tensors and
is an instance of ``pyarrow.DataType`` or ``pyarrow.Field``.
* **list_size** is the product of all the elements in tensor shape.

* Extension type parameters:

* **value_type** = Arrow DataType of the tensor elements
* **shape** = shape of the contained tensors as a tuple
Comment thread
AlenkaF marked this conversation as resolved.
Outdated

* Description of the serialization:

The metadata must be a valid JSON object including shape of
the contained tensors as an array with key "shape".

For example: `{ "shape": [2, 5]}`

.. note::

Elements in an fixed shape tensor extension array are stored
Comment thread
AlenkaF marked this conversation as resolved.
Outdated
in row-major/C-contiguous order.

@paleolimbot paleolimbot Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note that this will prevent storing R matrices as zero-copy. I seem to remember an earlier version of this had an option for storing column-major tensors zero-copy?

@paleolimbot paleolimbot Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also the case for Julia: https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-column-major

@jorisvandenbossche jorisvandenbossche Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have the same problem right now with FixedSizeList arrays? Or what is this converted to in R now? (can you convert that to a single matrix?)

Also, even if we allow a different order or custom strides for each individual tensor, the full array backing the FixedSizeListArray (the flat values child array) still needs the first dimension (with size == length of the logical array) with the biggest strides. So if R doesn't support that, I don't think zero-copy conversion is ever possible?

@paleolimbot paleolimbot Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd just have to make a chunked array where every chunk had a single element. We don't currently support converting a fixed-size list as a matrix (if we did it would be faster -- even if it wouldn't be zero copy).

@jorisvandenbossche jorisvandenbossche Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On further thinking, I am also not sure this is actually a problem. Languages like Julia and R can still create an array/matrix from the raw memory of the ListArray zero-copy. This array will only look transposed to their users if you need to prevent the copy. It's then up to the application/user to decide if you want to keep that or not (have a strided view in expected dimension order, or create a copy)?

@paleolimbot paleolimbot Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the argument for standardization, too, but I also want to make sure that the less represented languages (R, Julia, Matlab) aren't being marginalized here (I get that the heavy hitters here are TensorFlow and Torch, both of which are uncompromizingly row-major).

@paleolimbot paleolimbot Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This array will only look transposed to their users if you need to prevent the copy.

Users have always been able to transport tensors if they are willing to keep track of that kind of metadata themselves.

One potential use of this is communicating a single view of one very large tensor, for which case copies are expensive and in most cases unnecessary.

@pitrou pitrou Feb 2, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, if you have only one tensor, then both ways would work anyway, right?