Skip to content

chore: Migrate docs from EDoc to Markdown #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Benjamin-Philip wants to merge 13 commits into
base: main
Choose a base branch
from
Open

chore: Migrate docs from EDoc to Markdown #104

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
638c6bf
Update Erlang/OTP, Rebar3 and ExDoc
Benjamin-Philip Jul 14, 2026
896a974
Migrate arrow_array
Benjamin-Philip Jul 14, 2026
1b907e5
Fix erlfmt errors
Benjamin-Philip Jul 14, 2026
b4d3301
Quote OTP and Rebar3 versions
Benjamin-Philip Jul 14, 2026
47c2d54
Simplify type signatures
Benjamin-Philip Jul 14, 2026
b273e8e
Migrate arrow_buffer
Benjamin-Philip Jul 15, 2026
20937ac
Migrate arrow_offsets
Benjamin-Philip Jul 19, 2026
a59967c
Migrate arrow_utils
Benjamin-Philip Jul 19, 2026
3160a51
Migrate arrow_bitmap
Benjamin-Philip Jul 19, 2026
4ab8f1b
Migrate arrow_type
Benjamin-Philip Jul 19, 2026
25174ea
Migrate arrow_ipc_*
Benjamin-Philip Jul 19, 2026
36ad36d
Simplify arrow_ipc_* type signatures
Benjamin-Philip Jul 19, 2026
ec66c41
Migrate arrow array modules
Benjamin-Philip Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
otp-version: 25.1.0
rebar3-version: '3.18.0'
otp-version: 29.0
rebar3-version: 3.27.0
Comment thread
Benjamin-Philip marked this conversation as resolved.
Outdated

- name: Cache Hex packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/erlang.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
otp-version: 25.1.0
rebar3-version: '3.18.0'
otp-version: 29.0
rebar3-version: 3.27.0
Comment thread
Benjamin-Philip marked this conversation as resolved.
Outdated

- name: Cache Hex packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
Expand Down Expand Up @@ -81,8 +81,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
otp-version: 25.1.0
rebar3-version: '3.18.0'
otp-version: 29.0
rebar3-version: 3.27.0
Comment thread
Benjamin-Philip marked this conversation as resolved.
Outdated

- name: Cache Hex packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
Expand Down Expand Up @@ -122,8 +122,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
otp-version: 25.1.0
rebar3-version: '3.18.0'
otp-version: 29.0
rebar3-version: 3.27.0
Comment thread
Benjamin-Philip marked this conversation as resolved.
Outdated

- name: Cache Hex packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
Expand Down
2 changes: 1 addition & 1 deletion rebar.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
]}.

{plugins, [rebar3_cargo]}.
{project_plugins, [erlfmt, {rebar3_ex_doc, "0.2.18"}]}.
{project_plugins, [erlfmt, {rebar3_ex_doc, "0.2.30"}]}.

% Let erlfmt write files on format as opposed to not doing anything
{erlfmt, [write]}.
Expand Down
192 changes: 80 additions & 112 deletions src/arrow_array.erl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,87 +15,51 @@
% specific language governing permissions and limitations
% under the License.

%% @doc Provides a record, a behaviour, and functions to work with Apache Arrow
%% Arrays.
%%
%% This module as mentioned, provides conveniences for working with Apache Arrow
%% Arrays[1] of different Layouts[2]. Firstly, it provides a record to represent
%% all the Layout. Secondly, it provides a behaviour for different layouts to
%% adhere to common functionality. Lastly, it provides functions to work with
%% Arrays in a common manner.
%%
%% == The Structure of an Array ==
%%
%% `arrow''s implementation of an Array has the following fields in its
%% record definition:
%%
%% <ol>
%% <li>
%% `layout', of type {@link atom()}, which represents the Layout of the
%% Array.
%% </li>
%% <li>
%% `type', of type {@link arrow_type:arrow_type()}, which represents
%% the Logical Type[3] of the Array.
%% </li>
%% <li>`len', of type {@link pos_integer()}, which represents the Array's Length[4].</li>
%% <li>
%% `element_len', of type {@link pos_integer()} or `undefined', which
%% represents the Length of each element in an Array. Currently it only has
%% an integer value in the Fixed-Size List Layout[5]
%% </li>
%% <li>
%% `null_count', of type {@link non_neg_integer()}, which represents the
%% Array's Null Count[4], or the number of undefined values in the Array.
%% </li>
%% <li>
%% `validity_bitmap', which is a buffer (`arrow_buffer') or the atom
%% `undefined', which represents the Array's Validity Bitmap[6].
%% </li>
%% <li>
%% `offsets', which is a buffer (`arrow_buffer') which represents the
%% Offsets[7], or the start position of each slot in the data buffer of an
%% Array.
%% </li>
%% <li>
%% `data', which is a buffer (`arrow_buffer'), which represents the
%% Array's Value Buffer, whose layout differs based on the Array Layout.
%% </li>
%% </ol>
%%
%% Certain fields are not required for certain layouts. For example, for the
%% Fixed-Sized Primitive Layout, the offsets field is not required, in which
%% case it is assigned as `undefined'. Similarly, the `validity_bitmap' is not
%% required if there are no null values, in which case it is also assigned as
%% `undefined'.
%%
%% == The Behaviour of an Array ==
%%
%% As of right now, a layout needs to implement the `c:from_erlang/2' callback. This is
%% then used in the `from_erlang/3' function to create new arrays.
%%
%% == Functions for working with Arrays ==
%%
%% As of right now, only functions to access the various fields, to create
%% new arrays, and to serialize arrays to arrow exist.
%%
%% == References ==
%%
%% [1]: [https://arrow.apache.org/docs/format/Glossary.html#term-array]
%%
%% [2]: [https://arrow.apache.org/docs/format/Glossary.html#term-physical-layout]
%%
%% [3]: [https://arrow.apache.org/docs/format/Glossary.html#term-type]
%%
%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#null-count]
%%
%% [5]: [https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout]
%%
%% [6]: [https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps]
%%
%% [7]: [https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout]
%% @end
-module(arrow_array).
-moduledoc """
Provides a record, a behaviour, and functions to work with Apache Arrow Arrays.

This module as mentioned, provides conveniences for working with Apache Arrow
[Arrays](https://arrow.apache.org/docs/format/Glossary.html#term-array) of
different
[Layouts](https://arrow.apache.org/docs/format/Glossary.html#term-physical-layout).
Firstly, it provides a record to represent all the Layout. Secondly, it provides
a behaviour for different layouts to adhere to common functionality. Lastly, it
provides functions to work with Arrays in a common manner.

# The Structure of an Array

`arrow`'s implementation of an Array has the following fields in its record
definition, each corresponding to a field in the Arrow Columnar Format
specification:

| Field | Type | Representing |
|-------------------|-----------------------------|-------------------------------------------------------------------------------------------|
| `layout` | `t:atom/0` | Layout |
| `type` | `t:arrow_type:arrow_type/0` | [Logical Type](https://arrow.apache.org/docs/format/Glossary.html#term-type) |
| `len` | `t:pos_integer/0` | [Length](https://arrow.apache.org/docs/format/Columnar.html#array-lengths) |
| `element_len` | `t:pos_integer/0` | Length of each element |
| `null_count` | `t:non_neg_integer/0` | [Null Count](https://arrow.apache.org/docs/format/Columnar.html#null-count) |
| `validity_bitmap` | `arrow_buffer` | [Validity Bitmap](https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps) |
| `offsets` | `arrow_buffer` | [Offsets](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) |
| `data` | `arrow_buffer` | Value Buffer |
Comment thread
Benjamin-Philip marked this conversation as resolved.
Outdated


Certain fields are not required for certain layouts. For example, for the
Fixed-Sized Primitive Layout, the offsets field is not required, in which case
it is assigned as `undefined`. Similarly, the `validity_bitmap` is not required
if there are no null values, in which case it is also assigned as `undefined`.

# The Behaviour of an Array

As of right now, a layout needs to implement the `c:from_erlang/2` callback.
This is then used in the `from_erlang/3` function to create new arrays.

# Functions for working with Arrays

As of right now, only functions to access the various fields, to create
new arrays, and to serialize arrays to arrow exist.
""".
-export([
from_erlang/3,
layout/1,
Expand All @@ -113,23 +77,27 @@

-export_type([array/0, layout/0]).

-doc "Represents an Array.".
-type array() :: #array{}.
%% Represents an Array.

-doc "Represents the Layout of an Array.".
-type layout() :: fixed_primitive | variable_binary | fixed_list | variable_list.
%% Represents the Layout of an Array.

%%%%%%%%%%%%%%%%%%%%
%% Array Creation %%
%%%%%%%%%%%%%%%%%%%%

-doc """
Creates a new array of a certain layout, given its value and options from its
erlang representation.
""".
-callback from_erlang(Value :: [arrow_type:native_type()], Opts :: map()) ->
Array :: #array{}.
%% Creates a new array of a certain layout, given its value and options from its
%% erlang representation.

%% @doc A common way to create a new array, given its layout, value, and options.
%% from its erlang representation.
-doc """
A common way to create a new array, given its layout, value, and options.
from its erlang representation.
""".
-spec from_erlang(
Layout :: layout(),
Value :: [arrow_type:native_type()],
Expand All @@ -148,42 +116,42 @@ from_erlang(Layout, Value, Opts) ->
%% Array Data and Metadata Access %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% @doc Returns the layout of an array.
-doc "Returns the layout of an array.".
-spec layout(Array :: #array{}) -> Layout :: layout().
layout(Array) ->
Array#array.layout.

%% @doc Returns the type of an array.
-doc "Returns the type of an array.".
-spec type(Array :: #array{}) -> Type :: arrow_type:arrow_type().
type(Array) ->
Array#array.type.

%% @doc Returns the length of an array.
-doc "Returns the length of an array.".
-spec len(Array :: #array{}) -> Length :: pos_integer().
len(Array) ->
Array#array.len.

%% @doc Returns the length of an array.
-doc "Returns the length of an array.".
Comment thread
Benjamin-Philip marked this conversation as resolved.
-spec element_len(Array :: #array{}) -> Length :: pos_integer() | undefined.
element_len(Array) ->
Array#array.element_len.

%% @doc Returns the null count of an array.
-doc "Returns the null count of an array.".
-spec null_count(Array :: #array{}) -> NullCount :: non_neg_integer().
null_count(Array) ->
Array#array.null_count.

%% @doc Returns the validity bitmap of an array.
-doc "Returns the validity bitmap of an array.".
-spec validity_bitmap(Array :: #array{}) -> ValidityBitmap :: #buffer{} | undefined.
validity_bitmap(Array) ->
Array#array.validity_bitmap.

%% @doc Returns the offsets of an array.
-doc "Returns the offsets of an array.".
-spec offsets(Array :: #array{}) -> Offsets :: #buffer{} | undefined.
offsets(Array) ->
Array#array.offsets.

%% @doc Returns the data of an array.
-doc "Returns the data of an array.".
-spec data(Array :: #array{}) -> Data :: #buffer{} | #array{} | undefined.
data(Array) ->
Array#array.data.
Expand All @@ -192,24 +160,24 @@ data(Array) ->
%% Array Serialization %%
%%%%%%%%%%%%%%%%%%%%%%%%%

%% @doc Serializes an array into the Arrow binary form.
%%
%% Serializes the buffers of an Array and concatenates them in the following
%% order:
%%
%% <ol>
%% <li>`validity'</li>
%% <li>`offsets'</li>
%% <li>`data'</li>
%% </ol>
%%
%% In case an array doesn't have any of the following buffers, it is ommitted.
%% (e.g. validity in arrays with a null count of 0, offsets in fixed primitive
%% arrays). In the case of a nested array, `data' will be serialized form of
%% nested array.
%%
%% Do note that this is just binary form that includes the buffers in an Array,
%% and not IPC.
-doc """
Serializes an array into the Arrow binary form.

Serializes the buffers of an Array and concatenates them in the following
order:

1. `validity`
2. `offsets`
3. `data`

In case an array doesn't have any of the following buffers, it is ommitted.
(e.g. validity in arrays with a null count of 0, offsets in fixed primitive
arrays). In the case of a nested array, `data` will be serialized form of nested
array.

Do note that this is just binary form that includes the buffers in an Array,
and not IPC.
""".
Comment on lines +168 to +185
-spec to_arrow(Array :: #array{}) -> Arrow :: binary().
to_arrow(Array) ->
Validity = some(validity_bitmap(Array)),
Expand Down
Loading