Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add encoding support. #13

Merged
merged 7 commits into from
Sep 11, 2024
Merged

Add encoding support. #13

merged 7 commits into from
Sep 11, 2024

Conversation

jheer
Copy link
Member

@jheer jheer commented Sep 9, 2024

  • Add tableToIPC to encode a table as IPC bytes. Supports both the stream and file IPC formats and all data types.
  • Add tableFromArrays and columnFromArray to encode JS data into a Flechette arrow table. Supports both explicit and inferred types, and fast paths for typed array data. Supports all data types except for "view" types.
  • Add data type methods (int8, float64, dateDay, etc.) to create type instances.
  • Add tableFromColumns method to create a table from existing Flechette columns.
  • Add useDecimalBigInt conversion flag to support lossless access to decimal data as unscaled big integers. While the default still (lossily) maps decimal types to number, using bigint allows direct manipulation (addition, subtraction, etc) of decimal values, so long as they all share the same scale (number of fractional digits).
  • Add column builder performance benchmarks, overall performance beats both Arrow-JS and JSON.stringify.
  • Export Table, Column and Batch classes.
  • Export batchType lookup method.
  • Update internal decimal type handling, use BigUint64Array and bigint operations to encode/decode data.
  • Update tests to cover new methods.
  • Refactor internal data type and batch lookup by type id.
  • Refactor dictionary batch initialization, allows batch completion prior to full dictionary completion.
  • Add library documentation pages.

Close #1

@jheer jheer merged commit 4cad0e5 into main Sep 11, 2024
2 checks passed
@jheer jheer deleted the jh/encode branch September 11, 2024 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for IPC encoding.
1 participant