Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for FLAC compression of 64bit integers. #9

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

tskisner
Copy link
Member

  • Add native support in the compiled extension for compressing 64bit integers with FLAC. The high and low 32bits are compressed in separate channels. On compression (for little-endian systems), the input 64bit integers are simply interpreted as interleaved stereo channels. On decompression, the separate channels in each frame are extracted to the high / low 32bit portions of the 64bit integers.

  • Add 32bit / 64bit wrappers to the bindings, and support for 64bit integers to the mid-level encode / decode functions.

  • Changes to the treatment of floating point data. float32 data is converted to int32 as before, but float64 data is now converted to int64 to gain additional precision.

  • Changes to int64 encoding- previously this was restricted to either 32bit range around an offset or forced conversion as floating point data. Now this data is supported natively. Remove the int32/int64 conversion routines which are no longer needed.

  • Change the HDF5 and Zarr on-disk format (bumping version to "1"). This change was necessary to support storing int64 data without an associated offset.

  • Modify and expand unit tests to cover new int64 cases.

- Add native support in the compiled extension for compressing 64bit
  integers with FLAC.  The high and low 32bits are compressed in
  separate channels.  On compression (for little-endian systems), the
  input 64bit integers are simply interpreted as interleaved stereo
  channels.  On decompression, the separate channels in each frame
  are extracted to the high / low 32bit portions of the 64bit integers.

- Add 32bit / 64bit wrappers to the bindings, and support for 64bit
  integers to the mid-level encode / decode functions.

- Changes to the treatment of floating point data.  float32 data is
  converted to int32 as before, but float64 data is now converted to
  int64 to gain additional precision.

- Changes to int64 encoding- previously this was restricted to either
  32bit range around an offset or forced conversion as floating point
  data.  Now this data is supported natively.  Remove the int32/int64
  conversion routines which are no longer needed.

- Change the HDF5 and Zarr on-disk format (bumping version to "1").
  This change was necessary to support storing int64 data without
  an associated offset.

- Modify and expand unit tests to cover new int64 cases.
@tskisner
Copy link
Member Author

This branch will be rebased after #8 is merged. I also want to expand the docs to describe the specific on-disk format. I also need to test the big-endian high / low 32bit handling in a multiarch qemu docker container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant