GH-50385: [Ruby] Add bitmap builder for red-arrow-format#50386
Merged
Conversation
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds bitmap builder utilities to red-arrow-format to construct Arrow-compatible bitmaps (with 64-byte padding) for validity and boolean buffers, and migrates the integration JSON reader to use the new builder logic.
Changes:
- Introduce
ArrowFormat::DenseBitmapBuilderandArrowFormat::SparseBitmapBuilder. - Add unit tests for both bitmap builders.
- Update integration JSON reader bitmap construction to use
DenseBitmapBuilder.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| ruby/red-arrow-format/lib/arrow-format/bitmap-builder.rb | Adds dense/sparse bitmap builder implementations with 64-byte padding behavior. |
| ruby/red-arrow-format/lib/arrow-format/integration/json-reader.rb | Switches JSON bitmap decoding to use the new dense bitmap builder. |
| ruby/red-arrow-format/test/test-dense-bitmap-builder.rb | Adds unit tests for dense bitmap builder output and padding. |
| ruby/red-arrow-format/test/test-sparse-bitmap-builder.rb | Adds unit tests for sparse bitmap builder output and padding. |
Comment on lines
+1
to
+2
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information |
Comment on lines
+38
to
+41
| padding_size = buffer_padding_size(@buffer) | ||
| @buffer.append_as_bytes(padding(padding_size)) if padding_size > 0 | ||
| @buffer.freeze | ||
| IO::Buffer.for(@buffer) |
Comment on lines
+45
to
+49
| def flush | ||
| @buffer.append_as_bytes([@byte].pack("C")) | ||
| @n_bits = 0 | ||
| @byte = 0 | ||
| end |
Comment on lines
+64
to
+85
| def finish(size) | ||
| builder = DenseBitmapBuilder.new | ||
| not_set_value = @default | ||
| set_value = !@default | ||
| previous_index = 0 | ||
| @indexes.sort.each do |index| | ||
| previous_index.upto(index - 1) do | ||
| builder.append(not_set_value) | ||
| end | ||
| builder.append(set_value) | ||
| previous_index = index + 1 | ||
| end | ||
| if previous_index.zero? | ||
| n_remains = size | ||
| else | ||
| n_remains = size - previous_index | ||
| end | ||
| n_remains.times do | ||
| builder.append(not_set_value) | ||
| end | ||
| builder.finish | ||
| end |
Comment on lines
+17
to
+18
|
|
||
| class TestDenseBitmapBuilder < Test::Unit::TestCase |
Comment on lines
+17
to
+18
|
|
||
| class TestSparseBitmapBuilder < Test::Unit::TestCase |
Comment on lines
207
to
211
| def read_bitmap(bitmap) | ||
| buffer = +"".b | ||
| bitmap.each_slice(8) do |bits| | ||
| byte = 0 | ||
| while bits.size < 8 | ||
| bits << 0 | ||
| end | ||
| bits.reverse_each do |bit| | ||
| byte = (byte << 1) + bit | ||
| end | ||
| buffer << [byte].pack("C") | ||
| builder = DenseBitmapBuilder.new | ||
| bitmap.each do |bit| | ||
| builder.append(bit == 1 ? true : false) | ||
| end |
Comment on lines
+75
to
+81
| @indexes.keys.sort.each do |index| | ||
| previous_index.upto(index - 1) do | ||
| builder.append(not_set_value) | ||
| end | ||
| builder.append(set_value) | ||
| previous_index = index + 1 | ||
| end |
Comment on lines
+61
to
+63
| def set(index) | ||
| @indexes[index] = true | ||
| end |
Comment on lines
+229
to
+240
| data.each_with_index do |value, i| | ||
| if value.nil? | ||
| validity_buffer_builder ||= SparseBitmapBuilder.new(true) | ||
| validity_buffer_builder.set(i) | ||
| values_buffer_builder.append(false) | ||
| elsif value | ||
| values_buffer_builder.append(true) | ||
| else | ||
| values_buffer_builder.append(false) | ||
| end | ||
| n += 1 | ||
| end |
Comment on lines
+37
to
+41
| def finish | ||
| flush if @n_bits > 0 | ||
| padding_size = buffer_padding_size(@buffer) | ||
| @buffer.append_as_bytes(padding(padding_size)) if padding_size > 0 | ||
| @buffer.freeze |
Comment on lines
+46
to
+50
| def flush | ||
| @buffer.append_as_bytes([@byte].pack("C")) | ||
| @n_bits = 0 | ||
| @byte = 0 | ||
| end |
Comment on lines
+64
to
+68
| def finish(size) | ||
| builder = DenseBitmapBuilder.new | ||
| set_value = true | ||
| unset_value = false | ||
| if @unset_indexes.empty? |
Member
Author
|
+1 |
|
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6570969. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
It can be used for validity bitmap and boolean array.
What changes are included in this PR?
ArrowFormat::DenseBitmapBuilderArrowFormat::SparseBitmapBuilderArrowFormat::BooleanArray.new(values)Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.