Convert JSON to VariantArray without copying (8 - 32% faster) #7911

alamb · 2025-07-11T18:46:42Z

Which issue does this PR close?

part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736
Closes [Variant] Convert JSON to Variant with fewer copies #7964
Follow on to Add VariantArray and VariantArrayBuilder for constructing Arrow Arrays of Variants #7905

Rationale for this change

In a quest to have the fastest and most efficient Variant implementation I would like to avoid copies if at all possible
Right now, to make a VariantArray first requires completing an individual buffer and appending it
to the array.

Let's make that faster by having the VariantBuilder append directly into the buffer

What changes are included in this PR?

Add VariantBuilder::new_from_existing
Add a VariantArrayBuilder::variant_builder that reuses the buffers

Are these changes tested?

New unit tests
Yes by existing tests

Are there any user-facing changes?

Hopefully faster performance

scovich

This is remarkably simpler than I had imagined it would need to be. Handing off ownership back and forth was a very useful trick.

My only concern is whether we might ever need to support a builder that isn't backed by Vec? I'm guessing not, but wanted to double check.

parquet-variant-compute/src/variant_array_builder.rs

parquet-variant/src/builder.rs

parquet-variant-compute/src/variant_array_builder.rs

alamb · 2025-07-12T20:17:35Z

My only concern is whether we might ever need to support a builder that isn't backed by Vec? I'm guessing not, but wanted to double check.

I think eventually we might, but I think the only way to really do so is via some sort of trait and a templated builder. I think we can pretty far without doing so and Vec seems to be the best / fastes / etc thing for managing owned memory in Rust

And there are zero copy APIs to/from Vec for the underlying Arrow arrays which I think is a pretty nice property too

parquet-variant/src/builder.rs

… buffers (#7912) # Which issue does this PR close? - closes #7805 - part of #6736 - part of #7911 # Rationale for this change I would like to be able to write Variants directly into the target buffer when writing multiple variants However, the current VariantBuilder allocates a new bufffer for each variant # What changes are included in this PR? 1. Add `VariantBuilder::new_with_buffers` and docs and tests You can see how this API can be used to write directly into a buffer in VariantArrayBuilder in this PR: - #7911 # Are these changes tested? Yes new tests # Are there any user-facing changes? New API

alamb · 2025-07-13T12:40:14Z

Update here is I think I have incorporated @scovich's comments and I am quite pleased with how it is looking

I think this code needs a few more tests and a benchmark or two and we'll be good.

I'll try and work on those in the next few days

parquet-variant-compute/src/variant_array_builder.rs

scovich · 2025-07-14T14:16:33Z

parquet-variant-compute/src/variant_array_builder.rs

+impl<'a> Drop for VariantArrayVariantBuilder<'a> {
+    /// If the builder was not finished, roll back any changes made to the
+    /// underlying buffers (by truncating them)
+    fn drop(&mut self) {


I really like this approach. I was thinking over the weekend that we may want to rework the other builders to follow a similar approach:

They can truncate the metadata dictionary on rollback, which would eliminate the false allocations that survive a rollback today

We can allocate the value bytes directly in the base buffer (instead of using a separate Vec)

On rollback, just truncate (like here)

On success, use Vec::splice to insert value offset and field id arrays, which slides over all the other bytes

Once we're using splice, it opens the door to pre-allocate the space for the value offset and field arrays, in case the caller knows how many fields or array elements there are.

If the prediction was correct, splice just replaces the pre-allocated space.

If incorrect, the pre-allocation is wasted (but we're no worse off than before -- the bytes just inject in)

The main complication would be guessing how many bytes to encode each offset with.

They can truncate the metadata dictionary on rollback, which would eliminate the false allocations that survive a rollback today

That is an excellent point

We can allocate the value bytes directly in the base buffer (instead of using a separate Vec)

That sounds like a great way to avoid the extra allocation

[Variant] Avoid extra allocation in ObjectBuilder #7899

Once we're using splice, it opens the door to pre-allocate the space for the value offset and field arrays, in case the caller knows how many fields or array elements there are.

This is also a great idea 🤯

As a follow up, @klion26 has a PR up to implement this:

[Variant] Avoid extra allocation in object builder #7935

alamb · 2025-07-14T15:22:17Z

I added some benchmarks and my local results suggest that avoiding the allocations makes parsing small repeated json objects about 10% faster.

I think once we stop copying stuff around in the sub builders, the other bencmarks will be quite a bit faster too

Gnuplot not found, using plotters backend
small_repeated_json 8k string
                        time:   [5.3628 ms 5.3743 ms 5.3862 ms]
                        change: [−11.062% −10.867% −10.654%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

Benchmarking small_random_json 8k string: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.2s, or reduce sample count to 60.
small_random_json 8k string
                        time:   [71.132 ms 71.245 ms 71.364 ms]
                        change: [−1.7432% −1.4915% −1.2496%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  9 (9.00%) high mild
  1 (1.00%) high severe

alamb · 2025-07-16T20:21:39Z

parquet-variant/src/builder.rs

@@ -1047,16 +1047,16 @@ impl Drop for ObjectBuilder<'_> {
 ///
 /// Allows users to append values to a [`VariantBuilder`], [`ListBuilder`] or
 /// [`ObjectBuilder`]. using the same interface.
-pub trait VariantBuilderExt<'m, 'v> {
-    fn append_value(&mut self, value: impl Into<Variant<'m, 'v>>);
+pub trait VariantBuilderExt {


There is no reason for the lifetimes to be attached to the trait itself -- if it is that means that the lifetimes trickle into the values -- since this trait is for actually constructing variant values (and copying the underlying bytes) I moved the lifetimes to just the arguments that need it

alamb · 2025-07-16T20:22:39Z

parquet-variant-compute/src/variant_array_builder.rs

-        // TODO make this more efficient by avoiding the intermediate buffers
-        let mut variant_builder = VariantBuilder::new();
-        variant_builder.append_value(variant);
-        let (metadata, value) = variant_builder.finish();


The whole point of this PR is to avoid this copy here and instead write directly into the output

@Samyak2

# Which issue does this PR close? We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. - Part of #7911 - Part of #6736 - Follow on to #7905 # Rationale for this change I wrote benchmark some changes to the json decoder in #7911 but they are non trivial. To keep #7911 easier to review I have pulled the benchmarks out to their own PR # What changes are included in this PR? 1. Add new json benchmark 2. Include the `variant_get` benchmark added in #7919 by @Samyak2 # Are these changes tested? I tested them manually and clippy CI coverage ensures they compile # Are there any user-facing changes? No these are only benchmarks

alamb · 2025-07-18T15:31:27Z

parquet-variant-compute/src/from_json.rs

            json_to_variant(input_string_array.value(i), &mut vb)?;
-            let (metadata, value) = vb.finish();


The whole point if this PR is to avoid this copy / append

alamb · 2025-07-18T15:33:18Z

parquet-variant-compute/src/variant_array_builder.rs

@@ -55,9 +55,14 @@ use std::sync::Arc;
 /// };
 /// builder.append_variant_buffers(&metadata, &value);
 ///
+/// // Use `variant_builder` method to write values directly to the output array


this is the key new API -- a builder that can write directly to the correct output array location

alamb · 2025-07-18T15:34:53Z

parquet-variant-compute/src/variant_array_builder.rs

+impl<'a> Drop for VariantArrayVariantBuilder<'a> {
+    /// If the builder was not finished, roll back any changes made to the
+    /// underlying buffers (by truncating them)
+    fn drop(&mut self) {


As a follow up, @klion26 has a PR up to implement this:

[Variant] Avoid extra allocation in object builder #7935

alamb · 2025-07-18T15:44:34Z

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/append_variant_builder (fce19eb) to 99eb1bc diff
BENCH_NAME=variant_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench variant_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_append_variant_builder
Results will be posted here when complete

alamb · 2025-07-18T15:48:38Z

🤖: Benchmark completed

Details

group                                                                alamb_append_variant_builder           main
-----                                                                ----------------------------           ----
batch_json_string_to_variant json_list 8k string                     1.00     26.0±0.08ms        ? ?/sec    1.10     28.7±0.09ms        ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document)    1.00    364.4±6.10ms        ? ?/sec    1.08    393.1±6.36ms        ? ?/sec
batch_json_string_to_variant repeated_struct 8k string               1.00      7.5±0.02ms        ? ?/sec    1.14      8.6±0.03ms        ? ?/sec
variant_get_primitive                                                1.00   1040.3±4.08µs        ? ?/sec    1.32   1378.3±3.39µs        ? ?/sec

alamb · 2025-07-18T16:03:09Z

This one is now ready for review. I am quite pleased it already shows some benchmarks going 30% faster

Convert JSON to VariantArray without copying (8 - 32% faster) #7911 (comment)

Along with the following PR from @klion26

[Variant] Avoid extra allocation in object builder #7935

I think our JSON conversion is about as fast as it is going to get until we move away from serde_json for parsing

github-actions bot added the parquet Changes to the parquet crate label Jul 11, 2025

alamb mentioned this pull request Jul 11, 2025

Add VariantArray and VariantArrayBuilder for constructing Arrow Arrays of Variants #7905

Merged

alamb marked this pull request as draft July 11, 2025 18:51

alamb mentioned this pull request Jul 11, 2025

[Variant] Add VariantBuilder::new_with_buffers to write to existing buffers #7912

Merged

scovich reviewed Jul 11, 2025

View reviewed changes

alamb force-pushed the alamb/append_variant_builder branch from bb1502a to e07069d Compare July 12, 2025 20:31

scovich reviewed Jul 13, 2025

View reviewed changes

parquet-variant/src/builder.rs Outdated Show resolved Hide resolved

alamb force-pushed the alamb/append_variant_builder branch from e07069d to 8166cb6 Compare July 13, 2025 12:08

github-actions bot removed the parquet Changes to the parquet crate label Jul 13, 2025

alamb force-pushed the alamb/append_variant_builder branch from 8166cb6 to e10d41d Compare July 13, 2025 12:29

github-actions bot added the parquet Changes to the parquet crate label Jul 13, 2025

alamb self-assigned this Jul 14, 2025

scovich reviewed Jul 14, 2025

View reviewed changes

parquet-variant-compute/src/variant_array_builder.rs Outdated Show resolved Hide resolved

scovich reviewed Jul 14, 2025

View reviewed changes

alamb commented Jul 16, 2025

View reviewed changes

alamb mentioned this pull request Jul 16, 2025

[Variant] Add variant_kernels benchmark #7944

Merged

alamb force-pushed the alamb/append_variant_builder branch from 30ad86b to 3b6aef6 Compare July 16, 2025 20:54

alamb mentioned this pull request Jul 17, 2025

[Variant] API to construct Shredded Variant Arrays #7895

Open

alamb force-pushed the alamb/append_variant_builder branch from 3b6aef6 to 92e5c12 Compare July 18, 2025 15:29

alamb commented Jul 18, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

Fix Cargo for benchmark

fce19eb

alamb force-pushed the alamb/append_variant_builder branch from 92e5c12 to fce19eb Compare July 18, 2025 15:42

alamb commented Jul 18, 2025

View reviewed changes

alamb changed the title ~~Convert JSON to VariantArray without copying~~ Convert JSON to VariantArray without copying (8 - 32% faster) Jul 18, 2025

alamb marked this pull request as ready for review July 18, 2025 16:01

		json_to_variant(input_string_array.value(i), &mut vb)?;
		let (metadata, value) = vb.finish();

Convert JSON to VariantArray without copying (8 - 32% faster) #7911

Are you sure you want to change the base?

Convert JSON to VariantArray without copying (8 - 32% faster) #7911

Conversation

alamb commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Jul 12, 2025

Uh oh!

Uh oh!

alamb commented Jul 13, 2025

Uh oh!

Uh oh!

scovich Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 14, 2025

Uh oh!

alamb Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

alamb Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 18, 2025

Uh oh!

alamb commented Jul 18, 2025

Uh oh!

alamb commented Jul 18, 2025

Uh oh!

Uh oh!

alamb commented Jul 11, 2025 •

edited

Loading

scovich Jul 14, 2025 •

edited

Loading