[WIP][POC] Pgaur on pair evaluation - #50791
Draft
prtkgaur wants to merge 21 commits into
Draft
Conversation
A C++ implementation of the OnPair short-string codec (arXiv:2508.02280): a trained dictionary of up to 16-byte tokens, greedy longest-prefix tokenization, and a branch-free gather-copy decode that keeps per-row random access. The dictionary budget is configurable from 9 to 16 bits; codes are bit-packed at the dictionary's true width.
Splits each value into a shared prefix and a suffix before the symbol table sees it, following the FSST+ thesis (Alexandre, CWI 2025). Used to measure whether prefix extraction adds anything on top of either codec.
Measures FSST, OnPair, the prefix-extraction variants and zstd/lz4 pages on one corpus set, on three axes: compression ratio, whole-column decode with per-row random access, and encode. Every codec is charged a bit-packed per-row length array so column reconstruction costs the same across all of them. bench_common.h holds the timing and bit-packing helpers shared with the cascade benchmark. The Rust helper generates the corpora: TPC-H string columns, the OnPair paper's real-world datasets, ClickBench columns, and synthetic identifier and JSON sets.
Measures the native Parquet pages as libparquet writes them, each on its own and followed by zstd(1) or lz4, against FSST and OnPair with and without a generic codec on top. Also measures the dictionary-then-OnPair cascade, which dictionary-encodes the column and OnPairs only the distinct values. Native pages are charged only what the writer emits, since they carry their own lengths; the candidate codecs are charged a separate length array. Auto-budget selection runs in per-corpus setup, outside the timed region, as every codec's parameter choice does.
The roundtrip checker now runs every dictionary budget from 9 to 16 rather than only 16, for both the plain and the dedup layout, and exits non-zero on any mismatch so it can gate a benchmark run. The packed decode loop is templated on the code width and the auto budget picks a width per column, so checking only 16 left the width the benchmarks report unverified. The width sweep answers whether a wider code decodes faster without the confound a budget sweep carries. It holds one trained dictionary and its code stream fixed and re-packs the same codes at every width up to 16, so tokens, token count and copy width are identical and only the unpacking differs.
|
Thanks for opening a pull request! This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format. If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or After updating the title, you can mark the pull request as ready for review. See also: |
github-actions
Bot
added
awaiting review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
Please remove this line and the above text before creating your pull request.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)