Skip to content

Support Parquet key management tools #7256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Tracked by #7278
adamreeve opened this issue Mar 9, 2025 · 2 comments · May be fixed by #7387
Open
Tracked by #7278

Support Parquet key management tools #7256

adamreeve opened this issue Mar 9, 2025 · 2 comments · May be fixed by #7387
Assignees
@adamreeve
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@adamreeve
Copy link
Contributor

Follow up task to #6637 and #7111, which add support for Parquet modular encryption.

Parquet-java, C++ Parquet and PyArrow provide a higher level API on top of Parquet modular encryption to simplify integration with a key management system (KMS) and support encryption best-practices. For compatibility with these implementations and to simplify the use of Parquet encryption, arrow-rs should also support this API.

There's a design document that provides more details on this at https://docs.google.com/document/d/1bEu903840yb95k9q2X-BlsYKuXoygE4VnMDl9xz_zhk/edit?tab=t.0#heading=h.aatf1oymdx11

I have made some progress towards implementing this in a branch (https://github.com/adamreeve/arrow-rs/tree/kmt).
@adamreeve adamreeve added the enhancement Any new improvement worthy of a entry in the changelog label Mar 9, 2025
This was referenced Mar 9, 2025
Closed
Merged
@adamreeve adamreeve self-assigned this Mar 9, 2025
This was referenced Mar 12, 2025
Open
Merged
@AudriusButkevicius
Copy link

I guess this is not closed as needs to be weaved in for encryption?

@adamreeve
Copy link
Contributor Author

adamreeve commented Apr 2, 2025 •
edited
Loading

Yes this is not done yet. I should have a PR ready for it soon.

It will be a separate module that builds on top of the recently added encryption feature.

@adamreeve adamreeve linked a pull request Apr 5, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adamreeve adamreeve

Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Key Management Tools API for Parquet encryption adamreeve/arrow-rs
2 participants
@adamreeve @AudriusButkevicius