Skip to content

Add Key Management Tools API for Parquet encryption #7387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open

Add Key Management Tools API for Parquet encryption #7387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
f7d2e19
Add key material deserialization
adamreeve Feb 26, 2025
687b4e8
Add initial CryptoFactory implementation
adamreeve Feb 26, 2025
36cde66
Use boxed dyn KmsClient instead of type parameter in CryptoFactory
adamreeve Feb 26, 2025
839dfae
Add more realistic test KMS client
adamreeve Feb 26, 2025
5d156f3
Implement key material serialization
adamreeve Feb 26, 2025
981e6e0
Add basic unwrapping of double wrapped keys
adamreeve Feb 27, 2025
a3bcdb5
Add KMS config and client cache expiration
adamreeve Feb 27, 2025
6ec2c67
Cache decrypted key encryption keys
adamreeve Feb 27, 2025
fdcddcd
Prevent potential race condition reading key access token
adamreeve Feb 27, 2025
eb69f6e
Add a test of client caching and access token refresh
adamreeve Feb 28, 2025
9aa60f3
Implement building file encryption properties
adamreeve Feb 28, 2025
b494dfb
KmsConnectionConfig fix and tidy
adamreeve Mar 2, 2025
a779121
Only allow modifying key access token in kms config
adamreeve Mar 2, 2025
41b2c77
Refactor caching with expiration
adamreeve Mar 2, 2025
d3aa5c9
Fix getting key retriever for tests after rebase
adamreeve Mar 12, 2025
56abae6
Fix clippy errors
adamreeve Mar 12, 2025
9da61cb
Switch to using actual encryption properties
adamreeve Mar 12, 2025
d747b51
Fix double wrapping default
adamreeve Mar 12, 2025
8f9600d
Add cache cleaning
adamreeve Mar 12, 2025
c629c88
Fix ups after rebase
adamreeve Mar 21, 2025
b3f3b00
Add test of client expiration
adamreeve Mar 21, 2025
93cb522
Add test for key encryption key caching
adamreeve Mar 21, 2025
a505500
Add tests for interoperability with PyArrow encryption
adamreeve Mar 21, 2025
7d818bf
Set non-empty default for KMS instance ID and URL
adamreeve Mar 21, 2025
e37ee79
Add tests for write in Python, read in Rust
adamreeve Mar 21, 2025
2ca12fe
Test single wrapping in PyArrow integration tests
adamreeve Mar 23, 2025
e15c331
Tidy ups
adamreeve Mar 23, 2025
d6d69c0
Allow getting KMS instance ID and URL from footer key metadata
adamreeve Mar 24, 2025
1917b2c
Fixes and tidy ups after rebase
adamreeve Mar 27, 2025
8d469e4
Remove unused encryption algorithm config option
adamreeve Mar 28, 2025
29bc0cd
Add an example of encrypting a Parquet file using a KMS
adamreeve Mar 12, 2025
70a9b33
Integration test tidy up
adamreeve Apr 4, 2025
6dd2f71
Update docs
adamreeve Apr 4, 2025
71a4014
Make KmsClientFactory require Sync
adamreeve Apr 4, 2025
0fb5372
Remove parquet-pyarrow-integration-testing
adamreeve Apr 5, 2025
2325c52
Tidy ups
adamreeve Apr 5, 2025
19914bd
Renaming and comment updates to address PR feedback
adamreeve Apr 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion parquet/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ crc = ["dep:crc32fast"]
simdutf8 = ["dep:simdutf8"]
# Enable Parquet modular encryption support
encryption = ["dep:ring"]

# Enable high level encryption key management tools
key_management = ["encryption", "base64", "serde", "serde_json"]

[[example]]
name = "external_metadata"
Expand Down
Loading
Loading