Skip to content

First implementation for ExternalEncryptionConfiguration#15

Merged
cristekdatum merged 10 commits into
dev_phase2from
ExternalEncryptionConfiguration
Aug 8, 2025
Merged

First implementation for ExternalEncryptionConfiguration#15
cristekdatum merged 10 commits into
dev_phase2from
ExternalEncryptionConfiguration

Conversation

@cristekdatum

@cristekdatum cristekdatum commented Jul 31, 2025

Copy link
Copy Markdown
Collaborator

Introducing the new ExternalEncryptionConfiguration request in #24

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

Comment thread python/pyarrow/tests/parquet/test_external_encryption.py
@cristekdatum

Copy link
Copy Markdown
Collaborator Author

@sofia-tekdatum I found this code at the existing Encryptor
image

as you can notice here is where column_keys changes to string format before get into the C++ code. I want to check if you still wan to use this pattern for the app_context

Comment thread python/pyarrow/tests/parquet/test_external_encryption.py
Comment thread python/pyarrow/tests/parquet/test_encryption.py Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/includes/libparquet_encryption.pxd
Comment thread python/pyarrow/tests/parquet/test_external_encryption.py
@sofia-tekdatum

Copy link
Copy Markdown
Collaborator

@sofia-tekdatum I found this code at the existing Encryptor <img alt="image" width="1593" height="549"
as you can notice here is where column_keys changes to string format before get into the C++ code. I want to check if you still wan to use this pattern for the app_context

Yes, we want that string passed as is all the way to the ExternalEncryptorImpl

Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
internal_key_material=internal_key_material,
data_key_length_bits=data_key_length_bits)
#Holds a shared pointer to the underlying C++ external encryption configuration object.
self._external_config = shared_ptr[CExternalEncryptionConfiguration](

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then here, we shouldn't be doing a new call at all, but rather a self.external_config.reset

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is important to above errors during the destruction of the objects when the test ends. But I will see if the changes in the previous comment works there is a possibility to make this change

Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx
Comment thread python/pyarrow/_parquet_encryption.pyx
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated
@cristekdatum cristekdatum force-pushed the ExternalEncryptionConfiguration branch 2 times, most recently from 88ccd14 to a8c6fa5 Compare August 6, 2025 22:33
Comment thread python/pyarrow/_parquet_encryption.pyx Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add space after comma

Comment thread python/pyarrow/_parquet_encryption.pyx Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also print the value here as in the @Property exception above.

Comment thread python/pyarrow/_parquet_encryption.pyx

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define separate test methods for each test case you want to create.

i.e. one for EncryptionConfiguration only
Another for ExternalEncryptionConfiguration with correct params
Another for ExternalEncryptionConfiguration with a poorly formed json

Don't just test the "sunny" cases.

@cristekdatum cristekdatum force-pushed the ExternalEncryptionConfiguration branch from a8c6fa5 to 0e69beb Compare August 7, 2025 22:28

@sofia-tekdatum sofia-tekdatum left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@cristekdatum cristekdatum merged commit 6aeb6c1 into dev_phase2 Aug 8, 2025
7 of 24 checks passed
@cristekdatum cristekdatum deleted the ExternalEncryptionConfiguration branch August 12, 2025 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement changes to CryptoFactory > Create ExternalEncryptionProperties (getMethod + datatype) @Cython

2 participants