Skip to content

Change connection_config from a string:string map to a map from Parqu…#56

Merged
sofia-tekdatum merged 3 commits into
dev_phase2from
change_connection_config_map
Aug 14, 2025
Merged

Change connection_config from a string:string map to a map from Parqu…#56
sofia-tekdatum merged 3 commits into
dev_phase2from
change_connection_config_map

Conversation

@sofia-tekdatum

@sofia-tekdatum sofia-tekdatum commented Aug 12, 2025

Copy link
Copy Markdown
Collaborator

Change the connection config from a map to a map<ParquetCipher : map>

This will allow us the flexibility of specifying a connection config for more than one external encryptor at a time.

Also changed in Cython bindings.

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@avalerio-tkd

Copy link
Copy Markdown
Collaborator

@sofia-tekdatum I don't understand this PR. Let's chat offline quickly to get more context.

@avalerio-tkd avalerio-tkd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just the change in the connection_config to add the algorithm_enum as key. Discussed offline. Thanks.

@sofia-tekdatum

Copy link
Copy Markdown
Collaborator Author

@avalerio-tkd After discussin with @cristekdatum this afternoon, I added the changes to the Python bindings as well. Can you PTAL?

@avalerio-tkd avalerio-tkd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM+. I had a couple of questions, but not blocking to merge. Thanks.

Comment thread python/pyarrow/_parquet_encryption.pyx Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this clear() needed? Are we reusing an auxiliary data struct? Let's add a comment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, clearing from the values of the previous iteration. Comment added.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance of these look ups failing? Seems to follow the pattern of the previous version, but just checking.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this specific line no lookup is happening, but in general, no. If there is an incorrect value sent in as ParquetCipher the cipher_to_name and cipher_from_name methods will catch this and raise ValueErrors accordingly.

@sofia-tekdatum
sofia-tekdatum force-pushed the change_connection_config_map branch from 304c1a7 to 48f025b Compare August 14, 2025 16:21
@sofia-tekdatum
sofia-tekdatum merged commit 424d88c into dev_phase2 Aug 14, 2025
18 of 61 checks passed
@sofia-tekdatum
sofia-tekdatum deleted the change_connection_config_map branch August 14, 2025 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants