Modifying ExternalDBPAEncryptor/DecryptorAdapter to obtain agent instance from shared lib (DLL)

[argmarco-tkd]

Modifying ExternalDBPAEncryptor/DecryptorAdapter to obtain agent instance from shared lib (DLL)

This is the last piece which connects together the base work in Arrow (dev_phase2), and DLL-based agent loading.

Both ExternalDBPAEncryptorAdapter and ExternalDBPADecryptorAdapter have been modified to abandon the simple, pass-through encryption/decryption, and to make use of the DLL-loaded agent (which must be specific in the connection_config object passed in. The modifications include (a) looking for the DLL-file name in the properties, (b) instantiating a new agent from the DLL, and (c) forwarding the encrypt/decrypt operations to the DLL-based agent.

Testing

• Tests inside external_dbpa_encryption_tests.cc had to be modified to account for the need of DLL. For these tests, we're making use of libDBPATestAgent.so, which is built out of external/dbpa_test_agent.* (and used in a few different test scenarios).
• Newly modified tests pass.
• All existing Parquet (ctest -L parquet) tests pass.
• All other tests were unaffected.
• Manually verified that the end-to-end workflow in base_app.py completes succesfully.

Notes

• A similar change was performed in the dev-miniApp codebase: Wiring DBPAgent into ExternalEncryptorImpl/ExternalDecryptor impl #68, however bear in mind that the structure of encryptors/decryptors is/was quite different in the miniApp
• There will be another PR with some modifications to base_app.py (it needs to account for DLL-based agent loading)

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[argmarco-tkd]

I made this name change so source files are easier to navigate. Sharing the same name among encryptor and decryptor was confusing.

[argmarco-tkd]

this is just formatting, so that the encryptor and decryptor construction are similar in the code (no functional changes)

[argmarco-tkd]

We missed this in an earlier PR. Because the code was not being used at the time, there were no compilation errors.

[avalerio-tkd]

Added an Issue to track agent init errors since these are in a couple places. protegrity/DataBatchProtectionService#100

[argmarco-tkd]

thank you!

[avalerio-tkd]

This is where the "executor" would be hooked? (for my understanding)

[argmarco-tkd]

The calls to InvokeExternalEncrypt() and InvokeExternalDecrypt() will definitely be controlled by the "executor". But the overall behavioral reach of the "executor" is TBD. It will really depend on the decision we make on the behavior for init() and how those errors propagate up (e.g. we will likely want to protect against a remote DBPA's init() that takes 10 mins to verify server connections)