Support data re-encryption for Azure Key Vault key rotation

[taa-epam]

Name and Version

0.40.1

What is the problem this feature will solve?

Currently, ai-dial-core is limited to working with a specific key version in Azure Key Vault. There is no mechanism to update or re-encrypt existing data when the cryptographic key is changed.

This limitation prevents the usage of automatic key rotation policies in Azure. If we rotate the key to improve security, the application cannot seamlessly migrate the existing data to the new key version, leading to potential security compliance issues or data accessibility problems.

What is the feature you are proposing to solve the problem?

I propose implementing a mechanism to support Key Vault key rotation. The system needs to be able to:

1. Read with old keys: Decrypt data using previous key versions (as long as they are still valid in the Vault).
2. Re-encrypt with new keys: Provide a method (e.g., lazy migration on read or a specific background job) to re-encrypt the data using the latest key version.
3. Handle rotation: Allow the application to switch to the new key version for all new encryption operations automatically.

This will allow us to safely rotate keys periodically without losing access to historical data.

What alternatives have you considered?

No response