Destination S3 / S3 Data Lake: use micronaut injection instead of System.getenv for assume role

[edgao]

rebased on #51600.

recommend reviewing by commit:

1. add wiring for connectors to enable additional micronaut environments at runtime, and update tests appropriately
   1. Feedback appreciated - if there's some way to do this automatically, that would be much nicer (i.e. as soon as you add the toolkit-load-aws dependency, can we enable the aws environment?)
2. update s3 to use the aws environment; add application-aws to inject the assume role creds; wire everything together
3. do the same for s3-data-lake
4. update base tests + destination-s3 to support passing micronaut properties to the connector (probably start with DestinationProcess + AwsAssumeRoleCredentialsTestUtil, most of the rest of the diff is just passing properties through the callstack)
   1. adds S3V2CsvAssumeRole, which brings the assume role test into the new CDK (!!)
5. do the same for s3-data-lake (in particular, kill the preexisting envVars thing, which only supported System.getenv)

future PRs:

• delete all the env var stuff from integration tests
• switch the USE_FILE_TRANSFER flag to use micronaut properties at test time, instead of mocking an env var
• figure out dev-null's deployment: mode: thing in its application.yaml

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 22, 2025 0:16am

[frifriSF59]

This is great, Thanks!

Anyway we could write some sort of documentation about what the process should be if want to add new properties to be used in our code?
Do you think that has value?

[jdpgrailsdev]

One thing to potentially consider is using a masking pattern like Micronaut supports here (https://docs.micronaut.io/latest/guide/#environmentEndpoint) instead of relying a comment to ensure that secrets are printed. Instead of passing a list/map of String around, we could potentially wrap each env var in an object whose toString implements the masking logic you can see in something like EnvironmentEndpoint or EnvironmentFilterSpecification. Definitely something that could be done separate from this if we are really concerned about it.

[edgao]

Anyway we could write some sort of documentation about what the process should be if want to add new properties to be used in our code?

probably yeah. lemme post in the team slack and see if we have a typical process for this, it doesn't seem like a good fit for random code comments

masking

the problem is that at the end, we still have to render it down to List<String> (to pass into ProcessBuilder) :/ probably worth doing for the rest of our code though

(for now I'll just override the toString on AwsAssumeRoleCredentials to nothing 🤷 )