[BUG] DD_API_KEY env var change not propagated to agent

[guidojw]

Agent Environment

7.58.0

Describe what happened:
I am using the datadog-agent Helm chart to deploy an agent to AKS.
We sync the keys from an Azure key vault to a kubernetes secret using akv2k8s and then use the (api|app)KeyExistingSecret values of the Helm chart to pass these to the containers.
When rotating the secrets in the key vault, this way the updates are propagated all the way to the env vars in the container.
The agent process however start erroring with API key invalid errors, I expect this is the case because the agent application only reads from the environment variable once and keeps it in memory, while for this to work with kubernetes secret changes it's best practice to check for changes to it.

Describe what you expected:
I expect env var changes to be supported in the agent without having to restart the process.

Steps to reproduce the issue:

1. Create kubernetes secret with app-key and api-key dataKeys set
2. Install agent using Helm chart with apiKeyExistingSecret and appKeyExistingSecret values pointed to this secret
3. Update the secret in-place
4. Invalidate old keys on Datadog portal

Additional environment details (Operating System, Cloud provider, etc):
AKS Linux node pool, Azure

[hush-hush]

Hi @guidojw,

Thank you for filling this issue. This is the expected behavior for now. The Agent configuration and the secrets it contains are loaded at startup and cached for the entire lifespan of the Agent.

We are currently exploring ways to refresh the API key at runtime without restart but I can't give you a precise ETA for now.