Performance issue with K8S Secret Provider

[barkbay]

Hi 👋 ,

We have an Agent configuration which is referencing 2 Secrets using the K8S Secret Provider, for example:

          streams:
            - id: elasticsearch/metrics-elasticsearch.stack_monitoring.ccr
              data_stream:
                dataset: elasticsearch.stack_monitoring.ccr
                type: metrics
              metricsets:
                - ccr
              hosts:
                - 'https://${kubernetes.pod.ip}:9200'
              headers:
                Authorization: Bearer ${kubernetes_secrets.elastic-agent.somesecret1.value}
                AnotherHeader: SharedSecret ${kubernetes_secrets.elastic-agent.somesecret2.value}
              scope: node
              period: 10s

While investigating a global performance issue in our K8S cluster we observed a fair amount of requests to our K8S API server, with a constant rate of around 4 to 8 GET requests on these 2 Secrets per second. The audit trace suggests that that those requests are made by Agents managed by a DaemonSet on our clusters:

Audit Event Sample

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Metadata",
    "auditID": "4a1822f1-6509-45ad-a767-4174f0beded",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/elastic-agent/secrets/somesecret1",
    "verb": "get",
    "user": {
        "username": "system:serviceaccount:elastic-agent:elastic-agent",
        "uid": "7322c0d3-f498-4dae-a7e5-699d899fdfdf",
        "groups": [
            "system:serviceaccounts",
            "system:serviceaccounts:elastic-agent",
            "system:authenticated"
        ],
        "extra": {
            "authentication.kubernetes.io/pod-name": [
                "elastic-agent-b5hff"
            ],
            "authentication.kubernetes.io/pod-uid": [
                "64cb7be8-3d16-470f-afe6-c77fd092ffff"
            ]
        }
    },
    "sourceIPs": [
        "192.168.92.12"
    ],
    "userAgent": "Go-http-client/2.0",
    "objectRef": {
        "resource": "secrets",
        "namespace": "elastic-agent",
        "name": "somesecret1",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 200
    },
    "requestReceivedTimestamp": "2023-09-20T14:55:44.652043Z",
    "stageTimestamp": "2023-09-20T14:55:44.655909Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"elastic-agent\" of ClusterRole \"elastic-agent\" to ServiceAccount \"elastic-agent/elastic-agent\""
    }
}

I think these Secrets should be cached, as it is usually the case when using the K8S client from the controller runtime for example. It would definitely help in our case.

We tried to have a look at the code and we were wondering if the client calls are coming from here.

Thanks!

For confirmed bugs, please report:

• Version: docker.elastic.co/beats/elastic-agent:8.8.2
• Operating System: K8S 1.25 on EKS

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

I believe this is happening in this function which indeed has no cache

elastic-agent/internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets.go

Line 54 in 69cc860

func (p *contextProviderK8sSecrets) Fetch(key string) (string, bool) {

[cmacknz]

The fastest way to get this fixed will be to open a PR adding the cache yourself, the changes needed should be contained to a single source file.

[blakerouse]

Adding caching always has the issue of what invalidates the cache. Which I think needs to be discussed before a cache is added.

I think updating the provider to be more efficient as well as more responsive to changes could be done.

The order of operations I believe could fix this issue as well as make it more responsive.

First loop

For each fetched secret the provider makes we store the key and value fetched, as well as subscribe to kubernetes events for a change in that secrets value.

Sequential loop

Mark all current secret keys as old. Then do the same as above from the first loop, except don't fetch the value unless it was marked dirty from a secret change event. Then all secrets that are marked old can be removed from the cache and unsubscribe from the events.

[barkbay]

@blakerouse the idea is to reuse the cache from the controller runtime: https://github.com/kubernetes-sigs/controller-runtime/blob/d5bc8734caccabddac6a1bea250b0b9d771d318d/pkg/cache/cache.go#L61-L70

[jlind23]

@bturquet after discussing with @michalpristas and team it looks that this rather falls into your area.
Would you mind finding someone to take a stab at it? This is severely impacting serverless deployments.
cc @pierrehilbert @cmacknz

[bturquet]

@constanca-m as you are on rotation this week, could you have a look please ?

[constanca-m]

Hi @barkbay, I noticed you opened this PR and closed it: #3464. Why was that? Is the intention of this issue to continue the work on that PR or to create a new PR to fix the performance issue?

[barkbay]

Why was that?

I wrongly assumed that my experience building K8s controllers might help, however to solve this issue I think I would have to invest more time understanding the Agent architecture, which is not possible given my current priorities.

Is the intention of this issue to continue the work on that PR or to create a new PR to fix the performance issue?

I think you are in a better position to answer this question 😄

[constanca-m]

I created a separate issue for this #3594 and added it to our boards.