Nothing ingested after upgrade

[maretodoric]

Describe the bug
After upgrading cluster and fleet agents from 8.9.1 to 8.15.2, i have no services or no traces uploaded to elasticsearch, with following error message in all elastic agents:

Document produced by APM Server v8.15.2, which is newer than the installed APM integration (v8.9.0-preview-1686650721)

I have made sure that Elastic APM Integration is updated to 8.15.2 with all agents

I have also attempted to rollover the data stream but it did not help.

I had the same issue with a different agent policy using different namespace and only way to fix it is by removing all data streams and templates of that namespace. But that way I've lost a lot of data, default namespace has a lot more data and i would like to avoid losing any if possible.

To Reproduce
Steps to reproduce the behavior:

1. Upgrade cluster
2. Upgrade Fleet Agents
   3a. Go to Observability -> APM and witness no services or
   3b. Go to Observability -> APM -> Select Service -> Select Transaction and witness no traces for selected transaction

Expected behavior
I should have a list of all transactions, instead i get a message The selected trace cannot be found

Debug logs

Click to expand

{"log.level":"error","@timestamp":"2024-10-08T15:06:59.799+0200","message":"failed to index documents in 'traces-apm-default' (fail_processor_exception): fail_processor_exception: Document produced by APM Server v8.15.2, which is newer than the installed APM integration (v8.9.0-preview-1686650721). The APM integration must be upgraded.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","documents":8,"ecs.version":"1.6.0","log.origin":{"file.line":443,"file.name":"[email protected]/appender.go","function":"github.com/elastic/go-docappender/v2.(*Appender).flush"},"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-10-08T15:06:59.799+0200","message":"failed to index documents in 'metrics-apm.internal-default' (fail_processor_exception): fail_processor_exception: Document produced by APM Server v8.15.2, which is newer than the installed APM integration (v8.9.0-preview-1686650721). The APM integration must be upgraded.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":443,"file.name":"[email protected]/appender.go","function":"github.com/elastic/go-docappender/v2.(*Appender).flush"},"service.name":"apm-server","documents":13,"ecs.version":"1.6.0","ecs.version":"1.6.0"}

[graphaelli]

My apologies for not seeing this report sooner. This is documented as a known issue including instructions for remediation.