Switch apm integration package to input-only package

[simitt]

Migrate the APM integration package to become an input-only package (elastic/package-spec#319), meaning that the package would not define any data streams.

This can only be done once #11528 is implemented.

[axw]

We should take a staged approach to minimise risk:

• Enable the plugin in Serverless QA, concurrently with the APM Fleet integration package. This will build confidence that enabling the plugin doesn't break anything, but we will continue relying on the integration package for the templates & pipelines.
• Disable the integration package in QA, so we start relying on the plugin for index templates & pipelines. One complication here is that the Serverless Kibana config files (e.g. https://github.com/elastic/kibana/blob/main/config/serverless.oblt.yml) are not environment-specific, so we'll need to do something about that.
• Repeat above steps for non-QA environments.
• Enable the plugin by default in Elasticsearch.
• Remove the data stream definitions from the integration package.

In the end we should still have the integration installed for non-Serverless, but not in any Serverless environments. For steps 4 and 5, we can test locally in apm-server by adapting #12066.

Some other things require further thought and discussion before going all the way with rolling this out to non-Serverless:

• The index templates installed by the plugin use Data Stream Lifecycle (DSL) exclusively
  • DSL is in tech preview
  • This change would break existing ILM customisation -- https://www.elastic.co/guide/en/apm/guide/current/ilm-how-to.html
    • Verify that custom ILM takes precedence over DSL -- Switch apm integration package to input-only package #11529 (comment)
• Need to make sure @custom templates and pipelines are all working as they do with the Fleet integration package.
  • Pipeline fixes implemented in x-pack/plugin/apm-data: fix @custom pipeline support elasticsearch#104113
  • Component template fixes implemented in x-pack/plugin/apm-data: fix @custom component templates elasticsearch#104182
• We should run some Rally benchmarks before/after switching.

[axw]

This change would break existing ILM customisation

I'm not 100% sure about this now. If ILM overrides DSL then it's not breaking, will need to look into this some more.

index.lifecycle.prefer_ilm (defaults to true) controls whether ILM takes precedence. So I think we're good here, but we should confirm with some testing.

EDIT: confirmed that the existing instructions still work, even with DSL enabled. So, that's pretty great - we can make this change first and update the instructions later to cover both DSL (simple) and ILM (advanced).

[axw]

Enable the plugin in Serverless QA

Requires https://elasticco.atlassian.net/browse/CP-5487

[axw]

Enabled in serverless QA: