Merge branch '8.11' into mergify/bp/8.11/pr-12112

NOTICE.txt

@@ -573,11 +573,11 @@ Contents of probable licence file $GOMODCACHE/github.com/elastic/[email protected]
 
 --------------------------------------------------------------------------------
 Dependency : github.com/elastic/beats/v7
-Version: v7.0.0-alpha2.0.20231129140234-9a93c21c218c
+Version: v7.0.0-alpha2.0.20231204122024-b30a5ec82af0
 Licence type (autodetected): Elastic
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/elastic/beats/[email protected].20231129140234-9a93c21c218c/LICENSE.txt:
+Contents of probable licence file $GOMODCACHE/github.com/elastic/beats/[email protected].20231204122024-b30a5ec82af0/LICENSE.txt:
 
 Source code in this repository is variously licensed under the Apache License
 Version 2.0, an Apache compatible license, or the Elastic License. Outside of

apm-server.yml

@@ -170,14 +170,10 @@ apm-server:
       # Note that values configured without a time unit will be interpreted as seconds.
       #cache.expiration: 5m
 
-      # Source maps may be fetched from Elasticsearch by using the output.elasticsearch configuration,
-      # and running apm-server standalone.
+      # Source map retrieval location.
       #
-      # Note: fetching source maps from Elasticsearch is not supported if apm-server is being managed by
-      # Fleet. This configuration is only applicable to standalone apm-servers, for backwards compatibility
-      # with source maps stored in Elasticsearch by older versions of apm-server. New source maps must now
-      # be uploaded via Kibana, and `apm-server.kibana` configured in standalone apm-servers for fetching
-      # them.
+      # If using an output other than Elasticsearch that is writing to Elasticsearch, you must
+      # set this option. If not set, the standard output elasticsearch configuration is used.
       #elasticsearch:
         # Array of hosts to connect to.
         # Scheme and port can be left out and will be set to the default (`http` and `9200`).

docs/custom-index-template.asciidoc

@@ -15,7 +15,7 @@ These index templates are composed of multiple component templates--reusable bui
 that configure index mappings, settings, and aliases.
 
 The default APM index templates can be viewed in {kib}.
-Navigate to **{stack-manage-app}** > **Index Management** > **Index Templates**, and search for `apm`.
+Navigate to **{stack-manage-app}** → **Index Management** → **Index Templates**, and search for `apm`.
 Select any of the APM index templates to view their relevant component templates.
 
 [discrete]
@@ -30,21 +30,27 @@ When you install the APM integration, {fleet} creates a default `@custom` compon
 You can edit this `@custom` component template to customize your {es} indices.
 
 First, determine which <<apm-data-streams,data stream>> you'd like to edit.
-Then, open {kib} and navigate to **{stack-manage-app}** > **Index Management** > **Component Templates**.
+Then, open {kib} and navigate to **{stack-manage-app}** → **Index Management** → **Component Templates**.
 
 Custom component templates are named following this pattern: `<name_of_data_stream>@custom`.
 Search for the name of the data stream, like `traces-apm`, and select its custom component template.
 In this example, that'd be, `traces-apm@custom`.
-Then click **Manage** > **Edit**.
+Then click **Manage** → **Edit**.
 
-Add any custom index settings, metadata, or mappings.
-For example, you may want to...
+Add any custom metadata, index settings, or mappings.
+
+[discrete]
+[[custom-index-template-index-settings]]
+==== Index settings
+
+In the **Index settings** step, you can specify custom {ref}/index-modules.html#index-modules-settings[index settings].
+For example, you could:
 
 * Customize the index lifecycle policy applied to a data stream.
 See <<data-streams-custom-policy,custom index lifecycle policies>> for a walk-through.
 
 * Change the number of {ref}/scalability.html[shards] per index.
-Specify the number of primary shards in the **index settings**:
+Specify the number of primary shards:
 +
 [source,json]
 ----
@@ -56,7 +62,7 @@ Specify the number of primary shards in the **index settings**:
 ----
 
 * Change the number of {ref}/docs-replication.html[replicas] per index.
-Specify the number of replica shards in the **index settings**:
+Specify the number of replica shards:
 +
 [source,json]
 ----
@@ -67,6 +73,29 @@ Specify the number of replica shards in the **index settings**:
 }
 ----
 
+[discrete]
+[[custom-index-template-mappings]]
+==== Mappings
+
+{ref}/mapping.html[Mapping] is the process of defining how a document, and the fields it contains, are stored and indexed.
+In the *Mappings* step, you can add custom field mappings.
+For example, you could:
+
+* Add custom field mappings that you can index on and search.
+In the *Mapped fields* tab, add a new field including the {ref}/mapping-types.html[field type]:
++
+image::images/custom-index-template-mapped-fields.png[Editing a component template to add a new mapped field]
+
+* Add a {ref}/runtime.html[runtime field] that is evaluated at query time.
+In the *Runtime fields* tab, click *Create runtime field* and provide a field name,
+type, and optionally a script:
++
+image::images/custom-index-template-runtime-fields.png[Editing a component template to add a new runtime field]
+
+[discrete]
+[[custom-index-template-rollover]]
+=== Roll over the data stream
+
 Changes to component templates are not applied retroactively to existing indices.
 For changes to take effect, you must create a new write index for the data stream.
 This can be done with the {es} {ref}/indices-rollover-index.html[Rollover API].

docs/source-map-how-to.asciidoc

@@ -95,19 +95,21 @@ module.exports = {
 
 [float]
 [[source-map-rum-upload]]
-=== Upload the source map to {kib}
+=== Upload the source map
+
+TIP: When uploading a source map, ensure that RUM support is enabled in the APM integration.
 
 {kib} exposes a {kibana-ref}/rum-sourcemap-api.html[source map endpoint] for uploading source maps.
 Source maps can be uploaded as a string, or as a file upload.
-Before uploading a source map, ensure that RUM support is enabled in the APM integration
 
 Let's look at two different ways to upload a source map: curl and a custom application.
 Each example includes the four fields necessary for APM Server to later map minified code to its source:
 
-* `service_name` - Should match the `serviceName` from step one
-* `service_version` - Should match the `serviceVersion` from step one
-* `bundle_filepath` - The absolute path of the final bundle as used in the web application
-* `sourcemap` - The location of the source map.
+* `service_name`: Should match the `serviceName` from step one.
+* `service_version`: Should match the `serviceVersion` from step one.
+* `bundle_filepath`: The absolute path of the final bundle as used in the web application.
+* `sourcemap`: The location of the source map.
+
 If you have multiple source maps, you'll need to upload each individually.
 
 [float]
@@ -164,3 +166,12 @@ request.post({url: 'http://localhost:5601/api/apm/sourcemaps',formData: formData
   }
 })
 ----
+
+[float]
+[[source-map-next]]
+=== What happens next
+
+Source maps are stored in {es}. When you upload a source map, a new {es} document is created
+containing the contents of the source map.
+When a RUM request comes in, APM Server will make use of these source map documents to apply the
+source map logic to the event's stack traces.

go.mod

@@ -10,7 +10,7 @@ require (
 	github.com/dustin/go-humanize v1.0.1
 	github.com/elastic/apm-aggregation v0.0.0-20230815024520-e75a37d9ddd6
 	github.com/elastic/apm-data v0.1.1-0.20230928144734-40b63726f3b7
-	github.com/elastic/beats/v7 v7.0.0-alpha2.0.20231129140234-9a93c21c218c
+	github.com/elastic/beats/v7 v7.0.0-alpha2.0.20231204122024-b30a5ec82af0
 	github.com/elastic/elastic-agent-client/v7 v7.4.0
 	github.com/elastic/elastic-agent-libs v0.6.2
 	github.com/elastic/elastic-agent-system-metrics v0.8.2

go.sum

@@ -136,8 +136,8 @@ github.com/elastic/apm-aggregation v0.0.0-20230815024520-e75a37d9ddd6 h1:Js+C3HE
 github.com/elastic/apm-aggregation v0.0.0-20230815024520-e75a37d9ddd6/go.mod h1:ba3gaJCuhxXN/O5AuiI56xxd6DukQdVOK0NfpzBntNo=
 github.com/elastic/apm-data v0.1.1-0.20230928144734-40b63726f3b7 h1:QcfCHAh1bHj9WN6y75bD6mBlIIlz9dJVKJSJVXCu4TM=
 github.com/elastic/apm-data v0.1.1-0.20230928144734-40b63726f3b7/go.mod h1:lMTMoCWNadiDJih/tLechuMTtumEeedtKJlBOYAv030=
-github.com/elastic/beats/v7 v7.0.0-alpha2.0.20231129140234-9a93c21c218c h1:ngARH1pZ3zP5nf1SnNRfBhewDSemoWOaPj/Y0y2A9S0=
-github.com/elastic/beats/v7 v7.0.0-alpha2.0.20231129140234-9a93c21c218c/go.mod h1:yIwy1g6P8eX/AmxB1qu86PC+HxH6VTAyE4Jnjb9TkoE=
+github.com/elastic/beats/v7 v7.0.0-alpha2.0.20231204122024-b30a5ec82af0 h1:2+kLmovv8rWdK4A0zH8XBGlEUQy0fXOlSy/DfcGwq6c=
+github.com/elastic/beats/v7 v7.0.0-alpha2.0.20231204122024-b30a5ec82af0/go.mod h1:tyPnUi2tTvUlfby7/yBWoDvibj5DKidutSS2GItT6qM=
 github.com/elastic/elastic-agent-autodiscover v0.6.4 h1:K+xC7OGgcy4fLXVuGgOGLs+eXCqRnRg2SQQinxP+KsA=
 github.com/elastic/elastic-agent-autodiscover v0.6.4/go.mod h1:5+7NIBAILc0GkgxYW3ckXncu5wRZfltZhTY4aZAYP4M=
 github.com/elastic/elastic-agent-client/v7 v7.4.0 h1:h75oTkkvIjgiKVm61NpvTZP4cy6QbQ3zrIpXKGigyjo=