[k8s docs] include how to ingest rotated container log files, including GZIP compressed

deploy/kubernetes/filebeat-kubernetes.yaml

@@ -112,23 +112,52 @@ metadata:
 data:
   filebeat.yml: |-
     filebeat.inputs:
-    - type: filestream
-      id: kubernetes-container-logs
-      paths:
-        - /var/log/containers/*.log
-      parsers:
-        - container: ~
-      prospector:
-        scanner:
-          fingerprint.enabled: true
-          symlinks: true
-      file_identity.fingerprint: ~
-      processors:
-        - add_kubernetes_metadata:
-            host: ${NODE_NAME}
-            matchers:
-            - logs_path:
-                logs_path: "/var/log/containers/"
+      - type: filestream
+        id: kubernetes-container-logs
+        paths:
+          - /var/log/containers/*.log
+        parsers:
+          - container: ~
+        prospector:
+          scanner:
+            fingerprint.enabled: true
+            symlinks: true
+        file_identity.fingerprint: ~
+        processors:
+          - add_kubernetes_metadata:
+              host: ${NODE_NAME}
+              matchers:
+                - logs_path:
+                    logs_path: "/var/log/containers/"
+
+          ## To enable collection of rotated logs, replace the above `filebeat.inputs` configuration with this.
+          ## WARNING:
+          ##    - enabling rotated pod logs ingestion might cause data re-ingestion, refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/running-on-kubernetes#_kubernetes_deploy_manifests
+          ##    - container metadata isn't available when collecting logs from /var/log/pods/. Refer to add_kubernetes_metadata docs for details: https://www.elastic.co/docs/reference/beats/filebeat/add-kubernetes-metadata#_logs_path
+          #- type: filestream
+          #  id: kubernetes-container-logs
+          #  gzip_experimental: true # BETA: enable gzip decompression. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files
+          #  parsers:
+          #    - container: ~
+          #  paths:
+          #    - /var/log/pods/*/*/*.log*
+          #  prospector:
+          #    scanner:
+          #      fingerprint.enabled: true
+          #      symlinks: true
+          #  file_identity.fingerprint: ~
+          #  processors:
+          #    - add_kubernetes_metadata:
+          #        host: ${NODE_NAME}
+          #        default_indexers.enabled: false
+          #        default_matchers.enabled: false
+          #        indexers:
+          #          - pod_uid:
+          #        matchers:
+          #          - logs_path:
+          #              logs_path: "/var/log/pods/"
+          #              resource_type: "pod"
+          #
 
     # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
     # filebeat.autodiscover:
@@ -141,14 +170,20 @@ data:
     #        id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
     #        paths:
     #        - /var/log/containers/*-${data.kubernetes.container.id}.log
+    # # To enable collection of rotated logs, use the following configuration instead. WARNING: It might cause data re-ingestion for existing deployments. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/running-on-kubernetes#ingesting-rotated-log-files
+    # #         id: kubernetes-container-logs-${data.kubernetes.pod.uid}-${data.kubernetes.container.name}
+    # #         gzip_experimental: true # BETA: enable gzip decompression. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files
+    # #         paths:
+    # #           - /var/log/pods/${data.kubernetes.namespace}_${data.kubernetes.pod.name}_${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/*.log*
+
     #        parsers:
     #        - container: ~
     #        prospector:
     #         scanner:
     #           fingerprint.enabled: true
     #           symlinks: true
     #        file_identity.fingerprint: ~
-
+    
     processors:
       - add_cloud_metadata:
       - add_host_metadata:

deploy/kubernetes/filebeat/README.md

@@ -7,13 +7,20 @@
 By deploying filebeat as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)
 we ensure we get a running filebeat daemon on each node of the cluster.
 
-Docker logs host folder (`/var/lib/docker/containers`) is mounted on the
-filebeat container. Filebeat will start an input for these files and start
-harvesting them as they appear.
+Kubernetes stores logs on `/var/log/pods` and uses symlinks on `/var/log/containers`
+for active log files. Refer to the official [Kubernetes documentation on log rotation](https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation)
+for more details.
+
+When the directory is mounted on the filebeat container. Filebeat will start an 
+input for these files and start harvesting them as they appear.
 
 Everything is deployed under `kube-system` namespace, you can change that by
 updating YAML manifests under this folder.
 
+Filebeat can also ship rotated logs, including the GZIP-compressed. Refer
+to [Run Filebeat on Kubernetes](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files)
+for instructions on how to enable this.
+
 ### Settings
 
 We use official [Beats Docker images](https://github.com/elastic/beats-docker),

deploy/kubernetes/filebeat/filebeat-configmap.yaml

@@ -8,23 +8,52 @@ metadata:
 data:
   filebeat.yml: |-
     filebeat.inputs:
-    - type: filestream
-      id: kubernetes-container-logs
-      paths:
-        - /var/log/containers/*.log
-      parsers:
-        - container: ~
-      prospector:
-        scanner:
-          fingerprint.enabled: true
-          symlinks: true
-      file_identity.fingerprint: ~
-      processors:
-        - add_kubernetes_metadata:
-            host: ${NODE_NAME}
-            matchers:
-            - logs_path:
-                logs_path: "/var/log/containers/"
+      - type: filestream
+        id: kubernetes-container-logs
+        paths:
+          - /var/log/containers/*.log
+        parsers:
+          - container: ~
+        prospector:
+          scanner:
+            fingerprint.enabled: true
+            symlinks: true
+        file_identity.fingerprint: ~
+        processors:
+          - add_kubernetes_metadata:
+              host: ${NODE_NAME}
+              matchers:
+                - logs_path:
+                    logs_path: "/var/log/containers/"
+
+          ## To enable collection of rotated logs, replace the above `filebeat.inputs` configuration with this.
+          ## WARNING:
+          ##    - enabling rotated pod logs ingestion might cause data re-ingestion, refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/running-on-kubernetes#_kubernetes_deploy_manifests
+          ##    - container metadata isn't available when collecting logs from /var/log/pods/. Refer to add_kubernetes_metadata docs for details: https://www.elastic.co/docs/reference/beats/filebeat/add-kubernetes-metadata#_logs_path
+          #- type: filestream
+          #  id: kubernetes-container-logs
+          #  gzip_experimental: true # BETA: enable gzip decompression. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files
+          #  parsers:
+          #    - container: ~
+          #  paths:
+          #    - /var/log/pods/*/*/*.log*
+          #  prospector:
+          #    scanner:
+          #      fingerprint.enabled: true
+          #      symlinks: true
+          #  file_identity.fingerprint: ~
+          #  processors:
+          #    - add_kubernetes_metadata:
+          #        host: ${NODE_NAME}
+          #        default_indexers.enabled: false
+          #        default_matchers.enabled: false
+          #        indexers:
+          #          - pod_uid:
+          #        matchers:
+          #          - logs_path:
+          #              logs_path: "/var/log/pods/"
+          #              resource_type: "pod"
+          #
 
     # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
     # filebeat.autodiscover:
@@ -37,14 +66,20 @@ data:
     #        id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
     #        paths:
     #        - /var/log/containers/*-${data.kubernetes.container.id}.log
+    # # To enable collection of rotated logs, use the following configuration instead. WARNING: It might cause data re-ingestion for existing deployments. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/running-on-kubernetes#ingesting-rotated-log-files
+    # #         id: kubernetes-container-logs-${data.kubernetes.pod.uid}-${data.kubernetes.container.name}
+    # #         gzip_experimental: true # BETA: enable gzip decompression. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files
+    # #         paths:
+    # #           - /var/log/pods/${data.kubernetes.namespace}_${data.kubernetes.pod.name}_${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/*.log*
+
     #        parsers:
     #        - container: ~
     #        prospector:
     #         scanner:
     #           fingerprint.enabled: true
     #           symlinks: true
     #        file_identity.fingerprint: ~
-
+    
     processors:
       - add_cloud_metadata:
       - add_host_metadata:

docs/reference/filebeat/file-log-rotation.md

@@ -7,7 +7,7 @@ applies_to:
 
 # Log rotation results in lost or duplicate events [file-log-rotation]
 
-Filebeat supports reading from rotating log files. However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. To avoid this issue:
+Filebeat supports reading from rotating log files, [including GZIP compressed files](/reference/filebeat/filebeat-input-filestream.md#reading-gzip-files). However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. To avoid this issue:
 
 * **Be careful when changing the default file identity of the `filestream` input**
 

docs/reference/filebeat/running-on-kubernetes.md

@@ -48,6 +48,69 @@ By default, Filebeat sends events to an existing Elasticsearch deployment, if pr
   value: changeme
 ```
 
+## Ingesting rotated log files
+
+Filebeat can also ship the rotated logs, including the GZIP-compressed logs.
+Kubernetes stores logs on `/var/log/pods` and uses symlinks on `/var/log/containers`
+for active log files. For full details, refer to the official
+[Kubernetes documentation on log rotation](https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation).
+
+Ingest rotated logs by enabling decompression of GZIP files and changing the monitored 
+path to `/var/log/pods/` instead of `/var/log/containers`, which only contains
+active log files.
+
+::::{warning}
+Data Duplication: When you change the path on an existing deployment,
+filebeat reads all existing files in the new directory from the beginning.
+This action causes a one-time re-ingestion of the log files.
+
+After the initial scan, filebeat tracks files normally and will only
+ingest new log data.
+::::
+The following is an example configuration for ingesting rotated log files:
+
+```yaml
+    filebeat.inputs:
+       - type: filestream
+         id: kubernetes-container-logs
+         gzip_experimental: true # BETA: enable gzip decompression. Refer to the docs for details: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files
+         parsers:
+            - container: ~
+         paths:
+            - /var/log/pods/*/*/*.log* <1>
+         prospector:
+            scanner:
+               fingerprint.enabled: true
+               symlinks: true
+         file_identity.fingerprint: ~
+         processors:
+            - add_kubernetes_metadata:
+                 host: ${NODE_NAME}
+                 default_indexers.enabled: false
+                 default_matchers.enabled: false
+                 indexers:
+                    - pod_uid:
+                 matchers:
+                    - logs_path:
+                         logs_path: "/var/log/pods/" <2>
+                         resource_type: "pod" <2>
+```
+
+1. `/var/log/pods/` contains the active log files as well as rotated log files.
+
+2. `add_kubernetes_metadata` needs to be configured to match pod metadata based
+on the new path, `/var/log/pods/`. Pod metadata do not include container 
+metadata. Refer to the [add_kubernetes_metadata](https://www.elastic.co/docs/reference/beats/filebeat/add-kubernetes-metadata#_logs_path)
+documentation for details.
+
+::::{warning}
+[add_kubernetes_metadata](https://www.elastic.co/docs/reference/beats/filebeat/add-kubernetes-metadata#_logs_path)
+configured as shown above adds *pod* metadata, which does not include
+container data (such as `kubernetes.container.name`). If you need container
+metadata, you must consider using autodiscover instead. Refer to the
+[autodiscover documentation](https://www.elastic.co/docs/reference/beats/filebeat/configuration-autodiscover#_kubernetes) for details.
+::::
+
 
 ### Running Filebeat on control plane nodes [_running_filebeat_on_control_plane_nodes]
 
@@ -263,4 +326,4 @@ For the example we're using:
 
 Refer to the official [Kubernetes documentation on log rotation](https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation).
 
-Filebeat supports reading from rotating log files. However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. For more information, refer to [Log rotation results in lost or duplicate events](/reference/filebeat/file-log-rotation.md).
+Filebeat supports reading from rotating log files, [including GZIP file](/reference/filebeat/filebeat-input-filestream.md#reading-gzip-files). However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. For more information, refer to [Log rotation results in lost or duplicate events](/reference/filebeat/file-log-rotation.md).