Description
Bug Report
Describe the bug
The condition key seems to overlap with condition parameter for modify filter. This causes configuration errors when using modify filter as an input processor. I tried replacing modify filter with content_modifier processor but it does not support conditions such as key_exists. I have also noticed that when value field value is an empty string it causes content_modifier processor errors.
To Reproduce
Following input processor causes error messages during fluentbit init:
pipeline:
inputs:
- name: tail
tag: ecs.*
path: /var/lib/docker/containers/*/*-json.log
processors:
logs:
- name: modify
condition: key_does_not_exist container
set: container unknown
This example results in following error logs:
Fluent Bit v4.0.0
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ ___ _____
| ___| | | | | ___ (_) | / || _ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
[2025/04/02 20:49:15] [error] [processor] condition must be a map
[2025/04/02 20:49:15] [error] failed to set condition for processor 'modify'
[2025/04/02 20:49:15] [error] failed to load 'logs' processors
I have tried replacing modify filter with content_modifier processor with following config:
pipeline:
inputs:
- name: tail
tag: ecs.*
path: /var/lib/docker/containers/*/*-json.log
processors:
logs:
- name: content_modifier
condition:
op: and
rules:
- field: container
op: not_regex // not sure how else to check if field exists
value: ".+"
key: container
action: insert
value: unknown
Unfortunately, last example does not add container key when it does not exist.
Expected behavior
Be able to use filter conditions when using modify filter as input processor.
Be able to add keys when key/field does not exists already.
Your Environment
- Version used: 4.0.0
- Configuration: YAML
- Environment name and version (e.g. Kubernetes? What version?): AWS ECS
- Operating System and version: Linux
- Filters and plugins: Modify