Skip to content

Commit 67c6a3c

Browse files
niedbalskiniedbalski
authored
Merge pull request #1598 from fluent/docs/conditional-processing
conditional: add conditional processing documentation
2 parents 965257d + ba34a28 commit 67c6a3c

File tree

3 files changed

+260
-0
lines changed

3 files changed

+260
-0
lines changed

‎SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@
147147
* [Metrics Selector](pipeline/processors/metrics-selector.md)
148148
* [OpenTelemetry Envelope](pipeline/processors/opentelemetry-envelope.md)
149149
* [SQL](pipeline/processors/sql.md)
150+
* [Conditional processing](pipeline/processors/conditional-processing.md)
150151
* [Filters](pipeline/filters/README.md)
151152
* [AWS Metadata](pipeline/filters/aws-metadata.md)
152153
* [CheckList](pipeline/filters/checklist.md)

‎pipeline/processors/README.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,24 @@
11
# Processors
22

3+
Processors are components that modify, transform, or enhance data as it flows through Fluent Bit.
4+
Unlike [filters](../filters/README.md), processors are tightly coupled to inputs, which means they
5+
execute immediately and avoid creating a performance bottleneck.
6+
7+
Additionally, filters can be implemented in a way that mimics the behavior of processors, but
8+
processors can't be implemented in a way that mimics filters.
9+
10+
## Available processors
11+
12+
Fluent Bit offers the following processors:
13+
14+
- [Content Modifier](content-modifier.md): Manipulate the content, metadata, and attributes of logs and traces.
15+
- [Labels](labels.md): Add, update, or delete metric labels.
16+
- [Metrics Selector](metrics-selector.md): Choose which metrics to keep or discard.
17+
- [OpenTelemetry Envelope](opentelemetry-envelope.md): Transform logs into an OpenTelemetry-compatible format.
18+
- [SQL](sql.md): Use SQL queries to extract log content.
19+
20+
## Features
21+
22+
All available processors include the following features:
23+
24+
- [Conditional Processing](conditional-processing.md): Apply processors selectively to records that meet specific criteria.

‎pipeline/processors/conditional-processing.md

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
# Conditional processing
2+
3+
Conditional processing lets you selectively apply [processors](README.md) to logs based on
4+
the value of fields that those logs contain. This feature lets you create processing pipelines
5+
that only process records that meet certain criteria, and ignore the rest.
6+
7+
## Configuration
8+
9+
You can turn a standard processor into a conditional processor by adding a `condition` block to the
10+
processor's YAML configuration settings.
11+
12+
{% hint style="info" %}
13+
Conditional processing is only available for
14+
[YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md),
15+
not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).
16+
{% endhint %}
17+
18+
19+
These `condition` blocks use the following syntax:
20+
21+
```yaml
22+
pipeline:
23+
inputs:
24+
<...>
25+
processors:
26+
logs:
27+
- name: {processor_name}
28+
<...>
29+
condition:
30+
op: {and|or}
31+
rules:
32+
- field: {field_name1}
33+
op: {comparison_operator}
34+
value: {comparison_value1}
35+
- field: {field_name2}
36+
op: {comparison_operator}
37+
value: {comparison_value2}
38+
<...>
39+
```
40+
41+
Each processor can only have a single `condition` block, but can have multiple rules within that condition.
42+
These rules are stored as items in the `condition.rules` array.
43+
44+
### Condition evaluation
45+
46+
The `condition.op` parameter specifies the condition's evaluation logic. It has two possible values:
47+
48+
- `and`: All rules in the `condition.rules` array must evaluate to `true` for the condition to be met.
49+
- `or`: One or more rules in the `conditions.rules` array must evaluate to `true` for the condition to be met.
50+
51+
### Rules
52+
53+
Each item in the `condition.rules` array must include values for the following parameters:
54+
55+
| Parameter | Description |
56+
| --- | --- |
57+
| `field` | The field within your logs to evaluate. The value of this parameter must use [the correct syntax](#field-access) to access the fields inside logs. |
58+
| `op` | The [comparison operator](#comparison-operators) to evaluate whether the rule is true. This parameter (`condition.rules.op`) is distinct from the `condition.op` parameter and has different possible values. |
59+
| `value` | The value of the specified log field to use in your comparison. Optionally, you can provide [an array that contains multiple values](#array-of-values). |
60+
61+
Rules are evaluated against each log that passes through your data pipeline. For example, given a rule with these parameters:
62+
63+
```
64+
- field: "$status"
65+
op: eq
66+
value: 200
67+
```
68+
69+
This rule evaluates to `true` for a log that contains the string `'status':200`, but evaluates to `false` for a log that contains the string `'status':403`.
70+
71+
#### Field access
72+
73+
The `conditions.rules.field` parameter uses [record accessor syntax](/administration/configuring-fluent-bit/classic-mode/record-accessor.md) to reference fields inside logs.
74+
75+
You can use `$field` syntax to access a top-level field, and `$field['child']['subchild']` to access nested fields.
76+
77+
#### Comparison operators
78+
79+
The `conditions.rules.op` parameter has the following possible values:
80+
81+
- `eq`: equal to
82+
- `neq`: not equal to
83+
- `gt`: greater than
84+
- `lt`: less than
85+
- `gte`: greater than or equal to
86+
- `lte`: less than or equal to
87+
- `regex`: matches a regular expression
88+
- `not_regex`: does not match a regular expression
89+
- `in`: is included in the specified array
90+
- `not_in`: is not included in the specified array
91+
92+
## Examples
93+
94+
### Basic condition
95+
96+
This example applies a condition that only processes logs that contain the string `{"request": {"method": "POST"`:
97+
98+
```yaml
99+
pipeline:
100+
inputs:
101+
- name: dummy
102+
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource"}}'
103+
tag: request.log
104+
processors:
105+
logs:
106+
- name: content_modifier
107+
action: insert
108+
key: modified_if_post
109+
value: true
110+
condition:
111+
op: and
112+
rules:
113+
- field: "$request['method']"
114+
op: eq
115+
value: "POST"
116+
```
117+
118+
### Multiple conditions with `and`
119+
120+
This example applies a condition that only processes logs when all of the specified rules are met:
121+
122+
```yaml
123+
pipeline:
124+
inputs:
125+
- name: dummy
126+
dummy: '{"request": {"method": "POST", "path": "/api/v1/sensitive-data"}}'
127+
tag: request.log
128+
processors:
129+
logs:
130+
- name: content_modifier
131+
action: insert
132+
key: requires_audit
133+
value: true
134+
condition:
135+
op: and
136+
rules:
137+
- field: "$request['method']"
138+
op: eq
139+
value: "POST"
140+
- field: "$request['path']"
141+
op: regex
142+
value: "\/sensitive-.*"
143+
```
144+
145+
### Multiple conditions with `or`
146+
147+
This example applies a condition that only processes logs when one or more of the specified rules are met:
148+
149+
```yaml
150+
pipeline:
151+
inputs:
152+
- name: dummy
153+
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource", "status_code": 200, "response_time": 150}}'
154+
tag: request.log
155+
processors:
156+
logs:
157+
- name: content_modifier
158+
action: insert
159+
key: requires_performance_check
160+
value: true
161+
condition:
162+
op: or
163+
rules:
164+
- field: "$request['response_time']"
165+
op: gt
166+
value: 100
167+
- field: "$request['status_code']"
168+
op: gte
169+
value: 400
170+
```
171+
172+
### Array of values
173+
174+
This example uses an array for the value of `condition.rules.value`:
175+
176+
```yaml
177+
pipeline:
178+
inputs:
179+
- name: dummy
180+
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource"}}'
181+
tag: request.log
182+
processors:
183+
logs:
184+
- name: content_modifier
185+
action: insert
186+
key: high_priority_method
187+
value: true
188+
condition:
189+
op: and
190+
rules:
191+
- field: "$request['method']"
192+
op: in
193+
value: ["POST", "PUT", "DELETE"]
194+
```
195+
196+
### Multiple processors with conditions
197+
198+
This example uses multiple processors with conditional processing enabled for each:
199+
200+
```yaml
201+
pipeline:
202+
inputs:
203+
- name: dummy
204+
dummy: '{"log": "Error: Connection refused", "level": "error", "service": "api-gateway"}'
205+
tag: app.log
206+
processors:
207+
logs:
208+
- name: content_modifier
209+
action: insert
210+
key: alert
211+
value: true
212+
condition:
213+
op: and
214+
rules:
215+
- field: "$level"
216+
op: eq
217+
value: "error"
218+
- field: "$service"
219+
op: in
220+
value: ["api-gateway", "authentication", "database"]
221+
222+
- name: content_modifier
223+
action: insert
224+
key: paging_required
225+
value: true
226+
condition:
227+
op: and
228+
rules:
229+
- field: "$log"
230+
op: regex
231+
value: "(?i)(connection refused|timeout|crash)"
232+
- field: "$level"
233+
op: in
234+
value: ["error", "fatal"]
235+
```
236+
237+
This configuration would add the `alert` field to error logs from critical services, and add the `paging_required` field to errors containing specific critical patterns.

0 commit comments

Comments
 (0)