Description
Setup as follows.
- Fluentbit latest (3.2.3)
- AWS Linux 2023 latest AMI
- Input is UDP syslog
- Syslog processes the whole line as a single string/key-value called 'message'.
- Lua gets called to remove any of the single quotes in string that Sonicwall handily decided to fire in on random values.
- Lua successfully removes single quotes.
- Log is now able to be Parsed by Logfmt, and it does this perfectly without the single quotes.
- Content modifier called on record to extract ip and port from combined ip:port entry in log.
- Content modifier seems to cause a load of failed to process chunk messages in stdout :(
- Happens only a few seconds after starting fluent-bit process, is not sporadic, loads of entries.
[2024/12/29 00:40:43] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:44] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:44] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:45] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:45] [ warn] [processor] failed to process chunk
No clue what I am doing wrong! This ONLY happens when the content modifier is added. Content modifier MUST be called after LogFmt Parsing, so that it has key-value pairs to process.
Example of log being processed:-
<134> id=firewall000 sn=000000000 time="2024-12-27 10:41:29" fw=10.11.112.4 pri=6 c=1024 gcat=6 m=537 src=100.64.2.172:19690 dst=10.11.11.11:443 proto=tcp/https sent=216 rcvd=164 cdur=2016 app=49177 appName='General HTTPS' msg="Connection Closed" fw_action="NA"
Example of final log result in Cloudwatch:-
{
"<134>": true,
"id": "firewall000",
"sn": "0000000000",
"time": "2024-12-29 00:52:06",
"fw": "10.10.12.4",
"pri": "6",
"c": "1024",
"gcat": "6",
"m": "537",
"src": "100.64.1.164:5520",
"dst": "10.10.10.148:8444",
"proto": "tcp/8444",
"sent": "240",
"cdur": "38016",
"app": "49201",
"appName": "General TCP",
"msg": "Connection Closed",
"fw_action": "NA",
"srcip": "100.64.1.164",
"srcport": "5520"
}
Config files below:
/etc/fluent-bit/fluent-bit.yaml
service:
flush: 10
daemon: Off
log_level: debug
parsers_file: parsers.yaml
plugins_file: plugins.conf
http_server: Off
http_listen: 0.0.0.0
http_port: 2020
storage.metrics: on
pipeline:
inputs:
- name: syslog
listen: 0.0.0.0
port: 5140
mode: udp
parser: syslog-sonic1
tag: sonicwall
processors:
logs:
- name: lua
match: sonicwall
script: sonic.lua
call: replace_single_quote
- name: parser
match: sonicwall
parser: logfmt
key_name: message
- name: content_modifier
action: extract
key: "src"
pattern: '(?<srcip>.*):(?<srcport>.*)?'
outputs:
- name: cloudwatch_logs
match: 'sonicwall'
region: eu-west-2
log_group_name: sonicwall-fluent-bit
log_stream_name: firewall001-fluent-bit
auto_create_group: true
log_format: json/emf
log_retention_days: 730
EDIT:- BRAINWAVE?! I see whats happening. When the firewall is pumping out ICMP logs, the port is missing from the ip:port combined value, but my regex says the port is an optional named group, so how could this cause this error message, and it it anything to worry about or is it working as designed? Thanks.
Edit2:-
Here is the record that generates the unable to process chunk from the modifier:-
{
"<134>": true,
"id": "firewall001",
"sn": "00401111111",
"time": "2024-12-29 01:15:37",
"fw": "10.11.11.1",
"pri": "6",
"c": "512",
"gcat": "6",
"m": "597",
"srcMac": "06:00:01:15:00:d0",
"src": "10.10.112.20",
"dstMac": "06:00:dc:51:11:ef",
"dst": "10.11.22.33",
"proto": "icmp",
"type": "0",
"rcvd": "84",
"app": "49275",
"appName": "Service Echo",
"msg": "ICMP packet allowed",
"fw_action": "forward"
}
Can anyone smarter than me recommend a regex or a setting for the modifier to relax itself if the two records are unable to be extracted from the one, or do I need to live with these debug messages? Thanks again.