Calling a content modifier that looks like it works OK, dumps a load of "unable to process chunk" out in the trace/debug.

[RossOH]

Setup as follows.

• Fluentbit latest (3.2.3)
• AWS Linux 2023 latest AMI
• Input is UDP syslog
• Syslog processes the whole line as a single string/key-value called 'message'.
• Lua gets called to remove any of the single quotes in string that Sonicwall handily decided to fire in on random values.
• Lua successfully removes single quotes.
• Log is now able to be Parsed by Logfmt, and it does this perfectly without the single quotes.
• Content modifier called on record to extract ip and port from combined ip:port entry in log.
• Content modifier seems to cause a load of failed to process chunk messages in stdout :(
• Happens only a few seconds after starting fluent-bit process, is not sporadic, loads of entries.

[2024/12/29 00:40:43] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:44] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:44] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:45] [ warn] [processor] failed to process chunk
[2024/12/29 00:40:45] [ warn] [processor] failed to process chunk

No clue what I am doing wrong! This ONLY happens when the content modifier is added. Content modifier MUST be called after LogFmt Parsing, so that it has key-value pairs to process.

Example of log being processed:-

<134> id=firewall000 sn=000000000 time="2024-12-27 10:41:29" fw=10.11.112.4 pri=6 c=1024 gcat=6 m=537 src=100.64.2.172:19690 dst=10.11.11.11:443 proto=tcp/https sent=216 rcvd=164 cdur=2016 app=49177 appName='General HTTPS' msg="Connection Closed" fw_action="NA"

Example of final log result in Cloudwatch:-

{
"<134>": true,
"id": "firewall000",
"sn": "0000000000",
"time": "2024-12-29 00:52:06",
"fw": "10.10.12.4",
"pri": "6",
"c": "1024",
"gcat": "6",
"m": "537",
"src": "100.64.1.164:5520",
"dst": "10.10.10.148:8444",
"proto": "tcp/8444",
"sent": "240",
"cdur": "38016",
"app": "49201",
"appName": "General TCP",
"msg": "Connection Closed",
"fw_action": "NA",
"srcip": "100.64.1.164",
"srcport": "5520"
}

Config files below:

 /etc/fluent-bit/fluent-bit.yaml
service:
  flush: 10
  daemon: Off
  log_level: debug
  parsers_file: parsers.yaml
  plugins_file: plugins.conf
  http_server: Off
  http_listen: 0.0.0.0
  http_port: 2020
  storage.metrics: on

pipeline:
  inputs:
    - name: syslog
      listen: 0.0.0.0
      port: 5140
      mode: udp
      parser: syslog-sonic1
      tag: sonicwall
      processors:
        logs:
          - name: lua
            match: sonicwall
            script: sonic.lua
            call: replace_single_quote

          - name: parser
            match: sonicwall
            parser: logfmt
            key_name: message

          - name: content_modifier
            action: extract
            key: "src"
            pattern: '(?<srcip>.*):(?<srcport>.*)?'

  outputs:
    - name: cloudwatch_logs
      match: 'sonicwall'
      region: eu-west-2
      log_group_name: sonicwall-fluent-bit
      log_stream_name: firewall001-fluent-bit
      auto_create_group: true
      log_format: json/emf
      log_retention_days: 730

EDIT:- BRAINWAVE?! I see whats happening. When the firewall is pumping out ICMP logs, the port is missing from the ip:port combined value, but my regex says the port is an optional named group, so how could this cause this error message, and it it anything to worry about or is it working as designed? Thanks.

Edit2:-

Here is the record that generates the unable to process chunk from the modifier:-
{
"<134>": true,
"id": "firewall001",
"sn": "00401111111",
"time": "2024-12-29 01:15:37",
"fw": "10.11.11.1",
"pri": "6",
"c": "512",
"gcat": "6",
"m": "597",
"srcMac": "06:00:01:15:00:d0",
"src": "10.10.112.20",
"dstMac": "06:00:dc:51:11:ef",
"dst": "10.11.22.33",
"proto": "icmp",
"type": "0",
"rcvd": "84",
"app": "49275",
"appName": "Service Echo",
"msg": "ICMP packet allowed",
"fw_action": "forward"
}

Can anyone smarter than me recommend a regex or a setting for the modifier to relax itself if the two records are unable to be extracted from the one, or do I need to live with these debug messages? Thanks again.

[edsiper]

@RossOH pls format the content of the config file, so it becomes easier for us to troubleshoot, thanks

[Mohamezezo]

Format