Memory Leak running Fluent-bit v3.2.3

[davidtilloy]

Bug Report

Describe the bug

Running fluent-bit v3.2.3 is consuming all the system memory, ending with a crash of the system at one point (when no more memory).

To Reproduce

• Set up your system with this configuration:

[SERVICE]
  flush            15
  daemon           Off
  log_level        info
  parsers_file     parsers.conf
  parsers_file     parsers-avx.conf
  plugins_file     plugins.conf
  http_server      Off
  http_listen      127.0.0.1
  http_port        2020
  storage.metrics  on

[INPUT]
  Name               systemd
  Tag                node.systemd.*
  Path               /var/log/journal
  DB                 /var/log/flb_systemd.db
  Mem_Buf_Limit      32MB
  Read_From_Tail     On
  Strip_Underscores  On
  Lowercase          On

[INPUT]
  Name              tail
  Tag               nginx.access
  Parser            nginx-avx
  Path              /var/log/nginx/access.log
  DB                /var/log/flb_nginx_access.db
  Mem_Buf_Limit     5MB
  Refresh_Interval  10

[INPUT]
  Name              tail
  Tag               nginx.error
  Parser            nginx_errorlog
  Path              /var/log/nginx/error.log
  DB                /var/log/flb_nginx_error.db
  Mem_Buf_Limit     5MB
  Refresh_Interval  10

[FILTER]
  Name       record_modifier
  Alias      remove_systemd_keys
  Match      node.systemd.*
  Remove_Key cursor
  Remove_Key boot_id
  Remove_Key code_line
  Remove_Key stream_id
  Remove_Key machine_id
  Remove_Key message_id
  Remove_Key realtime_timestamp
  Remove_Key monotonic_timestamp
  Remove_Key source_realtime_timestamp
  Record     log_origin systemd

[FILTER]
  Name    record_modifier
  Alias   filter_product
  Match   *
  Record  node ${HOSTNAME}
  Record  product nginx-s3

[FILTER]
  Name         modify
  Alias        final_format
  Match        *
  Rename       error      short_message
  Rename       log        short_message
  Rename       message    short_message
  Rename       priority   severity
  Add          log_origin nginx
  Add          severity   info
  Remove_regex (process|thread)_id

@INCLUDE output.conf

• Then run fluent-bit with fluent-bit -c fluent-bit.conf
• Wait for a couple of hours (in my case, 1GB is used per hour)
• Then you'll see:

$ ps faux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root        6626  0.1 82.4 130767788 6603148 ?   Ssl  03:26   0:47 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Reverting to version v3.2.2 (same configuration, same OS) fix this issue (~7pm on this graph):

Expected behavior

Memory usage should be stable, fluent-bit should not use all the memory.

Your Environment

• Version used: v3.2.3
• Configuration: See the details in the 'how to reproduce' section
• Environment name and version (e.g. Kubernetes? What version?): 6.8.0-1019-aws
• Server type and version: EC2 Linux/AWS
• Operating System and version: Ubuntu 22.04 / AWS AMI ami-048f97d041d14fd4e
• Filters and plugins: filters: modify and record_modifier

[bpetermannS11]

The leak was introduced with commit 6790b26
We noticed it after an update to v3.2.3 (v3.2.2 was still fine).

The systemd input plugin will leak memory if the lowercase option is on. The temporary buffer to convert a key to the lower-case string is not freed.

Reproducible with the following configuration:

[INPUT]
    name              systemd
    tag               journald
    db                /var/log/fluent-bit/fluent-bit-journal.db
    lowercase         on
    path              /var/log/journal

and some activity in the systemd journal

[gwharton]

Thanks for the swift update. Been struggling with OOM over the last week.

Can you spot when I updated my server to 3.2.4