Skip to content

input: update mem buf limit logging to include size info #10578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nuclearpidgeon
Copy link
Contributor

@nuclearpidgeon nuclearpidgeon commented Jul 10, 2025

Addresses #10384

Example of new output

> bin/fluent-bit -i tail -p path=./logslurp -p mem_buf_limit=1m -o file -m '*' -H
Fluent Bit v4.0.4
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/07/11 08:47:05] [ info] [fluent bit] version=4.0.4, commit=a4e3d2b20d, pid=71384
[2025/07/11 08:47:05] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/07/11 08:47:05] [ info] [simd    ] disabled
[2025/07/11 08:47:05] [ info] [cmetrics] version=1.0.3
[2025/07/11 08:47:05] [ info] [ctraces ] version=0.6.6
[2025/07/11 08:47:05] [ info] [input:tail:tail.0] initializing
[2025/07/11 08:47:05] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/07/11 08:47:05] [ info] [output:file:file.0] worker #0 started
[2025/07/11 08:47:05] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/07/11 08:47:05] [ info] [sp] stream processor started
[2025/07/11 08:47:05] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/07/11 08:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=8005407 watch_fd=1 name=./logslurp

Dump some contents into ./logslurp by running this a few times:

cat /dev/random | xxd | dd of=./logslurp bs=1M count=1024

and we see this:

[2025/07/11 08:47:47] [ warn] [input] tail.0 paused (mem buf overlimit - event of size 49543B exceeded limit 1000000 to 1040403B)
[2025/07/11 08:47:47] [ info] [input] pausing tail.0
[2025/07/11 08:47:47] [ info] [input] resume tail.0
[2025/07/11 08:47:47] [ info] [input] tail.0 resume (mem buf overlimit - buf size 0B now below limit 1000000B)
[2025/07/11 08:47:47] [ warn] [input] tail.0 paused (mem buf overlimit - event of size 49543B exceeded limit 1000000 to 1040403B)
[2025/07/11 08:47:47] [ info] [input] pausing tail.0
[2025/07/11 08:47:48] [ info] [input] resume tail.0
[2025/07/11 08:47:48] [ info] [input] tail.0 resume (mem buf overlimit - buf size 0B now below limit 1000000B)
[2025/07/11 08:47:48] [ warn] [input] tail.0 paused (mem buf overlimit - event of size 49543B exceeded limit 1000000 to 1040403B)
[2025/07/11 08:47:48] [ info] [input] pausing tail.0
[2025/07/11 08:47:49] [ info] [input] resume tail.0
[2025/07/11 08:47:49] [ info] [input] tail.0 resume (mem buf overlimit - buf size 0B now below limit 1000000B)
[2025/07/11 08:47:49] [ warn] [input] tail.0 paused (mem buf overlimit - event of size 49543B exceeded limit 1000000 to 1040403B)
[2025/07/11 08:47:49] [ info] [input] pausing tail.0
[2025/07/11 08:47:50] [ info] [input] resume tail.0
[2025/07/11 08:47:50] [ info] [input] tail.0 resume (mem buf overlimit - buf size 0B now below limit 1000000B)
[2025/07/11 08:47:50] [ warn] [input] tail.0 paused (mem buf overlimit - event of size 49543B exceeded limit 1000000 to 1040403B)
[2025/07/11 08:47:50] [ info] [input] pausing tail.0
[2025/07/11 08:47:51] [ info] [input] resume tail.0
[2025/07/11 08:47:51] [ info] [input] tail.0 resume (mem buf overlimit - buf size 0B now below limit 1000000B)
[2025/07/11 08:47:51] [ warn] [input] tail.0 paused (mem buf overlimit - event of size 49543B exceeded limit 1000000 to 1040403B)
[2025/07/11 08:47:51] [ info] [input] pausing tail.0
[2025/07/11 08:47:52] [ info] [input] resume tail.0
[2025/07/11 08:47:52] [ info] [input] tail.0 resume (mem buf overlimit - buf size 0B now below limit 1000000B)

Backporting
It would be nice to backport this to latest stable release if it's not too major a change.


Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@nuclearpidgeon
Copy link
Contributor Author

cc: @patrick-stephens

@nuclearpidgeon
Copy link
Contributor Author

Thanks Hiroshi

@cosmo0920 cosmo0920 added this to the Fluent Bit v4.1 milestone Jul 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants