document multiline parsing by docker parser

[amfleurke]

Bug Report

Describe the bug
I'm running applications in k8s, that log e.g. java stacktraces on multiple lines.
The documentation says you can use the tail parser with 'multiline.parser' set to 'docker'.
But what does the 'docker' parser do with regards to multi lines? What does it consider a multi-line log?

when searching on the internet, I see other people adding an additional 'multiline' filter. But the manual says:

If you wish to concatenate messages read from a log file, it is highly recommended to use the multiline support in the Tail plugin itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the Tail plugin.

So to me this seems like adding filters is not the way to go.

Expected behavior
Clear instructions how to set up multi-line parsing in combination with the docker parser and/or
clear description on what lines the docker parser can join, and when to use multiline filters instead and/or
clear description of the difference between docker/cri multiline capabilities and other multiline parsers/filters

[patrick-stephens]

It would be good to submit a PR to improve the documentation.

Both CRI and docker have defined approaches (I'm not going to say standards) for how they split lines. It's nothing to do with your application format, it's to do with what the runtime itself does to split lines. These parsers then reconstruct the log entry if it is split.

Now, if you send multiple logs entries to the runtime that should be then made into a single one you would need an additional parser on top.

The actual code includes the actual handling for the source of truth: https://github.com/fluent/fluent-bit/blob/master/src%2Fmultiline%2Fflb_ml_parser_docker.c
This includes a load of comments as well it would be useful to expose.

I agree the docs could be improved and I keep meaning to after doing various posts or webinars on it but would appreciate a contribution as well. Good worked examples are great.

[amfleurke]

Thanks for clarifying it a bit. Then the next question is: what is the cri/docker not doing, for the multiline filter to have a
mode=partial_message
config option?
ref: https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace

[amfleurke]

oh, that's explained a bit lower.

what is also confusing is the conf/parsers_java.conf with a parser called 'java_multiline' but it's not a multiline_parser. Idem the 'couchbase_java_multiline' and 'couchbase_erlang_multiline' parser in conf/parsers_extra.conf. Old style multi-line parsers? I think they should be removed.

Is the documentation in github too? Then I can make a patch.

my suggestions:

https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace

"Concatenating messages originally split by Docker or CRI container engines, is supported in the Tail plugin."
-->change to -->
"Concatenating messages that were originally one line, but splitby Docker or CRI container engines because of their size, is supported in the Tail plugin icw the docker or cri parser. To concatenate application logs like stacktraces on top of that, you can use this multiline filter."

"If your application emits a 100K log line, it will be split into 7 partial messages. If you are using the Fluentd Docker Log Driver to send the logs to Fluent Bit..."
-->
"If your application emits a 100K log line, it will be split into 7 partial messages. The docker parser will merge these back to one line. If instead you are using the Fluentd Docker Log Driver to send the logs to Fluent Bit..."

https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing

"This parser supports the concatenation of log entries split by Docker."
-->
"This parser supports the concatenation of large log entries split by Docker. If you use this parser, and you also want to concatenate loglines like stacktraces, you can add the multiline filter" to specify additional parsers."

https://docs.fluentbit.io/manual/pipeline/inputs/tail
"This will help to reassembly multiline messages originally split by Docker or CRI:"
-->
"This will help to reassemble large messages originally split by Docker or CRI:"

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[amfleurke]

unstale

[patrick-stephens]

Yeah a lot of the parsers provided were made prior to multiline_parser existing. I wrote those Couchbase ones for 1.9 when I worked there for example!
https://www.couchbase.com/blog/fluent-bit-tips-tricks-log-forwarding-couchbase/

Please send your suggestions as it's nice and easy to review a PR then merge :)

[amfleurke]

For the documentation: where can I find the source for a PR?

[amfleurke]

oh, found it at https://github.com/fluent/fluent-bit-docs