Description
I seem to have run into some incompatibility between a few Elastic efforts that I'm currently stuck on due to the use of dot notation for log.level
in this library. My end goal is for all ingested logs to be in the nested notation, per a comment from @felixbarny in this Community thread and personal preference. However I think I am blocked in my particular situation:
-
ecs-pino-format is used to add a foundation of ECS fields
-
elastic-serverless-forwarder is used to collect Lambda logs via AWS Kinesis
-
Elasticsearch Ingest Pipelines are used to remove some wrapping data structures around my original application logs (two layers, from Kinesis and CloudWatch)
The blocker I've run into is an inability to access the Pino formatter's log.level
in the pipeline because it uses dot notation. Docs indicate that I should use the dot_expander
processor first. However, it is limited itself to accessing top-level leaf fields. In my case I am parsing an incoming message
into a temporary top-level object in order to selectively pick data from it to merge into the root, using the set
processor.
If I try to expand log.level
as a child of the temporary object:
{
"dot_expander": {
"field": "parsed_app_log",
"path": "log.level"
}
}
...I get a pipeline validation exception:
[field] field does not contain a dot and is not a wildcard
If I try to add the parsed JSON to the root I also get an error:
{
"type": "illegal_argument_exception",
"reason": "cannot add non-map fields to root of document"
}
I may just need to reach out to support to understand this error that could have a straightforward explanation I'm not getting from the error itself.
But while I pursue that route I wanted to ask why log.level
uses dot notation here in the first place, when log { logger }
is added if the APM integration is enabled? My understanding is that level
also belongs to the log
ECS field. I assume this is for backwards compatibility reasons.
Also, if log.level
and log { logger }
do coexist at ingestion time, what exactly is the consequence of mixing notations? Why is @felixbarny advocating nested notation when Elastic's own libraries mix notations? Are users expected to use ingest pipelines as I am to normalize their idiosyncrasies?