Skip to content

Separation of log.level from log { logger } #140

Closed
@pushred

Description

@pushred

I seem to have run into some incompatibility between a few Elastic efforts that I'm currently stuck on due to the use of dot notation for log.level in this library. My end goal is for all ingested logs to be in the nested notation, per a comment from @felixbarny in this Community thread and personal preference. However I think I am blocked in my particular situation:

  • ecs-pino-format is used to add a foundation of ECS fields

  • elastic-serverless-forwarder is used to collect Lambda logs via AWS Kinesis

  • Elasticsearch Ingest Pipelines are used to remove some wrapping data structures around my original application logs (two layers, from Kinesis and CloudWatch)

The blocker I've run into is an inability to access the Pino formatter's log.level in the pipeline because it uses dot notation. Docs indicate that I should use the dot_expander processor first. However, it is limited itself to accessing top-level leaf fields. In my case I am parsing an incoming message into a temporary top-level object in order to selectively pick data from it to merge into the root, using the set processor.

If I try to expand log.level as a child of the temporary object:

{
  "dot_expander": {
    "field": "parsed_app_log",
    "path": "log.level"
  }
}

...I get a pipeline validation exception:

[field] field does not contain a dot and is not a wildcard

If I try to add the parsed JSON to the root I also get an error:

{
  "type": "illegal_argument_exception",
  "reason": "cannot add non-map fields to root of document"
}

I may just need to reach out to support to understand this error that could have a straightforward explanation I'm not getting from the error itself.

But while I pursue that route I wanted to ask why log.level uses dot notation here in the first place, when log { logger } is added if the APM integration is enabled? My understanding is that level also belongs to the log ECS field. I assume this is for backwards compatibility reasons.

Also, if log.level and log { logger } do coexist at ingestion time, what exactly is the consequence of mixing notations? Why is @felixbarny advocating nested notation when Elastic's own libraries mix notations? Are users expected to use ingest pipelines as I am to normalize their idiosyncrasies?

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-nodejsMake available for APM Agents project planning.community

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions