APM App: Correlate spans to logs

[toby-sutor]

Describe the feature:
Similar to https://github.com/elastic/apm-dev/issues/864 and in relation to elastic/apm-data#40 it would be great if the Kibana UI for APM would provide the option to correlate the logs to a span. For example, the UI could add a Logs button as per the screenshot below:

The logs are already there and usually contain the span.id. As such, it should be possible to create this link in the UI for correlation.

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[gbamparop]

@elastic/apm-agent-devs, is span.id available in logs? It's not mentioned in the logs correlations docs (Java, .NET).

[gregkalapos]

@elastic/apm-agent-devs, is span.id available in logs? It's not mentioned in the logs correlations docs (Java, .NET).

.NET adds it.

[Mpdreamz]

.NET does https://github.com/elastic/ecs-dotnet/blob/main/src/Elastic.CommonSchema/EcsDocument.cs#L183

It's not a part of the ecs json logging spec yet though:

https://github.com/elastic/ecs-logging/blob/main/spec/spec.json

We should look into adding it to the spec and all implementations IMO.

[basepi]

Python adds span.id.

[trentm]

Node.js adds either transaction.id or span.id, depending on if the current active thing is a transaction or a span. (https://www.elastic.co/guide/en/apm/agent/nodejs/3.x/log-correlation.html)

Personally I think that is a little weird. I wonder if with the ECS/OTel-semconv work if it things will settle on just setting span.id (using that field name, even if the current thing is a transaction). I don't know anything authoritative, however.

[trentm]

Node.js adds either transaction.id or span.id

@gbamparop (or others on the APM UI team), do you happen to know off hand if this means that any log records produced by ecs-logging-nodejs libraries that have only trace.id and span.id (but not transaction.id) will NOT be shown in the Logs view? I.e. is the current ecs-logging-nodejs behaviour a bug?

[AlexanderWert]

OTel agents also do add span.id to logs

[sorenlouv]

Node.js adds either transaction.id or span.id

@gbamparop (or others on the APM UI team), do you happen to know off hand if this means that any log records produced by ecs-logging-nodejs libraries that have only trace.id and span.id (but not transaction.id) will NOT be shown in the Logs view? I.e. is the current ecs-logging-nodejs behaviour a bug?

@trentm We don't use span.id or transaction.id for log correlation yet. For service logs we use service.name and service.environment (+fallback to infrastructure fields). For trace logs we use trace.id.

Service logs
https://github.com/elastic/kibana/blob/7e7cf0fb96b6b138fe5832f6af86727bbdb4c989/x-pack/plugins/apm/public/components/app/service_logs/index.tsx#L68-L96

Trace logs

kibana/x-pack/plugins/apm/public/components/app/transaction_details/waterfall_with_summary/transaction_tabs.tsx

Line 177 in 7e7cf0f

query={`trace.id:"${traceId}" OR (not trace.id:* AND "${traceId}")`}

[SylvainJuge]

In Java, the logs contain trace.id and transaction.id, but NOT span.id, and that would be something relevant to add to all agents. Whatever the trace/transaction/span is currently active when a log message is written should be correlated.

The actual value we might put in span.id for the log events would depend on the strategy we pick for aligning with otel.

For example, if we add span.id with the same value as transaction.id in transactions and log events (both in correlated logs and in storage), then we could always rely on the span.id for log correlation on an individual span or transaction. This would also allow to define parent/child relationship by querying only on the parent.id = span.id unlike right now where we need to know if it's a transaction or a span.
Said it differently, it would just make the transactions a "special span" that has a transaction ID in addition to the span ID.

[SylvainJuge]

I've just opened elastic/apm#810 to properly capture span.id consistently in APM agents.