[BUG] Datadog serverless-init or datadog itself is not collecting all data from json log line

[piotrekkr]

Agent Environment

• serverless-init:1.5.1
• ddtrace==2.17.2
• Python 3.12

Describe what happened:

I'm trying to log simple message with some additional extra data and send it to DataDog. My simple python script is producing json message line containing all the data inside but somehow one of keys in a dictionary from my payload is missing in Datadog UI. It's like it was not send at all but other keys are all there. It is super strange because same payload send as a json string is working just fine. I've created very simple Dockerfile and python script to reproduce this and illustrate whats happening.

Describe what you expected:

I expect all data to be present in DataDog Log UI

Steps to reproduce the issue:

dd-test.py file:

import time
import json

payload = {
    "deliveryLocation": {
        "city": "New York",
    },
    "pickups": [],
}

msg = {
    "message": "Hello, World!",
    "payload": payload,
    "payload_json": json.dumps(payload),
}

while True:
    print(json.dumps(msg))
    time.sleep(20)

Dockerfile:

# syntax=docker/dockerfile:1
FROM python:3.12-slim-bookworm

ENV PYTHONFAULTHANDLER=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

RUN pip install --no-cache-dir --target=/dd_tracer/python/ ddtrace==2.17.2

COPY dd-test.py /app/dd-test.py

ENV DD_VERSION=test \
    DD_LOGS_ENABLED=true \
    DD_TRACE_ENABLED=false \
    DD_TRACE_PROPAGATION_STYLE=datadog \
    DD_APM_ENABLED=false \
    DD_SERVICE=gary \
    DD_SITE=datadoghq.eu

COPY --from=gcr.io/datadoghq/serverless-init:1.5.1 /datadog-init /datadog/datadog-init

ENTRYPOINT ["/datadog/datadog-init"]

CMD ["/dd_tracer/python/bin/ddtrace-run", "python", "/app/dd-test.py"]

Build image:

docker build -t gary:dd-test .

Export DataDog API Key:

export DD_API_KEY=xxxxx

Run script:

docker run -it --rm -e DD_API_KEY -e DD_ENV=staging gary:dd-test

This produces:

$ docker run -it --rm -e DD_API_KEY -e DD_ENV=staging gary:dd-test 
{"message": "Hello, World!", "payload": {"deliveryLocation": {"city": "New York"}, "pickups": []}, "payload_json": "{\"deliveryLocation\": {\"city\": \"New York\"}, \"pickups\": []}"}
{"message": "Hello, World!", "payload": {"deliveryLocation": {"city": "New York"}, "pickups": []}, "payload_json": "{\"deliveryLocation\": {\"city\": \"New York\"}, \"pickups\": []}"}
{"message": "Hello, World!", "payload": {"deliveryLocation": {"city": "New York"}, "pickups": []}, "payload_json": "{\"deliveryLocation\": {\"city\": \"New York\"}, \"pickups\": []}"}
{"message": "Hello, World!", "payload": {"deliveryLocation": {"city": "New York"}, "pickups": []}, "payload_json": "{\"deliveryLocation\": {\"city\": \"New York\"}, \"pickups\": []}"}
^Cobject address  : 0x7adfe165ed40
object refcount : 3
object type     : 0x7adfe1f529c0
object type name: KeyboardInterrupt
object repr     : KeyboardInterrupt()
lost sys.stderr

But when looking at DataDog UI I cannot see pickups: [] at all. It's like pickups were never sent in payload. However payload_json contain same data but encoded as json string and pickups is inside.

I'm not sure what exactly issue is here. Maybe datadog serverless-init is filtering something or maybe there is a bug in Datadog UI that just hide this "pickups": [].

It hit me today and I wasted few hours debugging production issue because I was sure pickups key was never set. In reality it was set but as empty list and just not displayed in datadod UI.

Let me know if you need any more details. Thank you