Skip to content

Commit fcacd83

Browse files
hooliohhoolioh
authored
chore: add logging for the native writer (#14403)
## TLDR This PR enables logging in the native writer. ## Description This PR add bindings for datadog-log crate which can enable logging in the native writer. I currently supports the following backends: - stdout: logs to the standard out file descriptor. - stderr: logs to the standard error file descriptor. - file: logs to a file configured as follows - path: file location. - max_files: configure the number of files being rotated. - max_size_bytes: configure the maximum length of the log file. ## Motivation This PR will allow to monitor `NativeWriter` bugs during the dogfooding phase. ## Testing Unit tests have added to the tracer suite. Microbenchmarks: - djangosimple-tracer-native: increased max_rss to 68MB. With the latest additions the reported size was slightly above 66MB. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
1 parent 5b4a26c commit fcacd83

File tree

9 files changed

+442
-35
lines changed

9 files changed

+442
-35
lines changed

‎.gitlab/benchmarks/bp-runner.microbenchmarks.fail-on-breach.yml‎

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -36,73 +36,73 @@ experiments:
3636
- name: djangosimple-appsec
3737
thresholds:
3838
- execution_time < 22.30 ms
39-
- max_rss_usage < 66.00 MB
39+
- max_rss_usage < 67.00 MB
4040
- name: djangosimple-exception-replay-enabled
4141
thresholds:
4242
- execution_time < 1.45 ms
43-
- max_rss_usage < 66.00 MB
43+
- max_rss_usage < 67.00 MB
4444
- name: djangosimple-iast
4545
thresholds:
4646
- execution_time < 22.25 ms
47-
- max_rss_usage < 66.00 MB
47+
- max_rss_usage < 67.00 MB
4848
- name: djangosimple-profiler
4949
thresholds:
5050
- execution_time < 16.55 ms
51-
- max_rss_usage < 53.50 MB
51+
- max_rss_usage < 54.50 MB
5252
- name: djangosimple-span-code-origin
5353
thresholds:
5454
- execution_time < 28.20 ms
55-
- max_rss_usage < 68.50 MB
55+
- max_rss_usage < 69.50 MB
5656
- name: djangosimple-tracer
5757
thresholds:
5858
- execution_time < 21.75 ms
59-
- max_rss_usage < 66.00 MB
59+
- max_rss_usage < 67.00 MB
6060
- name: djangosimple-tracer-minimal
6161
thresholds:
6262
- execution_time < 17.50 ms
6363
- max_rss_usage < 66.00 MB
6464
- name: djangosimple-tracer-native
6565
thresholds:
6666
- execution_time < 21.75 ms
67-
- max_rss_usage < 66.00 MB
67+
- max_rss_usage < 72.50 MB
6868
- name: djangosimple-tracer-and-profiler
6969
thresholds:
7070
- execution_time < 23.50 ms
71-
- max_rss_usage < 67.00 MB
71+
- max_rss_usage < 67.50 MB
7272
- name: djangosimple-tracer-no-caches
7373
thresholds:
7474
- execution_time < 19.65 ms
75-
- max_rss_usage < 66.00 MB
75+
- max_rss_usage < 67.00 MB
7676
- name: djangosimple-tracer-no-databases
7777
thresholds:
7878
- execution_time < 20.10 ms
79-
- max_rss_usage < 66.00 MB
79+
- max_rss_usage < 67.00 MB
8080
- name: djangosimple-tracer-dont-create-db-spans
8181
thresholds:
8282
- execution_time < 21.50 ms
8383
- max_rss_usage < 66.00 MB
8484
- name: djangosimple-tracer-no-middleware
8585
thresholds:
8686
- execution_time < 21.50 ms
87-
- max_rss_usage < 66.00 MB
87+
- max_rss_usage < 67.00 MB
8888
- name: djangosimple-tracer-no-templates
8989
thresholds:
9090
- execution_time < 22.00 ms
91-
- max_rss_usage < 66.00 MB
91+
- max_rss_usage < 67.00 MB
9292

9393
# errortrackingdjangosimple
9494
- name: errortrackingdjangosimple-errortracking-enabled-all
9595
thresholds:
9696
- execution_time < 19.85 ms
97-
- max_rss_usage < 65.50 MB
97+
- max_rss_usage < 66.50 MB
9898
- name: errortrackingdjangosimple-errortracking-enabled-user
9999
thresholds:
100100
- execution_time < 19.40 ms
101-
- max_rss_usage < 65.50 MB
101+
- max_rss_usage < 66.50 MB
102102
- name: errortrackingdjangosimple-tracer-enabled
103103
thresholds:
104104
- execution_time < 19.45 ms
105-
- max_rss_usage < 65.50 MB
105+
- max_rss_usage < 66.50 MB
106106

107107
# errortrackingflasksqli
108108
- name: errortrackingflasksqli-errortracking-enabled-all
@@ -126,32 +126,32 @@ experiments:
126126
- name: flasksimple-tracer-native
127127
thresholds:
128128
- execution_time < 3.65 ms
129-
- max_rss_usage < 53.50 MB
129+
- max_rss_usage < 60.00 MB
130130
- name: flasksimple-profiler
131131
thresholds:
132132
- execution_time < 2.10 ms
133-
- max_rss_usage < 46.50 MB
133+
- max_rss_usage < 47.00 MB
134134
- name: flasksimple-debugger
135135
thresholds:
136136
- execution_time < 2.00 ms
137-
- max_rss_usage < 46.50 MB
138-
- max_rss_usage < 45.00 MB
137+
- max_rss_usage < 47.00 MB
138+
- max_rss_usage < 47.00 MB
139139
- name: flasksimple-iast-get
140140
thresholds:
141141
- execution_time < 2.00 ms
142142
- max_rss_usage < 49.00 MB
143143
- name: flasksimple-appsec-get
144144
thresholds:
145145
- execution_time < 4.75 ms
146-
- max_rss_usage < 64.50 MB
146+
- max_rss_usage < 65.00 MB
147147
- name: flasksimple-appsec-post
148148
thresholds:
149149
- execution_time < 6.75 ms
150-
- max_rss_usage < 64.50 MB
150+
- max_rss_usage < 65.00 MB
151151
- name: flasksimple-appsec-telemetry
152152
thresholds:
153153
- execution_time < 4.75 ms
154-
- max_rss_usage < 64.50 MB
154+
- max_rss_usage < 65.00 MB
155155

156156
# flasksqli
157157
- name: flasksqli-appsec-enabled
@@ -161,11 +161,11 @@ experiments:
161161
- name: flasksqli-iast-enabled
162162
thresholds:
163163
- execution_time < 2.80 ms
164-
- max_rss_usage < 59.00 MB
164+
- max_rss_usage < 60.00 MB
165165
- name: flasksqli-tracer-enabled
166166
thresholds:
167167
- execution_time < 2.25 ms
168-
- max_rss_usage < 53.50 MB
168+
- max_rss_usage < 54.50 MB
169169

170170
# httppropagationextract
171171
- name: httppropagationextract-all_styles_all_headers
@@ -791,7 +791,7 @@ experiments:
791791
- name: otelspan-add-event
792792
thresholds:
793793
- execution_time < 47.15 ms
794-
- max_rss_usage < 46.50 MB
794+
- max_rss_usage < 47.00 MB
795795
- name: otelspan-add-metrics
796796
thresholds:
797797
- execution_time < 344.80 ms
@@ -807,19 +807,19 @@ experiments:
807807
- name: otelspan-is-recording
808808
thresholds:
809809
- execution_time < 44.50 ms
810-
- max_rss_usage < 46.50 MB
810+
- max_rss_usage < 47.50 MB
811811
- name: otelspan-record-exception
812812
thresholds:
813813
- execution_time < 67.65 ms
814-
- max_rss_usage < 46.50 MB
814+
- max_rss_usage < 47.00 MB
815815
- name: otelspan-set-status
816816
thresholds:
817817
- execution_time < 50.40 ms
818-
- max_rss_usage < 46.50 MB
818+
- max_rss_usage < 47.00 MB
819819
- name: otelspan-start
820820
thresholds:
821821
- execution_time < 43.45 ms
822-
- max_rss_usage < 46.50 MB
822+
- max_rss_usage < 47.00 MB
823823
- name: otelspan-start-finish
824824
thresholds:
825825
- execution_time < 88.00 ms
@@ -831,7 +831,7 @@ experiments:
831831
- name: otelspan-update-name
832832
thresholds:
833833
- execution_time < 45.15 ms
834-
- max_rss_usage < 46.50 MB
834+
- max_rss_usage < 47.00 MB
835835

836836
# packagespackageforrootmodulemapping
837837
- name: packagespackageforrootmodulemapping-cache_off

‎ddtrace/_logger.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,13 @@
22
from os import path
33
from typing import Optional
44

5+
from ddtrace.internal.logger import get_logger
56
from ddtrace.internal.telemetry import get_config
67
from ddtrace.internal.utils.formats import asbool
78

89

10+
log = get_logger(__name__)
11+
912
DD_LOG_FORMAT = "%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] {}- %(message)s".format(
1013
"[dd.service=%(dd.service)s dd.env=%(dd.env)s dd.version=%(dd.version)s"
1114
" dd.trace_id=%(dd.trace_id)s dd.span_id=%(dd.span_id)s] "
@@ -53,6 +56,8 @@ def configure_ddtrace_logger():
5356

5457
_configure_ddtrace_debug_logger(ddtrace_logger)
5558
_configure_ddtrace_file_logger(ddtrace_logger)
59+
# Calling _configure_ddtrace_native_logger should come after Python logging has been configured.
60+
_configure_ddtrace_native_logger()
5661

5762

5863
def _configure_ddtrace_debug_logger(logger):
@@ -125,3 +130,24 @@ def get_log_injection_state(raw_config: Optional[str]) -> bool:
125130
normalized,
126131
)
127132
return False
133+
134+
135+
def _configure_ddtrace_native_logger():
136+
try:
137+
from ddtrace.internal.native._native import logger
138+
139+
native_writer_enabled = get_config("_DD_TRACE_WRITER_NATIVE", False, asbool, report_telemetry=True)
140+
if native_writer_enabled:
141+
backend = get_config("_DD_NATIVE_LOGGING_BACKEND", "file", report_telemetry=True)
142+
kwargs = {"output": backend}
143+
if backend == "file":
144+
kwargs["path"] = get_config("_DD_NATIVE_LOGGING_FILE_PATH", "native.log", report_telemetry=True)
145+
kwargs["max_size_bytes"] = get_config(
146+
"_DD_NATIVE_LOGGING_FILE_SIZE_BYTES", 4096, int, report_telemetry=True
147+
)
148+
kwargs["max_files"] = get_config("_DD_NATIVE_LOGGING_FILE_ROTATION_LEN", 1, int, report_telemetry=True)
149+
150+
logger.configure(**kwargs)
151+
logger.set_log_level(get_config("_DD_NATIVE_LOGGING_LOG_LEVEL", "warn", report_telemetry=True))
152+
except Exception:
153+
log.warning("Failed to initialize native logger", exc_info=True)

‎ddtrace/internal/native/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
from ._native import SerializationError # noqa: F401
1616
from ._native import TraceExporter # noqa: F401
1717
from ._native import TraceExporterBuilder # noqa: F401
18+
from ._native import logger # noqa: F401
1819
from ._native import store_metadata # noqa: F401
1920

2021

‎ddtrace/internal/native/_native.pyi‎

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from typing import Dict, List, Optional
1+
from typing import Dict, List, Literal, Optional
22

33
class DDSketch:
44
def __init__(self): ...
@@ -346,6 +346,57 @@ class BuilderError(Exception):
346346

347347
...
348348

349+
class logger:
350+
"""
351+
Native logging module for configuring and managing log output.
352+
"""
353+
354+
@staticmethod
355+
def configure(
356+
output: Literal["stdout", "stderr", "file"] = "stdout",
357+
path: Optional[str] = None,
358+
max_files: Optional[int] = None,
359+
max_size_bytes: Optional[int] = None,
360+
) -> None:
361+
"""
362+
Configure the logger with the specified output destination.
363+
364+
:param output: Output destination ("stdout", "stderr", or "file")
365+
:param path: File path (required if output is "file")
366+
:param max_files: Maximum number of log files to keep (for file output)
367+
:param max_size_bytes: Maximum size of each log file in bytes (for file output)
368+
:raises ValueError: If configuration is invalid
369+
"""
370+
...
371+
@staticmethod
372+
def disable(output: str) -> None:
373+
"""
374+
Disable logging output by type.
375+
376+
:param output: Output type to disable ("file", "stdout", or "stderr")
377+
:raises ValueError: If output type is invalid
378+
"""
379+
...
380+
@staticmethod
381+
def set_log_level(level: str) -> None:
382+
"""
383+
Set the log level for the logger.
384+
385+
:param level: Log level ("trace", "debug", "info", "warn", or "error")
386+
:raises ValueError: If log level is invalid
387+
"""
388+
...
389+
@staticmethod
390+
def log(level: str, message: str) -> None:
391+
"""
392+
Logs messages
393+
394+
:param level: Log level ("trace", "debug", "info", "warn", or "error")
395+
:param message: message to be displayed in the log.
396+
:raises ValueError: If log level is invalid
397+
"""
398+
...
399+
349400
class DeserializationError(Exception):
350401
"""
351402
Raised when there is an error deserializing trace payload.

0 commit comments

Comments
 (0)