Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 43 additions & 1 deletion src/sentry/taskworker/runtime.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import logging
import os

from django.conf import settings
from django.core.cache import cache
from taskbroker_client.app import TaskbrokerApp
from taskbroker_client.metrics import DatadogMetrics, MetricsBackend

from sentry.taskworker.adapters import (
DjangoCacheAtMostOnceStore,
Expand All @@ -10,10 +14,48 @@
make_producer,
)

logger = logging.getLogger(__name__)


def _extract_metrics_config() -> tuple[str | None, int | None]:
host, port = None, None
metric_options = settings.SENTRY_METRICS_OPTIONS
try:
# Use the metrics settings options to infer the host/port.
# The metrics options have different structures depending on which backend is used.
if settings.SENTRY_METRICS_BACKEND == "sentry.metrics.dualwrite.DualWriteMetricsBackend":
metric_options = settings.SENTRY_METRICS_OPTIONS["primary_backend_args"]

# Some backends use `host` and others use `statsd_host`
host = metric_options.get("statsd_host", None) or metric_options.get("host", None)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I regret asking for this. but i think architecturally it's probably still better?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its a bit gnarly, but better than having more environment variables.

raw_port = metric_options.get("statsd_port", None) or metric_options.get("port", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Host fallback misroutes StatsD

Medium Severity

When USE_TASKWORKER_METRICS is enabled, _extract_metrics_config falls back from statsd_host to host. In DatadogMetricsBackend, host is the metric reporting hostname, not the DogStatsD agent. If statsd_host is omitted but host and statsd_port are set, taskworker DatadogMetrics can send to the wrong address while Sentry’s main metrics backend still uses the default agent.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 785bf98. Configure here.

if isinstance(raw_port, (str, int)):
port = int(raw_port)
Comment thread
cursor[bot] marked this conversation as resolved.
except Exception as e:
logger.warning("Could not extract metrics settings", extra={"error": str(e)})
Comment thread
sentry[bot] marked this conversation as resolved.
return host, port


metrics_class: MetricsBackend = SentryMetricsBackend()

if os.getenv("USE_TASKWORKER_METRICS", None) == "1":
host, port = _extract_metrics_config()
if host and port:
Comment thread
cursor[bot] marked this conversation as resolved.
# Metrics created by this interface will not
# have `sentry.` prefix, and will not have
# K8S_LABEL applied.
metrics_class = DatadogMetrics(
application="sentry",
statsd_host=host,
Comment thread
sentry[bot] marked this conversation as resolved.
statsd_port=port,
sample_rate=settings.SENTRY_METRICS_SAMPLE_RATE,
enable_prefixed_metrics=True,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Env enabled, metrics silently unchanged

Medium Severity

With USE_TASKWORKER_METRICS=1, if _extract_metrics_config returns a missing host or port (empty options, UDS-only StatsD, dummy backend), the code keeps SentryMetricsBackend and emits no warning. Rollout validation can look enabled while taskworker still uses the old metrics shape.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 6fc270b. Configure here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have options defined in all the environments, and if extracting configuration fails we'll get a log message.


app = TaskbrokerApp(
name="sentry",
producer_factory=make_producer,
metrics_class=SentryMetricsBackend(),
metrics_class=metrics_class,
router_class=SentryRouter(),
at_most_once_store=DjangoCacheAtMostOnceStore(cache),
context_hooks=[ViewerContextHook()],
Expand Down
Loading