feat(tasks) Add taskbroker-client metrics for tasks

[markstory]

We want to standardize metrics prefixes and tags for taskworkers. Having standardized metrics makes building dashboards and alerting easier for platform teams, and makes documenting metrics for others simpler.

I've put usage of the new metrics backend behind an env var so that I can incrementally enable and validate the new metrics.

Refs STREAM-1174

[linear-code]

STREAM-1174

[untitaker]

can we try to funnel this through our existing django settings? having the envvar be read in many places is not great

[markstory]

That's a good question. I'm not thrilled about the extra env var for the port either. Sadly our production metrics are all configured through getsentry and not env vars. I'll rework this to work with the production & self-hosted django settings.

[cursor]

Env enabled, metrics silently unchanged

Medium Severity

With USE_TASKWORKER_METRICS=1, if _extract_metrics_config returns a missing host or port (empty options, UDS-only StatsD, dummy backend), the code keeps SentryMetricsBackend and emits no warning. Rollout validation can look enabled while taskworker still uses the old metrics shape.

 

^{Reviewed by Cursor Bugbot for commit 6fc270b. Configure here.}

[markstory]

We'll have options defined in all the environments, and if extracting configuration fails we'll get a log message.

[untitaker]

I regret asking for this. but i think architecturally it's probably still better?

[markstory]

Its a bit gnarly, but better than having more environment variables.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 785bf98. Configure here.}

[cursor]

Host fallback misroutes StatsD

Medium Severity

When USE_TASKWORKER_METRICS is enabled, _extract_metrics_config falls back from statsd_host to host. In DatadogMetricsBackend, host is the metric reporting hostname, not the DogStatsD agent. If statsd_host is omitted but host and statsd_port are set, taskworker DatadogMetrics can send to the wrong address while Sentry’s main metrics backend still uses the default agent.

 

^{Reviewed by Cursor Bugbot for commit 785bf98. Configure here.}

[ceorourke]

fyi I am seeing failures when trying to run any migration commands locally after this was merged, e.g. from sentry django migrate sentry 1234

  File "/Users/ceorourke/code/sentry/src/sentry/eventstream/__init__.py", line 5, in <module>
    from .base import EventStream
  File "/Users/ceorourke/code/sentry/src/sentry/eventstream/base.py", line 9, in <module>
    from sentry.tasks.post_process import post_process_group
  File "/Users/ceorourke/code/sentry/src/sentry/tasks/post_process.py", line 28, in <module>
    from sentry.taskworker.namespaces import ingest_errors_postprocess_tasks
  File "/Users/ceorourke/code/sentry/src/sentry/taskworker/namespaces.py", line 3, in <module>
    from sentry.taskworker.runtime import app
  File "/Users/ceorourke/code/sentry/src/sentry/taskworker/runtime.py", line 7, in <module>
    from taskbroker_client.metrics import DatadogMetrics, MetricsBackend
ImportError: cannot import name 'DatadogMetrics' from 'taskbroker_client.metrics' (/Users/ceorourke/code/sentry/.venv/lib/python3.13/site-packages/taskbroker_client/metrics.py)