[twpsfirmament]

Tracking issue

Closes #8191

Why are the changes needed?

Nothing reports how many runs are currently executing.

What changes were proposed in this pull request?

• Add a runs.active observable gauge in the runs repository: root actions in
  RUNNING, counted from the DB on each collection (5s timeout). Registered with
  RegisterCallback like taskaction.active; skipped for nil/noop providers.
• Add an "Active runs (RUNNING)" panel to flyte-execution.json, using max()
  because every replica reports the same DB-wide count.
• Report the RUNNING run count to Sentry every 60s via
  sentryutils.Gauge("flyte.active_runs"). The worker is registered inside the
  existing sentryutils.Init block, so it does not run when telemetry is off
  (DO_NOT_TRACK).

Note: both are sampled (per OTel export / every 60s), so a run shorter than the
interval can be missed.

How was this patch tested?

• TestActiveRunsGauge: counts RUNNING runs; ignores QUEUED, SUCCEEDED and RUNNING
  child actions. TestRegisterActiveRunsGaugeNoop: nil/noop provider is a no-op.
  go test ./runs/repository/impl/ passes.
• TestActiveRunsReporter_*: reports the count as flyte.active_runs; skips
  reporting when the query fails. Not exercised against a real Sentry project.
  go test ./runs/... passes.

Labels

• added

Setup process

devbox with FLYTE_DEV=True + make devbox-monitoring, local manager with an otel
otlpgrpc exporter (port-forward the otel-collector service on 4317). See screenshot.

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[cloud-verifier-agent]

cloud-verifier · ⚠️ 1 finding, not blocking

🟠 runs.active count scans all RUNNING child actions on every export · runs/repository/impl/action.go:108 · medium · medium likelihood · see inline comment

_{Reviewed 8df03fc0cd · turn 1 · re-run with /cloud-verifier review}

[cloud-verifier-agent]

🟠 runs.active count scans all RUNNING child actions on every export

This count scans all RUNNING actions, including child actions, and every replica repeats it on each export. Consider a partial index on root actions by phase, or caching the count.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[cloud-verifier-agent]

cloud-verifier · ⚠️ 1 finding, not blocking

🟠 runs.active count scans all RUNNING child actions on every poll · runs/repository/impl/action.go:88 · medium · medium likelihood
The count query can only use the generic phase index, so it visits every RUNNING action, including child actions. It now runs from two pollers on every replica (metric export and a periodic reporter). Add a partial index on root actions by phase, or cache and share the count.

_{Reviewed 84250d1b6d · turn 2 · re-run with /cloud-verifier review}