Akshay/datadog dashboard

[Akkikens]

feat(datadog): infrastructure health dashboard and metrics ingestion

implements datadog ingestion pipeline for infrastructure health metrics per plan:

• CI metrics collector (workflow success, duration, flakiness, PR stats)
• training & eval health collectors
• kubernetes CronJob deployment with GitHub Actions CI/CD
• dashboard deployed and showing real metrics: https://p.datadoghq.com/sb/d5c8e3aa-6cc1-11f0-8647-42f10350ee07-c29d3c5ffe74823278e9234cc4729b74

metrics are flowing from my branch and visible in datadog. ready for review!
Asana Task

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[graphite-app]

The training-health-cronjob is missing the command field but includes args. In Kubernetes, when args is specified without command, it replaces the Dockerfile's CMD entirely. This means the pod will try to execute ["collect", "training", "--push"] directly instead of ["python", "-m", "devops.datadog.cli", "collect", "training", "--push"], causing immediate failure.

Fix: Add the command field:

- command:
    - python
    - -m
    - devops.datadog.cli
- args:
    - collect
    - training
    - --push

Suggested change

	- name: training-health-cronjob
	<<: *cronjob_template
	values:
	- schedule: "0 * * * *" # Hourly
	- image:
	tag: latest-dashboard
	- datadog:
	service: training-health-collector
	- args:
	- collect
	- training
	- --push
	- name: training-health-cronjob
	<<: *cronjob_template
	values:
	- schedule: "0 * * * *" # Hourly
	- image:
	tag: latest-dashboard
	- datadog:
	service: training-health-collector
	- command:
	- python
	- -m
	- devops.datadog.cli
	- args:
	- collect
	- training
	- --push

Spotted by Graphite Agent



Is this helpful? React 👍 or 👎 to let us know.

[graphite-app]

Workflow exclusion logic is broken when workflows are configured by ID. The exclude_workflows set contains workflow IDs as strings (e.g., "163901189" from CI_TESTS_BLOCKING_MERGE_WORKFLOWS), but line 326 compares against workflow_name which is the workflow's name string from the API (e.g., "Test and Benchmark"). This mismatch means workflows configured by ID will never be excluded, causing incorrect failing workflow counts.

Fix: Also check against workflow_id:

exclude_workflows = set(self.config.tests_blocking_merge_workflows) | set(self.config.benchmarks_workflows)

failing_count = 0
for workflow in all_workflows:
    workflow_name = workflow.get("name") or workflow.get("path", "").split("/")[-1]
    workflow_id = str(workflow.get("id", ""))

    # Skip if this workflow matches by name OR id
    if workflow_name in exclude_workflows or workflow_id in exclude_workflows:
        continue

Suggested change

	def _count_other_workflows_failing(self) -> int:
	"""Count workflows (excluding tests/benchmarks) whose latest run is failing."""
	all_workflows = self.github.list_workflows(self.config.repo)
	exclude_workflows = set(self.config.tests_blocking_merge_workflows) | set(self.config.benchmarks_workflows)
	failing_count = 0
	for workflow in all_workflows:
	workflow_name = workflow.get("name") or workflow.get("path", "").split("/")[-1]
	workflow_id = workflow.get("id")
	# Skip if this is a test/benchmark workflow
	if workflow_name in exclude_workflows:
	continue
	def _count_other_workflows_failing(self) -> int:
	"""Count workflows (excluding tests/benchmarks) whose latest run is failing."""
	all_workflows = self.github.list_workflows(self.config.repo)
	exclude_workflows = set(self.config.tests_blocking_merge_workflows) | set(self.config.benchmarks_workflows)
	failing_count = 0
	for workflow in all_workflows:
	workflow_name = workflow.get("name") or workflow.get("path", "").split("/")[-1]
	workflow_id = str(workflow.get("id", ""))
	# Skip if this workflow matches by name OR id
	if workflow_name in exclude_workflows or workflow_id in exclude_workflows:
	continue

Spotted by Graphite Agent



Is this helpful? React 👍 or 👎 to let us know.

[Akkikens]

@codex review

[chatgpt-codex-connector]

Codex review is not enabled for this repo. Please contact the admins of this repo to enable Codex.