Add metrics about exponential backoff

[whywaita]

This pull request introduces new Prometheus metrics to improve observability for retry and backoff behaviors in both runner deletion and instance addition workflows. Metrics are now recorded for the number of retries and the duration of exponential backoff in each process, allowing for better monitoring and troubleshooting.

Metrics instrumentation for runner deletion (pkg/runner):

• Added DeleteRunnerBackoffDuration histogram and DeleteRunnerRetryTotal counter to metrics.go to track backoff duration and retry counts when deleting runners, labeled by runner_uuid.
• Updated removeRunners logic in runner_delete.go to increment retry count and observe backoff duration in Prometheus metrics whenever a retry occurs.

Metrics instrumentation for instance addition (pkg/starter):

• Added AddInstanceBackoffDuration histogram and AddInstanceRetryTotal counter to metrics.go to track backoff duration and retry counts when adding instances, labeled by job_uuid.
• Updated run logic in starter.go to increment retry count and observe backoff duration in Prometheus metrics whenever a retry occurs.

[Copilot]

Pull Request Overview

This PR adds Prometheus metrics to improve observability for exponential backoff behavior in both runner deletion and instance addition workflows. The changes enable monitoring of retry counts and backoff durations to help with troubleshooting and performance analysis.

• Added histogram metrics for exponential backoff duration tracking
• Added counter metrics for retry count tracking
• Instrumented retry logic in both starter and runner modules with metrics recording

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pkg/starter/metrics.go	Defines Prometheus metrics for instance addition backoff duration and retry counts
pkg/starter/starter.go	Instruments retry logic to record metrics when retries occur during instance addition
pkg/runner/metrics.go	Defines Prometheus metrics for runner deletion backoff duration and retry counts
pkg/runner/runner_delete.go	Instruments retry logic to record metrics when retries occur during runner deletion

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[site0801]

LGTM