Intermittent 401 Unauthorized for Google Managed Prometheus Datasource (Grafana Cloud v12.4.0)

[aaabramov]

Starting on Jan 26, 2026, our Google Managed Prometheus (GMP) datasource in Grafana Cloud began experiencing intermittent 401 Unauthorized errors with the message ACCESS_TOKEN_EXPIRED.

We use the standard gmp-datasource-syncer (v0.17.3) running as a Cloud Run Job to rotate OAuth2 tokens every 10 minutes. Despite the job succeeding and the Grafana API acknowledging the update (incrementing the datasource version), queries fail with a 401 approximately 20 minutes after a fresh token is pushed, even though GCP access tokens have a default 60-minute TTL.

Steps to Reproduce

• Configure a Prometheus datasource in Grafana Cloud pointing to https://monitoring.googleapis.com/v1/projects/[PROJECT_ID]/location/global/prometheus/.
• Set Auth to "No Authentication".
• Use gmp-datasource-syncer to inject an Authorization: Bearer header via the Grafana API.
• Observe that queries work initially but fail with 401: ACCESS_TOKEN_EXPIRED well before the 60-minute token expiration.

Attempted Fixes:

• Manual Header Reset: Manually deleted the Authorization header in the Grafana UI and allowed the Cloud Run job to recreate it. This resolved the issue for ~20 minutes before it recurred.
• Job Health Verification: Confirmed the gmp-datasource-syncer job is completing successfully (EXECUTION_SUCCEEDED) and that the Grafana API returns 200 OK for the update.
• IAM Verification: Verified that the Google Service Account (GSA) used by the syncer has roles/monitoring.viewer permissions and that no IAM changes occurred recently.
• Token Lifetime Check: Observed that tokens appear to "expire" in the Grafana proxy's view in ~20 minutes, suggesting a potential caching or token-shadowing issue in the Grafana Cloud backend.
• Please note that the same issue is reproducible with an older image (v0.14.0-gke.0)

Environment Information

• Grafana Version: 12.4.0 (Cloud)
• Syncer Image: gke.gcr.io/prometheus-engine/datasource-syncer:v0.17.3-gke.0
• Infrastructure: Google Cloud Run (Job triggered by Cloud Scheduler)

apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: gmp-datasource-syncer-prd
  labels:
    component: gmp-datasource-syncer
spec:
  template:
    spec:
      template:
        spec:
          containers:
          - name: gmp-datasource-syncer
            image: gke.gcr.io/prometheus-engine/datasource-syncer:v0.17.3-gke.0
            args:
            - --datasource-uids=REDACTED_UID
            - --grafana-api-endpoint=https://REDACTED.grafana.net
            - --project-id=REDACTED-PROJECT-ID
            env:
            - name: GRAFANA_SERVICE_ACCOUNT_TOKEN
              valueFrom:
                secretKeyRef:
                  key: latest
                  name: grafana-sa-token
          serviceAccountName: gmp-syncer-sa@REDACTED.iam.gserviceaccount.com

[aaabramov]

Could this be related to some Grafana's underlying release changes?

[lyanco]

Hm. The Grafana Cloud setup that I have (and that is also using the Cloud Run datasource-syncer job) seems to be working.

The fact that it's failing after 20 minutes, but with the token refresh happening every 10 minutes, does make me think there's some sort of caching on the Grafana side.

Any chance your service account token in Grafana is set to an expiration other than "never"?

[aaabramov]

@lyanco, thanks for a quick reply.
Grafana's SA has no token expiration set.
I also manually (curl and running the image locally) executed requests, and they succeeded. The datasource versions are incremented.

[aaabramov]

Just letting you know that I created a support ticket with Grafana too for them to investigate possible caching issues.

[aaabramov]

We tried changing the job to run every 5 minutes, and it seems to have fixed the issue.
We have no custom constraints on default SA token expiration, so it's one hour (I checked the generated access token and it's confirmed).

[lyanco]

Glad to hear you fixed it... I don't know why that would have fixed it though!

I'm going to close this, but any readers feel free to re-open if you also encounter this issue.