vector gcp pub

[lhh528634141]

When using the latest version of Vector to consume GCP Pub/Sub, using the API key method causes an error.

source{component_kind="source" component_id=gcp_pubsub_cloudtrail component_type=gcp_pubsub}: vector::internal_events::gcp_pubsub: Failed to fetch events. error=status: PermissionDenied, message: "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.", details: [], metadata: MetadataMap { headers: {} } error_code="failed_fetching_events" error_type="request_failed" stage="receiving"

vector vection: 0.59.0,vector run in aws ec2

[kgeminicdev]

In the gcp_pubsub source, api_key never reaches Google. That's why you get "unregistered callers". The source talks gRPC through tonic. Vector adds the key as ?key=... on the endpoint URI (src/gcp.rs, apply_uri). Tonic only keeps the endpoint's scheme and host, and the path comes from the RPC method (/google.pubsub.v1.Subscriber/StreamingPull), so the query string is thrown away. With an API key, make_token() returns nothing, so no authorization metadata is sent either. The request arrives with no identity at all.

Use a service account instead. You're on EC2, so there's no GCE metadata server to fall back on. Give the source a service account JSON key that has roles/pubsub.subscriber on the subscription:

sources:
  gcp_pubsub_cloudtrail:
    type: gcp_pubsub
    project: my-project
    subscription: my-subscription
    credentials_path: /etc/vector/gcp-sa.json

You can also set GOOGLE_APPLICATION_CREDENTIALS=/etc/vector/gcp-sa.json in Vector's environment and leave credentials_path out. Either way, remove api_key. When a credentials file is present it takes precedence anyway.

[pront]

Note that Pub/Sub does not support API keys for authentication: https://docs.cloud.google.com/pubsub/docs/authentication.