Skip to content

[P1.5-S6-ARI-40] Refactor GCPLogConnector — configurable resource_type_templates #74

Description

@bjridicodes

Ticket: P1.5-S6-02

Type: Feature | Est: 1.5d

Goal: GCPLogConnector accepts a config dict of resource type → filter template list so new GCP service types can be added via conf.yaml without any Python code changes.

Scope:

  • implementations/clusters/cloud/gcp/log_connector.py:
    • Add resource_type_templates: dict[str, list[str]] | None = None to constructor
    • Refactor _build_filter(): when resource_type_templates is set and the target resource type is found, use the template's filter clauses; otherwise fall back to the existing generic filter (backward compatible)
    • Template format:
      {
        "cloud_dataproc_cluster": ['resource.type="cloud_dataproc_cluster"'],
        "bigquery_dataset": ['resource.type="bigquery_dataset"', 'protoPayload.serviceName="bigquery.googleapis.com"'],
        "cloud_function": ['resource.type="cloud_function"'],
        "pubsub_subscription": ['resource.type="pubsub_subscription"'],
        "gcs_bucket": ['resource.type="gcs_bucket"'],
      }
  • api/dependencies.pyget_agent2(): load resource_type_templates from gcp_resource_type_templates() config helper when constructing GCPLogConnector

Acceptance criteria:

  • Existing Dataproc queries still work without resource_type_templates set (backward compatible)
  • resource_type_templates={"bigquery_dataset": [...]} returns BigQuery logs from Cloud Logging
  • Unit test: connector built with templates; assert _build_filter() output contains correct resource.type clause for a configured type
  • Unit test: connector built without templates; assert existing generic filter behaviour unchanged

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1.5-S6S6: GCP Native Service ConnectorsfeatureNew capability or user-facing functionalityphase-1.5All Phase 1.5 issues

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions