| Status | |
|---|---|
| Stability | development: traces, logs, metrics |
| Distributions | none yet |
| Issues | |
| Maintainers | haiyuan-eng-google, schan1031 |
Exports OTLP traces, logs, and metrics into documented BigQuery relational tables via the BigQuery Storage Write API.
The OpenTelemetry Collector has no BigQuery exporter. Getting traces, logs, and metrics into BigQuery for SQL analytics today means routing through Cloud Logging sinks, standing up Pub/Sub plus Dataflow, or batch-loading files — an extra product hop, an extra pipeline to operate, or manual work.
This component has exactly one job: reliable transport with a stable, documented schema. It is not an analytics product. It does not aggregate, sample, downsample, or interpret telemetry.
Note
This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program. See SUPPORT.md for support and community channels.
The current release is v0.1.0, at development component stability. The exporter runs inside a custom OpenTelemetry Collector distribution built with the OpenTelemetry Collector Builder (see Examples), maps each record onto the documented v0alpha3 (v1.0.0-candidate) tables, and appends rows through the BigQuery Storage Write API default stream.
flowchart LR
sources["Applications and agent frameworks"]
subgraph collector["OpenTelemetry Collector (custom distribution)"]
receivers["OTLP receivers: gRPC 4317, HTTP 4318"]
exporter["bigquery exporter"]
receivers --> exporter
end
sources --> receivers
exporter -->|"Storage Write API default stream"| tables["otel_spans, otel_logs, otel_metrics"]
exporter -.->|"optional: rejected records"| quarantine["otel_quarantine"]
exporter -.->|"optional: oversized payloads"| gcs["Cloud Storage"]
tables --> views["Deduplication, companion, and GenAI views"]
views --> analysis["SQL, dashboards, Dataform marts"]
tables --> alerts["Continuous Queries for real-time alerting"]
- Documented, versioned schema. Traces, logs, and metrics land in
otel_spans,otel_logs, andotel_metricsunder thev0alpha3contract, which stores plainJSONattribute bags stamped withotel_schema_*table labels. - At-least-once delivery with read-time deduplication. Every row carries a
deterministic identity, and the
_dedupviews collapse retried appends. See Delivery semantics. - Least privilege by default.
auto_createcan create the dataset, tables, views, and search indexes, but it is off by default, so the exporter needs only metadata-read and append permissions. See IAM. - Protection before export. Optional redaction of sensitive keys and values, structural limits on pathological payloads, an optional dead-letter table for rejected records, and Cloud KMS keys and labels for the resources it creates. See Redaction and Structural limits.
- Large-payload offload and adaptive backpressure. Optional
content-addressed offload of oversized values to Cloud Storage, referenced
from BigQuery as
ObjectRefvalues, and optional AIMD control of request size and concurrency under Storage Write API congestion.
Validated performance. Throughput at write.stream_pool_size=4, with 1 KiB payloads and redaction off:
| Destination | Spans/s | Logs/s | Metric points/s |
|---|---|---|---|
In-process Storage Write API fake (bqfake) |
27,896 | 28,983 | 23,544 |
| Live BigQuery | 10,404 | 9,594 | 9,185 |
Against live BigQuery the exporter used 1.24–1.66 vCPU cores per 10,000
rows/s, with an 86.8 MiB process peak RSS and companion_views disabled during
the throughput runs. Across write.stream_pool_size=1..16 in bqfake,
throughput ranged from 18,791 to 29,459 spans/s, 22,650 to 28,983 logs/s, and
15,876 to 23,544 metric points/s. Under fault injection in bqfake, with every
second batch throttled and acknowledgements delayed by 25 ms, every throttled
batch succeeded on retry, and payload offload reached a 99.92% SHA-256
deduplication hit rate on the benchmark's repeated 8 KiB payloads. The live
integration test suite (-tags=integration) exercises auto_create view
provisioning, timeout recovery, and the deduplication views. Methodology and
raw results are in
docs/benchmarks/v0.1.0-performance-envelope.md,
and docs/cost-estimation.md models BigQuery and
Cloud Storage cost.
Agent framework inputs. “Supports OTLP” is not the same as “works with environment variables only.” The execution-verified agent framework guide records the activation tier, gRPC/HTTP support, signal placement, content defaults, and privacy caveats for Claude Code/Agent SDK, Google ADK, LangGraph, OpenAI, CrewAI, AWS Bedrock, and Microsoft Agent Framework. Two collector settings are non-negotiable for that matrix: enable both OTLP receivers because ADK and LangGraph are HTTP-only on their built-in paths, and keep the logs pipeline because several frameworks emit model content or cost as log events rather than span attributes.
Deployment options. The stock otelcol-contrib image does not include this exporter, so the
Collector options below run a custom distribution. The BigQuery options
provision the destination and build on the exported tables.
| Option | Start here | Use it to |
|---|---|---|
| Collector Builder | example/otelcol-builder.yaml and make collector-check |
Build the distribution and check component registration and configuration parsing without credentials. |
| Docker Compose | example/docker-compose.yml |
Run the Collector on one host for development; see Docker Compose deployment. |
| Helm on Kubernetes or GKE | deploy/helm/bigquery-otel-collector/ |
Run the Collector as a sidecar or a gateway. |
| Cloud Run | deploy/cloudrun/service.yaml |
Run a serverless, autoscaling Collector. |
| Terraform | deploy/terraform/ |
Provision the dataset, tables, views, Collector service account, and IAM bindings. |
| Dataform | deploy/dataform/ |
Build incremental marts over the companion views for the dashboards. |
| Continuous Queries | deploy/continuous-queries/ |
Send real-time alerts on newly appended spans. |
deploy/README.md has quick starts for Helm and Cloud Run,
docs/deployment-models.md compares the sidecar,
gateway, Cloud Run, and Docker Compose models,
dashboards/ has ready-made Looker Studio and Grafana
dashboards, and docs/operator-guides/ has
production runbooks for the four operator CLIs.
See the configuration inventory for the full option inventory and stability classification (stable vs. experimental), the migration guide for deprecation rules and schema compatibility across releases, and the cost estimation guide for BigQuery and Cloud Storage cost modeling.
| Option | Default | Description |
|---|---|---|
project |
required | Google Cloud project owning the destination dataset (supports domain-scoped IDs such as example.com:my-project). |
dataset |
required | Destination BigQuery dataset ID. |
location |
— | Dataset location. Required only when creating a dataset. |
kms_key_name |
— | Default Cloud KMS CryptoKey resource name (projects/{p}/locations/{l}/keyRings/{kr}/cryptoKeys/{ck}) applied when creating the dataset, signal tables, or quarantine table unless overridden per signal/quarantine. |
labels |
— | Default Google Cloud resource labels (map[string]string, up to 60 custom table labels — BigQuery's 64-label limit minus 4 reserved otel_schema_* labels) applied when creating the dataset, signal tables, quarantine table, and views. |
traces.table |
otel_spans |
Destination table for OTLP spans. |
traces.partition_granularity |
HOUR |
Creation-time partitioning granularity when creating traces.table (HOUR or DAY). Existing tables are not repartitioned in place. |
traces.clustering |
[service_name, trace_id, name] |
Optional 1–4 clustering columns when creating traces.table, bounded to otel_spans_dedup's PARTITION BY window (service_name, trace_id, span_id, name). |
traces.partition_expiration_days |
0 |
Partition retention in days when creating traces.table (0 omits partition expiration). |
traces.require_partition_filter |
false |
Require a partition filter on queries over traces.table when created. |
traces.kms_key_name |
— | Per-table Cloud KMS CryptoKey override when creating traces.table. |
traces.labels |
— | Labels merged on top of top-level labels when creating traces.table and its deduplication/companion views. |
traces.max_staleness |
— | BigQuery table max_staleness duration (1s..168h, e.g. 15m) when creating traces.table. |
traces.search_index.columns |
[trace_id, span_id] |
Create-only search index columns (or ["ALL COLUMNS"]) when auto_create.search_indexes is enabled. |
traces.search_index.analyzer |
NO_OP_ANALYZER |
Create-only search index analyzer (NO_OP_ANALYZER, LOG_ANALYZER, or PATTERN_ANALYZER). |
logs.table |
otel_logs |
Destination table for OTLP log records. |
logs.partition_granularity |
DAY |
Creation-time partitioning granularity when creating logs.table (DAY or HOUR). Existing tables are not repartitioned in place. |
logs.clustering |
[service_name, trace_id, severity_number] |
Optional 1–4 clustering columns when creating logs.table, bounded to otel_logs_dedup's PARTITION BY window (service_name, log_name, severity_number, trace_id, span_id, insert_id). |
logs.log_name_attribute |
— | Optional log attribute carrying log_name before falling back to gcp.log_name, logging.googleapis.com/logName, log.iostream, service_name, and "default". |
logs.insert_id_attribute |
— | Optional log attribute carrying insert_id before falling back to logging.googleapis.com/insertId, gcp.insert.id, gcp.insert_id, log.record.uid, and (when logs.synthesize_insert_id is true) an id1: synthesized hash. |
logs.synthesize_insert_id |
true |
Synthesize a compact 26-character content-hash insert_id (id1:<22-char base64url>) when no upstream insert ID attribute is present; set false to write NULL (insert_id IS NULL) and withdraw retry deduplication guarantees for ID-less logs. |
logs.partition_expiration_days |
0 |
Partition retention in days when creating logs.table. |
logs.require_partition_filter |
false |
Require a partition filter on queries over logs.table when created. |
logs.kms_key_name |
— | Per-table Cloud KMS CryptoKey override when creating logs.table. |
logs.labels |
— | Labels merged on top of top-level labels when creating logs.table and its deduplication/companion views. |
logs.max_staleness |
— | BigQuery table max_staleness duration (1s..168h) when creating logs.table. |
logs.search_index.columns |
[trace_id, span_id] |
Create-only search index columns (or ["ALL COLUMNS"]) when auto_create.search_indexes is enabled. |
logs.search_index.analyzer |
NO_OP_ANALYZER |
Create-only search index analyzer (NO_OP_ANALYZER, LOG_ANALYZER, or PATTERN_ANALYZER). |
metrics.table |
otel_metrics |
Destination table for OTLP metric data points. |
metrics.partition_granularity |
DAY |
Creation-time partitioning granularity when creating metrics.table (DAY or HOUR). |
metrics.clustering |
[service_name, metric_name, metric_type] |
Optional 1–4 clustering columns when creating metrics.table, bounded to otel_metrics_dedup's PARTITION BY window (service_name, metric_name, metric_type, point_fingerprint). |
metrics.partition_expiration_days |
0 |
Partition retention in days when creating metrics.table. |
metrics.require_partition_filter |
false |
Require a partition filter on queries over metrics.table when created. |
metrics.kms_key_name |
— | Per-table Cloud KMS CryptoKey override when creating metrics.table. |
metrics.labels |
— | Labels merged on top of top-level labels when creating metrics.table and its deduplication view. |
metrics.max_staleness |
— | BigQuery table max_staleness duration (1s..168h) when creating metrics.table. |
metrics.search_index.columns |
[ALL COLUMNS] |
Create-only search index columns (or ["ALL COLUMNS"]) when auto_create.search_indexes is enabled. |
metrics.search_index.analyzer |
NO_OP_ANALYZER |
Create-only search index analyzer (NO_OP_ANALYZER, LOG_ANALYZER, or PATTERN_ANALYZER). |
quarantine.enabled |
false |
Route permanently rejected spans, log records, and metric data points to the dead-letter table (otel_quarantine) instead of dropping them only in metrics. |
quarantine.table |
otel_quarantine |
Destination table for quarantined records when quarantine.enabled is true. |
quarantine.partition_expiration_days |
30 |
Partition retention in days when creating quarantine.table. |
quarantine.require_partition_filter |
false |
Require a partition filter on queries over quarantine.table when created. |
quarantine.kms_key_name |
— | Per-table Cloud KMS CryptoKey override when creating quarantine.table. |
quarantine.labels |
— | Labels merged on top of top-level labels when creating quarantine.table. |
quarantine.max_staleness |
— | BigQuery table max_staleness duration (1s..168h) when creating quarantine.table. |
redaction.enabled |
false |
Mask sensitive attribute keys and string values before payload offload, limits.* checks, fingerprinting, and quarantine. See Redaction. |
redaction.blocked_keys |
— | Extra attribute keys whose entire value is replaced by redaction.mask_replacement. Case-insensitive; * is a wildcard. Merged with the built-in list. |
redaction.disable_default_keys |
false |
Drop the built-in key list so only redaction.blocked_keys apply. |
redaction.blocked_value_patterns |
— | RE2 regular expressions; every match inside a string value is replaced by redaction.mask_replacement. A pattern that does not compile fails startup. |
redaction.mask_replacement |
[REDACTED] |
Replacement text, inserted literally (no $1 expansion). |
redaction.redact_log_body |
false |
Also scan string, map, and slice log record bodies. |
credentials.file |
ADC | Path to a Google credentials JSON file (mutually exclusive with credentials.json). Accepted type values: service_account, authorized_user, external_account, external_account_authorized_user, impersonated_service_account. Any other type is rejected at startup. |
credentials.json |
— | Inline credential JSON (service_account, authorized_user, external_account WIF, external_account_authorized_user, impersonated_service_account, or gdch_service_account), injectable via ${env:BQ_CREDENTIALS_JSON} and wrapped in configopaque.String. |
credentials.impersonate_service_account |
— | Target principal for service-account impersonation, using ambient ADC as the source. |
credentials.impersonate_delegates |
— | Ordered delegation chain of service account emails for credentials.impersonate_service_account. |
credentials.quota_project |
— | Billing/quota project passed via option.WithQuotaProject. |
credentials.universe_domain |
— | Sovereign/trusted partner cloud universe domain passed via option.WithUniverseDomain. |
endpoint.url |
— | Alternate Storage Write API gRPC endpoint (host:port or http(s)://host:port; leading http:// / https:// and trailing / are stripped automatically before dialing). Emulator and development only. |
endpoint.admin_url |
— | Alternate BigQuery REST Admin API endpoint (https://host:port/...) for Private Service Connect, or for emulators that serve REST metadata and gRPC Storage Write on separate ports. Metadata calls always carry credentials, so http:// is refused unless the host is loopback. |
endpoint.insecure |
false |
Disables TLS. Refused unless endpoint.url is loopback. |
endpoint.without_authentication |
false |
Sends no credentials. Refused unless the endpoint is loopback. |
auto_create.dataset |
false |
Create the dataset if absent. Requires location. |
auto_create.tables |
false |
Create the active signal's destination table (plus quarantine.table when enabled) and _dedup view if absent, and reconcile additive nullable columns on existing tables (ReconcileAdditiveSchema). An existing _dedup view is reconciled to the current definition (the query only, best effort), which needs bigquery.tables.update. |
auto_create.companion_views |
false |
Create the companion views if absent and reconcile existing ones to the current definition (the query only, best effort): otel_spans_as_allspans, otel_logs_as_alllogs, v_gen_ai_spans, v_gen_ai_model_costs, v_gen_ai_sessions, v_gen_ai_tool_usage, v_gen_ai_agent_trajectory, v_gen_ai_logs, v_gen_ai_metrics, and v_gen_ai_metric_exemplars. Requires auto_create.tables. |
auto_create.allow_unprunable_views |
false |
Create v_gen_ai_logs, v_gen_ai_metric_exemplars, and v_gen_ai_agent_trajectory even when the spans table requires a partition filter (Helm: bigquery.autoCreateAllowUnprunableViews). BigQuery rejects every query over those three views when otel_spans requires a partition filter, so by default startup skips them in that case and logs a warning; the other companion views are created either way, and startup does not fail. The spans table's metadata decides; traces.require_partition_filter counts only while that table does not exist yet. |
auto_create.search_indexes |
false |
Execute CREATE SEARCH INDEX IF NOT EXISTS ... OPTIONS (analyzer = 'NO_OP_ANALYZER') on (trace_id, span_id) for otel_spans and otel_logs and on (ALL COLUMNS) for otel_metrics. Requires auto_create.tables and bigquery.jobs.create. |
write.max_request_bytes |
8388608 (8 MiB) |
Our request-size headroom threshold, enforced after protobuf serialization and before AppendRows. Distinct from, and below, the API's own per-request limit. |
write.max_row_bytes |
1048576 (1 MiB) |
Rejects individual serialized signal rows above this size and bounds quarantine rows by truncating raw_payload. |
write.max_inflight_requests |
8 |
Bounds asynchronous in-flight appends per stream slot. |
write.max_inflight_bytes |
67108864 (64 MiB) |
Bounds memory held by in-flight appends across the destination table's stream pool. |
write.stream_pool_size |
1 |
Number of default-stream ManagedStream slots opened per destination table (1..16). When > 1, opens N dedicated per-stream AppendRows connections and sizes the gRPC connection pool to max(connection_pool_size, stream_pool_size). |
write.backpressure.enabled |
false |
Enable adaptive AIMD adjustment of the effective request byte split threshold and in-flight concurrency under Storage Write API congestion. |
write.backpressure.min_request_bytes |
1048576 (1 MiB) |
AIMD floor for the effective per-request byte split threshold (<= write.max_request_bytes). |
write.backpressure.min_inflight_requests |
1 |
AIMD floor for effective concurrent AppendRows sub-requests (>= 1 and <= write.max_inflight_requests). |
write.backpressure.decrease_factor |
0.5 |
Multiplicative decrease factor in (0.0, 1.0) applied on throttle or slow acknowledgements. |
write.backpressure.recovery_interval |
10s |
Minimum elapsed duration between additive recovery steps on clean acknowledgements. |
write.backpressure.latency_threshold |
2s |
Acknowledgement latency above which a clean AppendResult triggers a multiplicative step-down (0s disables latency triggers). |
limits.* |
see below | Structural bounds applied before serialization. |
sending_queue |
8 consumers | Standard exporterhelper queue, including the persistent queue. Consumer concurrency defaults to the managedwriter request window. |
retry_on_failure |
max_elapsed_time: 900s |
Standard exporterhelper retry. The horizon must be finite. |
timeout |
5s |
Budget for one export attempt (including slot waits and every acknowledgement) and for each individual startup metadata or provisioning API call. See Timeout and concurrency envelope. |
offload.enabled |
false |
Route oversized span/log values and quarantine payloads to Google Cloud Storage, recording BigQuery ObjectRef references. |
offload.bucket |
— | GCS bucket name for offloaded payloads. Required when offload.enabled is true. |
offload.prefix |
otel-offload |
Object name prefix within the bucket. |
offload.authorizer |
— | BigQuery connection resource name for authorized GCS reads. |
offload.min_bytes |
262144 (256 KiB) |
Minimum payload size to offload. Smaller payloads stay inline. |
offload.max_object_bytes |
67108864 (64 MiB) |
Maximum size of a single offloaded object. |
offload.max_concurrency |
8 |
Maximum concurrent GCS uploads. |
offload.timeout |
10s |
GCS upload timeout. |
offload.cache_size |
4096 |
Cross-batch LRU cache entries to dedup offloaded objects by content hash. |
Applied before serialization, these stop pathological payloads from becoming
unqueryable rows or consuming unbounded memory. A breach rejects only the
offending record, counted under rejected_rows{reason} and optionally written
to otel_quarantine when quarantine.enabled is true.
| Option | Default |
|---|---|
limits.max_attribute_count |
1024 |
limits.max_event_count |
1024 |
limits.max_link_count |
1024 |
limits.max_nesting_depth |
16 |
limits.max_collection_size |
4096 |
limits.max_value_bytes |
262144 |
auto_create is off by default deliberately: leaving it off avoids dataset,
table, and view creation permissions. Startup still reads destination metadata
to distinguish an absent resource from an authorization ambiguity and to
validate the complete typed schema before opening the Storage Write client.
An authoritative NotFound fails startup with the corresponding
auto_create setting named in the error; other metadata-read failures warn and
defer the authorization verdict to the real append.
Redaction is off by default. With redaction.enabled: true the exporter masks
sensitive attribute keys and string values before anything it writes can carry
them:
exporters:
bigquery:
redaction:
enabled: true
blocked_keys: ["x-customer-id", "*.ssn"]
blocked_value_patterns:
- '\b\d{3}-\d{2}-\d{4}\b'
- 'Bearer\s+[A-Za-z0-9._~+/=-]+'
redact_log_body: trueIt is the first step of every export and works on a copy: the batch the Collector handed over is never modified, so a retried batch is redacted again from the original. A batch with no match is scanned and left as it is; a batch with a match is copied once. The masked form is what reaches:
- payload offload, where object names are SHA-256 digests of the content, so a secret masked only at encoding time would still be uploaded and would still shape the object name;
- the
limits.*checks; point_fingerprintand synthesized loginsert_idvalues;- the Storage Write API rows;
otel_quarantinerows, includingraw_payloadandraw_attributes.
Scanned: resource and scope attributes; span, span event, and span link
attributes; log record attributes; metric metadata, data point attributes, and
exemplar filtered attributes; and, when redaction.redact_log_body is true,
string, map, and slice log bodies.
- Blocked keys replace the entire value, whatever its type, with
redaction.mask_replacement(default[REDACTED], inserted literally). Matching is case-insensitive,*matches any run of characters, and keys inside nested map values are matched too.redaction.blocked_keysadds to the built-in list;redaction.disable_default_keysdrops that list. - Blocked value patterns are RE2 regular expressions. Every match inside a string value is replaced and the rest of the string is kept. Integer, double, boolean, and bytes values are never scanned for patterns. A pattern that is empty or does not compile fails startup, even while redaction is disabled, so a typo surfaces before the switch is flipped.
The built-in key list is authorization, proxy-authorization, cookie,
set-cookie, x-api-key, api_key, api-key, api.key, apikey,
client_secret, access_token, refresh_token, id_token, token,
password, passwd, passphrase, secret, private_key, private-key,
private.key, privatekey, credential, and credentials, plus the globs
*secret*, *password*, *passwd*, *passphrase*, *private_key*,
*private-key*, *private.key*, *privatekey*, *credential*, *api.key*,
*token, *_api_key, *-api-key, *.api_key, *.api-key, *apikey,
*.authorization, *-authorization, *_authorization, and *cookie.
*token is anchored at the end on purpose. A *token* glob would also mask
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and
gen_ai.token.type, which are counts and an enum, and the GenAI views read the
counts as integers, so masking them turns those view columns to NULL. Keys
such as token_value or a plural tokens are therefore not blocked by
default; add *token* to redaction.blocked_keys if you want them masked.
Startup also refuses a configuration where logs.insert_id_attribute or
logs.log_name_attribute names a key the redaction would mask: every record
would then carry the same insert_id or log_name, and the deduplication view
would treat distinct records as duplicates.
otelcol_exporter_bigquery_redacted_values ({signal, match_type}, where
match_type is key or value) counts what was masked. It never records key
names or values, and it counts each time a batch is redacted, so a retried
batch is counted again.
Fingerprints, synthesized insert_id hashes, and offloaded object names are computed from the masked content,
so changing the redaction settings changes them for the same input. Rows
written before and after such a change are not matched as duplicates wherever
the deduplication identity includes insert_id or point_fingerprint.
What it does not do:
- It protects what this exporter writes. The Collector's
sending_queue, including a persistent queue, holds batches as they arrived, before the exporter runs, and other components in the pipeline see the original data. - Value patterns apply only to string attribute values and, when enabled, log bodies. Span, event, link, and metric names, span status messages, scope names, and schema URLs are not scanned, and blocked keys are matched against attribute and map-body keys only.
- It does not find secrets that no configured key or pattern describes.
The stock otelcol-contrib image does not contain this repository's exporter. Build the pinned custom distribution defined in example/otelcol-builder.yaml and verify component registration and configuration parsing without credentials:
make collector-checkMinimal exporter fragment:
exporters:
bigquery:
project: my-project
dataset: otel
traces:
table: otel_spans
logs:
table: otel_logs
service:
pipelines:
traces:
receivers: [otlp]
exporters: [bigquery]
logs:
receivers: [otlp]
exporters: [bigquery]- Application Default Credentials (ADC): Default when
credentials.*is omitted. Works automatically on GCE, GKE Workload Identity, Cloud Run, and localgcloud auth application-default login(orGOOGLE_APPLICATION_CREDENTIALS). Seeexample/otel-collector-config.ymlfor a complete production pipeline. - Service Account Impersonation: Uses ambient ADC to impersonate a target service account (for cross-project or least-privilege writes):
exporters:
bigquery:
project: my-project
dataset: otel
credentials:
impersonate_service_account: otel-writer@my-project.iam.gserviceaccount.com
quota_project: my-quota-project- Local emulator and testing (
endpoint.without_authenticationandendpoint.insecure):endpoint.without_authentication: truedrops credentials and skips startup REST metadata validation (exporter.go), whileendpoint.insecure: trueturns off TLS — so talking to a plaintext emulator viaendpoint.urlrequiresendpoint.insecure: trueas well asendpoint.without_authentication: true.Config.Validate()refusesendpoint.insecureandendpoint.without_authenticationunlessendpoint.urlmatcheslocalhostor a literal loopback IP (such as127.0.0.1or::1; it does not resolve hostnames via DNS), and rejects combiningendpoint.without_authenticationwithauto_create.dataset,auto_create.tables, orauto_create.search_indexes. (Whenendpoint.without_authenticationisfalse,endpoint.admin_urlseparately overrides the authenticated REST metadata endpoint and requires loopback only when using plainhttp://.)
exporters:
bigquery:
project: test-project
dataset: otel
endpoint:
url: localhost:9050
insecure: true
without_authentication: trueFor the single-host Docker Compose walkthrough and credentialed smoke test, see Docker Compose Deployment and Verifying ingestion (smoke test). For Helm, Cloud Run, and Terraform quick starts and topology comparisons, see deploy/README.md and docs/deployment-models.md.
Keep both OTLP receivers enabled: gRPC on
4317and HTTP on4318. The sending framework chooses the transport, not the collector. Google ADK and LangGraph (through LangSmith) can send OTLP/HTTP on4318only. The other frameworks in the compatibility matrix accept gRPC or HTTP, and most of their setup recipes use gRPC on4317. Dropping the4318receiver leaves ADK and LangGraph sending nothing; dropping4317breaks those gRPC recipes.example/otel-collector-config.ymlenables both, and CI exercises both ports against it.
None of these is the stock image's 'exporters' unknown type: "bigquery",
which means the binary is not this distribution. The collector prints the
exporter's startup errors after Error: cannot start pipelines: failed to start "bigquery" exporter:, and credential errors after a further
creating BigQuery metadata client: google:. Messages use the example values
above.
| Look for | Cause and action |
|---|---|
required variable BQ_ADC_PATH is missing a value: Set BQ_ADC_PATH to an ADC JSON file |
From Compose, before any container starts. Export BQ_ADC_PATH (see Running with Docker Compose). |
could not find default credentials |
No credentials reach the collector. Mount an ADC or service-account key file, as Compose does from BQ_ADC_PATH (see Running with Docker Compose). |
open /creds/adc.json: permission denied |
The image's non-root user cannot read an owner-only file. Export BQ_CONTAINER_USER="$(id -u):$(id -g)" (see Running with Docker Compose). |
missing 'type' field in credentials |
BQ_ADC_PATH names a file that is not a credential (see Running with Docker Compose). |
destination dataset my-project.otel does not exist; create it or enable auto_create.dataset |
The dataset is absent and creation is off. |
destination table my-project.otel.otel_spans does not exist; create it or enable auto_create.tables |
The table is absent and creation is off. Run the DDL in internal/schema, or enable creation. |
creating destination table my-project.otel.otel_spans: followed by the BigQuery error |
Creation is on but the identity may not create the table; the dataset and _dedup view fail the same way. Grant the IAM create permission, or pre-create and turn auto_create off. |
append denied on my-project.otel.otel_spans: the writing identity needs bigquery.tables.updateData on this table (the write-only custom role, or roles/bigquery.dataEditor) |
Logged when an append is refused. The collector keeps running and does not retry that batch. Grant the write permission. |
The published BigQuery schema (v0alpha3 / v1.0.0-candidate, defined in
internal/schema) aligns with Cloud Trace
_AllSpans, Cloud Logging _AllLogs, OpenTelemetry GenAI semantic conventions
(v1.37+ / v1.40+), and Google ADK / Vertex Agent Analytics, and stamps
table metadata labels (otel_schema_major = "0", otel_schema_minor = "3",
otel_schema_patch = "0", otel_schema_variant = "default"):
-
Plain JSON attributes without per-row type sidecars.
attributes,resource.attributes,instrumentation_scope.attributes,events.attributes,links.attributes,otel_logs.body, andotel_metrics.exemplars.filtered_attributesare stored as plain JSON ({"gen_ai.system":"gemini","http.status_code":"200"}) so queries can use directJSON_VALUE(attributes, '$."gen_ai.system"')andLAX_INT64expressions without paying storage and wire overhead for per-row type sidecars. OTLPint64values are encoded as decimal JSON strings (strconv.FormatInt) so 64-bit precision outside IEEE-754 ($|n| > 2^{53}-1$ ) is preserved, while OTLPbytesserialize as base64 strings without distinguishingbytesvs. string or string vs. quoted integer at read time. -
Multimodal
object_refs(ObjectRef). Bothotel_spansandotel_logsincludeobject_refs ARRAY<STRUCT<uri STRING NOT NULL, version STRING, authorizer STRING, details JSON>>so offloaded prompts, completions, and multimodal blobs can be referenced via BigQueryObjectReffunctions (OBJ.FETCH_METADATA,OBJ.GET_ACCESS_URL). -
Cloud Logging identity columns on
otel_logs. Each log row carrieslog_name STRING NOT NULLandinsert_id STRING(nullable). When no upstreaminsert_idattribute is present,logs.synthesize_insert_id: truederives a 26-characterid1:<22-char base64url>content hash (4-byte"id1:"prefix + unpadded base64url encoding of the 128-bit truncated SHA-256); settinglogs.synthesize_insert_id: falsewritesNULLand withdraws retry deduplication guarantees for ID-less logs. -
Relational OTLP metrics table (
otel_metricsandotel_metrics_dedup).otel_metricsstores all five OTLP metric data-point types (GAUGE,SUM,HISTOGRAM,EXPONENTIAL_HISTOGRAM,SUMMARY) in a single 37-column relational table partitioned byDATE(timestamp)and clustered by(service_name, metric_name, metric_type), withpoint_fingerprint STRING NOT NULL(fp1:<sha256_hex>over canonicalattributesJSON) andexemplars ARRAY<STRUCT<time TIMESTAMP, time_unix_nano INT64, value_int64 INT64, value_double FLOAT64, span_id STRING, trace_id STRING, filtered_attributes JSON>>so GenAI token/latency histograms join directly back tootel_spans_dedup. -
Companion and GenAI rollup views (
auto_create.companion_views).-
otel_spans_as_allspans: projectsotel_spans_dedupinto the 24-column Cloud Trace_AllSpanslayout (statusnarrowed toSTRUCT<code INT64, message STRING>). -
otel_logs_as_alllogs: projects rawotel_logsinto the Cloud Logging Analytics_AllLogs/LogEntryshape (timestamp,receive_timestamp,log_name,log_id,insert_id,severity,severity_number,otel_severity_number,otel_severity_text,text_payload,json_payload,labels,attributes,resourceSTRUCT<type STRING, labels JSON>, synthesizedhttp_requestSTRUCTin_AllLogspositional field order including typed-NULLcompatibility fieldslatency,cache_lookup,cache_hit,cache_validated_with_origin_server, andcache_fill_bytes,trace,span_id,trace_sampledfrom(flags & 1) = 1, and top-levelservice_name).traceis the bare trace ID fromotel_logs.trace_id, not theprojects/PROJECT_ID/traces/TRACE_IDresource name that Cloud Logging stores, so join it tootel_spans.trace_iddirectly and rewrite predicates copied from Cloud Logging queries that match the full name.log_idis the URL-decoded last segment oflog_name(for examplecloudaudit.googleapis.com%2Fdata_accessbecomescloudaudit.googleapis.com/data_access; escapes are decoded once and a malformed escape is kept as is).severityandseverity_numberare one canonical Cloud LoggingLogSeveritypair (DEFAULT0,DEBUG100,INFO200,NOTICE300,WARNING400,ERROR500,CRITICAL600,ALERT700,EMERGENCY800): an OTLPseverity_numberin1..24maps via Google Cloud's OTLP-to-Cloud-Logging exporter mapping (1–8DEBUG,9–10INFO,11–12NOTICE,13–16WARNING,17–20ERROR,21–22CRITICAL,23ALERT,24EMERGENCY), and normalizedseverity_textdecides when the number is0,NULL, or out of range; the raw OTLP values stay inotel_severity_numberandotel_severity_text.http_request.request_sizeandhttp_request.response_sizecoalesce total-size attributes (http.request.size,http.response.size) ahead of body-size and legacy content-length attributes. Unlikeotel_spans_as_allspans(which readsotel_spans_dedup),otel_logs_as_alllogsreads rawotel_logsdirectly so BigQuery search-index and cluster predicates push down to base storage without aROW_NUMBER()barrier, and it is created in logs-only pipelines without requiringotel_spans_dedup. Compatibility caveat: JSON-synthesizedSTRUCTfields (http_request.*,resource.type,resource.labels) provide SQL migration compatibility, not physical column pruning or performance parity, and producer-omitted fields remainNULL. -
v_gen_ai_spans: filtersotel_spans_dedupto GenAI/agent spans (including spans carrying onlygen_ai.main_agent.nameorgen_ai.main_agent.idon spanattributes) and promotesgen_ai.*,error.type, andgcp.vertex.agent.*attributes (gen_ai_operation_name,gen_ai_system,gen_ai_request_model,gen_ai_response_model,gen_ai_agent_name,gen_ai_agent_id,gen_ai_conversation_id,gen_ai_usage_input_tokens,gen_ai_usage_output_tokens,gen_ai_usage_cache_read_input_tokens,gen_ai_usage_cache_creation_input_tokens,gen_ai_usage_reasoning_tokens,gen_ai_request_temperature,gen_ai_request_top_p,gen_ai_request_max_tokens,gen_ai_response_finish_reason,gen_ai_tool_name,gen_ai_tool_call_id,gen_ai_tool_type,gen_ai_tool_description,gen_ai_data_source_id,gen_ai_output_type,error_type,adk_event_id,adk_session_id,adk_invocation_id,adk_user_id,gen_ai_main_agent_id,gen_ai_main_agent_name,gen_ai_main_agent_description,gen_ai_agent_version,gen_ai_response_id,gen_ai_response_time_to_first_chunk) into top-level columns.gen_ai_usage_reasoning_tokensreads the currentgen_ai.usage.reasoning.output_tokensattribute first and the earliergen_ai.usage.reasoning_tokensspelling second;gen_ai_main_agent_*andgen_ai_agent_versionread spanattributesfirst andresource.attributessecond;error_typeis theerror.typeattribute. -
v_gen_ai_model_costs: hourly token, cache-read/write, reasoning, latency (APPROX_QUANTILES), and error-count (error_count,typed_error_count, andtop_error_typesover non-nullerror_typevalues) rollups overv_gen_ai_spansgrouped by(usage_hour, service_name, gen_ai_agent_name, provider, model). Each LLM call is counted once: a span with a direct child span that reports input or output tokens is taken to report the sum of its children and is left out, and so is an agent or tool span that reports no tokens of its own. An agent span that names a request model and reports tokens, with no child that reports tokens, is still counted, as one call. Only direct children are examined, so tokens that a grandparent span sums over a deeper descendant are not removed. A parent is matched with its children only when they share a trace, a service name, and a usage hour: a pair that straddles an hour boundary, or whose spans carry different service names, is not matched, so both spans count and the parent's tokens are added to its children's. An agent or tool span that reports tokens but names no request model is not an inference span, so it is not counted as a call, and its tokens are not in this view, althoughv_gen_ai_sessionsstill adds them to the session totals. -
v_gen_ai_sessions: session/conversation rollups overv_gen_ai_spans, one row per(service_name, session_start_date, session_id), wheresession_idisCOALESCE(adk_session_id, gen_ai_conversation_id)andsession_start_dateis the UTC date of the span start time. A session that runs across midnight UTC therefore has one row for each date it touches. Token totals leave out any span that has a direct child span reporting tokens, so a parent that reports the sum of its children is not counted with them. Only spans that carry a session id take part, and a parent is matched with its children only within one service name, UTC date, session, and trace. A child without a session id is therefore not seen: its parent keeps its own tokens, which already include the child's, and the child is not counted. A pair split by midnight UTC or by service name is not matched, so both spans count. The token totals cover every span that has no token-reporting child, not only LLM calls, so aninvoke_agentspan that reports tokens but names no request model adds to them without being counted inllm_call_countor inv_gen_ai_model_costs.llm_call_countcounts LLM calls by the same rule asv_gen_ai_model_costs, andagent_invocation_countcountsinvoke_agentspans.session_start_date,total_cache_creation_input_tokens,llm_call_count,agent_invocation_count,typed_error_count, andtop_error_types(top 5 non-nullerror_typevalues) are the last six columns, so every earlier column keeps its position. -
v_gen_ai_tool_usage: hourly tool invocation count, error rate, typed error count (typed_error_count,top_error_types), and latency rollups overv_gen_ai_spansgrouped by(usage_hour, service_name, gen_ai_agent_name, gen_ai_tool_name, gen_ai_tool_type). -
v_gen_ai_agent_trajectory: single-pass span DAG overv_gen_ai_spans(non-recursiveWITHCTEs pre-aggregating per-(trace_id, service_name)node arrays intopo_raw..resolved_topologywith depth cap16and cycle guard) with 40 columns: 31 base span and GenAI columns plusdepth,root_span_id,nearest_agent_span_id,nearest_agent_name,agent_path(consecutive duplicates skipped),is_orphan(true whenparent_span_idis absent fromv_gen_ai_spansfor the sametrace_idand null-safeservice_namewithin 24 hours before to 1 hour after the child'sstart_time),step_index(ROW_NUMBER()per(trace_id, service_name)ordered by(start_time ASC, span_id ASC)),is_leaf, andis_token_leaf. Before graph resolution runs,spanskeeps one canonical row per(trace_id, service_name, span_id)(ORDER BY start_time ASC, end_time DESC, span_name ASC, duration_nano DESC), so duplicate span IDs that surviveotel_spans_dedupwith a differentstart_timeornamecannot multiply child paths. The agent columns use the agent name a span declares itself (gen_ai.agent.name, thengcp.vertex.agent.name) rather than theservice_namefallback ofgen_ai_agent_name: aninvoke_agentorcreate_agentspan always starts a new nearest agent, a span that only declares an agent name starts one when no span above it has one, and achatorexecute_toolspan that declares no agent inherits its parent's nearest agent and adds nothing toagent_path.is_token_leafis true for a span that has at least one uncovered non-null token family (input,output, orreasoning; an explicit0counts) after aggregating per-family coverage (topo_covviaLOGICAL_ORover 1..32-hopemitted_ancsintopo_ancestors) across all emitted trajectory descendants up to 32 parent hops (even across tokenless intermediate spans such asexecute_tooland across parent/child links that straddle the[-24h, +1h]tree-parent window, such as an out-of-window child emitted as an orphan root or a child of adepth = 16span, regardless ofgen_ai_operation_name). When a span is only partially covered by descendants (for example, a parent reportsinput,output, andreasoningwhile its child reports onlyinputandoutput),is_token_leafremainsTRUEfor the uncovered family and the covered token families on that row are projected asNULLso summinggen_ai_usage_*_tokens WHERE is_token_leaf = TRUEneither double-counts covered families nor drops uncovered ones; fully covered rollup rows (is_token_leaf = FALSE) retain their raw token counts. When spans form a parent cycle broken by the[-24h, +1h]window,topo_covdetects mutual reachability acrossemitted_ancsand breaks cycle ties on(start_time, span_id)without discarding acyclic earlier children of orphan trees. Adepth = 16span is never suppressed by unemitteddepth > 16children so it retains the leaf token totals for its truncated subtree when ancestors report subtree rollups (tokens reported only on unemitteddepth > 16spans are outside this view). Unlikev_gen_ai_model_costs(hourly direct-child check over inference spans withgen_ai_request_modeland input/output tokens) andv_gen_ai_sessions(UTC-date, session-scoped direct-child check over input/output tokens),is_token_leaffollows multi-hop trajectory paths up todepth <= 16(17 levels,0..16) across the[-24h, +1h]parent/child window and checks token-family coverage across input, output, and reasoning tokens. A group of spans whose parent links form a cycle within[-24h, +1h]with no root or orphan to start from (a span that names itself as its parent included) fails to terminate within 16 parent hops (depth = 17) and produces no rows, and every emitted path stops at depth16. Likev_gen_ai_logsandv_gen_ai_metric_exemplars, it joins spans on a time window that depends on each row, so it cannot be queried whenotel_spansrequires a partition filter, and startup then skips it with a warning unlessauto_create.allow_unprunable_viewsis set. -
v_mcp_spans: filtersotel_spans_dedupto Model Context Protocol spans (any span carryingmcp.method.name,mcp.session.id, ormcp.resource.uri, pinned to OpenTelemetry MCP Semantic Conventions1.40.0, Development status) so non-GenAI MCP operations (initialize,tools/list,resources/read,prompts/get) are visible without widening thev_gen_ai_spanspredicate. Promotesmcp_method_name,mcp_protocol_version,mcp_session_id,mcp_resource_uri,jsonrpc_request_id,rpc_response_status_code, GenAI correlation columns (gen_ai_operation_name,gen_ai_system,gen_ai_tool_name,gen_ai_tool_call_id,gen_ai_prompt_name,gen_ai_agent_namefromgen_ai.agent.name,gcp.vertex.agent.name, thengen_ai.main_agent.name, anderror_type), server endpoint columns (server_address,server_port,network_transport),is_tool_call, andalso_in_gen_ai_view(true when the span also satisfies thev_gen_ai_spansWHEREclause, so dashboards can deduplicate dual-taggedtools/callspans acrossv_mcp_spansandv_gen_ai_spans). -
v_mcp_tool_usage: hourly MCP method, tool, resource, and prompt rollup overv_mcp_spansgrouped by(usage_hour, service_name, mcp_method_name, mcp_protocol_version, gen_ai_tool_name, mcp_target_name, span_kind, server_address)wheremcp_target_nameisCOALESCE(gen_ai_tool_name, mcp_resource_uri, gen_ai_prompt_name)soresources/readandprompts/getspans group by resource URI or prompt name alongsidegen_ai_tool_name. Aggregates at the per-span observation grain (span_kindseparatesCLIENTandSERVERspans so paired client/server traces produce separateCLIENTandSERVERrows, and session-only or resource-only MCP spans withoutmcp.method.namegroup undermcp_method_name IS NULL). Exposescall_count,gen_ai_tagged_call_count(COUNTIF(also_in_gen_ai_view)),error_count(status_code = 2 OR error_type IS NOT NULL, capturing JSON-RPC or MCP error responses that leavestatus_codeunset, whereasv_gen_ai_tool_usage.error_countcountsstatus_code = 2),typed_error_count,top_error_types(top 5 non-nullerror_typeentries),error_rate,avg_duration_ms, andp50/p95/p99duration percentiles. -
v_gen_ai_logs: filtersotel_logs_dedupto GenAI/agent log events and left-joinsotel_spans_dedupon(trace_id, span_id)and a null-safeservice_namematch, within the window described under Time-boundedv_gen_ai_logsspan enrichment below, to promotegen_ai.*andgcp.vertex.agent.*correlation columns pluserror_type(from the log record's ownerror.typeattribute). A value the log record carries wins over the same attribute on its span, and the agent name, session ID, and user ID are coalesced over the same attribute keys, in the same order, as the columns ofv_gen_ai_spans. A filter ontimestampcan narrow the scan ofotel_logs, but not the span side of the join, andv_gen_ai_logscannot be queried whentraces.require_partition_filteristrue. -
v_gen_ai_metrics: filtersotel_metrics_dedupto metric names matchingLIKE 'gen_ai.%'(for examplegen_ai.client.token.usage,gen_ai.client.operation.duration, andgen_ai.client.inference.usage.*) and promotesgen_ai_operation_name,gen_ai_system(fromgen_ai.system, thengen_ai.provider.name),gen_ai_provider_name,gen_ai_request_model,gen_ai_response_model,gen_ai_token_type,error_type,gen_ai_token_modality,gen_ai_agent_name(fromgen_ai.agent.namethengen_ai.main_agent.nameon pointattributes, thengen_ai.main_agent.nameonresource.attributes), andgen_ai_inference_usage_dimension(input,output,cache_read_input,cache_write_input, orreasoning_outputforgen_ai.client.inference.usage.*metrics, falling back togen_ai.token.type) next to every point-value column, including theexemplarswhosetrace_idandspan_idlink a point back to a trace. The last two columns,time_unix_nanoandpoint_fingerprint, complete the identity of a point inotel_metrics_dedup(withtimestamp,service_name,metric_name, andmetric_type), so two points that share a microsecondtimestampstay distinguishable. -
v_gen_ai_metric_exemplars: unnestsexemplarsfromv_gen_ai_metrics, keeps exemplars with non-emptytrace_idandspan_id, and left-joinsotel_spans_dedupon(trace_id, span_id)and a null-safeservice_namematch within 24 hours before to 1 hour after the metric point'stimestampto expose exemplar latency/token values alongside span duration, status,span_gen_ai_agent_name(fromgen_ai.agent.nameorgcp.vertex.agent.nameon spanattributes,gen_ai.agent.nameon spanresource.attributes, thengen_ai.main_agent.nameon span or resourceattributes),span_gen_ai_tool_name,adk_session_id(fromgcp.vertex.agent.session_idthengen_ai.conversation.id), andspan_error_type. When multipleotel_spans_deduprows match an exemplar in that window (for example after a span name or start time update), the view selects one span row per exemplar ordered bys.end_time DESC, s.start_time DESC, s.receive_time DESC, s.name ASC. That choice is made per exemplar slot: one metric point, identified as inotel_metrics_dedup, and the exemplar's position in itsexemplarsarray, so distinct points that share a microsecondtimestamp, and repeated exemplars of one point, each keep their row. An exemplar whose span is absent or outside that window remains in the view withNULLspan_*columns. Likev_gen_ai_logs, it cannot be queried whenotel_spansrequires a partition filter, and startup then skips it with a warning unlessauto_create.allow_unprunable_viewsis set. - The two rollup views,
v_gen_ai_model_costsandv_gen_ai_sessions, readv_gen_ai_spansonce. A window function over that read finds the parents that report the sum of their children's tokens, and each view computes one column of its own before the window, so a filter onusage_hour(model costs) or onsession_start_date(sessions) narrows the scan ofotel_spansto the partitions it keeps. A filter onservice_name, or onsession_id, alone does not. See Filtering the GenAI views by time below.
-
-
Additive schema evolution and view reconciliation (
auto_create.tables). Whenauto_create.tablesistrue, startup reconciles existing tables that are missing newly introducedNULLABLEorREPEATEDcolumns (at top level or inside nestedRECORDfields) viaUpdateTablewith optimisticETagconcurrency. Incompatible type or mode drift is always rejected. Startup also keeps the_dedupviews, and withauto_create.companion_viewsthe companion views, in step with the exporter: a missing view is created, and an existing view whose SQL differs from the exporter's current definition (other than in whitespace) is updated in place throughUpdateTable, so an upgrade that changes a view's SQL needs no manual step. Only the view query is reconciled; labels and other view properties are left alone. These views have fixed names and belong to the exporter: a customized definition under one of those names is replaced at the next start, so keep a variation under another name. Reconciliation is best effort. A view that cannot be read or updated, for example because the identity lacksbigquery.tables.update, is kept as it is and logged as a warning, and export continues; a context cancellation or timeout still fails startup. -
Opt-in quarantine dead-letter table (
otel_quarantine). Whenquarantine.enabledistrue, spans, log records, and metric data points that fail validation, breach structural limits, exceedwrite.max_row_bytes, or are rejected per-row by BigQuery are routed tootel_quarantine(partitioned byDATE(ingest_time)with default 30-day partition expiration and clustered by(signal, error_code, service_name)). -
Partitioning and clustering. By default
otel_spansis hourly partitioned onPARTITION BY TIMESTAMP_TRUNC(start_time, HOUR)and clustered by(service_name, trace_id, name),otel_logsis daily partitioned onPARTITION BY DATE(timestamp)and clustered by(service_name, trace_id, severity_number), andotel_metricsis daily partitioned onPARTITION BY DATE(timestamp)and clustered by(service_name, metric_name, metric_type).traces.partition_granularity,logs.partition_granularity, andmetrics.partition_granularitycreate a table with the other granularity instead; the partition cap below applies to every table created with hourly partitions. -
Migrating from earlier tables.
v0alpha3(v1.0.0-candidate) drops*_schematype sidecars, per-rowschema_version, andrecord_fingerprint, moves schema versioning tootel_schema_*table labels, relaxesotel_logs.insert_idtoNULLABLE, and simplifiespayload_refsto the 4-field BigQueryObjectRefcolumnobject_refs. Create fresh tables and_dedupviews using the DDL ininternal/schema(orauto_create.tables: true); seedocs/migration.md.
For the full design rationale behind the table layout, identity synthesis, deduplication SQL, and rejected alternatives, see docs/schema-rationale.md.
- Hourly partition cap (10,000 partitions / ~416 days).
otel_spansusesPARTITION BY TIMESTAMP_TRUNC(start_time, HOUR)unless it was created withtraces.partition_granularity: DAY, andotel_logsandotel_metricsare partitioned by hour when created withlogs.partition_granularity: HOURormetrics.partition_granularity: HOUR(Terraform always partitionsotel_spansby hour andotel_metricsby day, andotel_logsbylogs_partition_type). Auto-created signal tables omit a default partition expiration unlesstraces.partition_expiration_days,logs.partition_expiration_days, ormetrics.partition_expiration_daysis configured, so telemetry is never dropped without operator consent. Because BigQuery enforces a limit of 10,000 partitions per table (about 416 days of hourly partitions), every hourly table retained longer than ~416 days needs a partition expiration: set that table'spartition_expiration_days(Terraformspans_partition_expiration_days,logs_partition_expiration_days, ormetrics_partition_expiration_days), optionally withrequire_partition_filter, or applyALTER TABLE:
ALTER TABLE `my-project.otel.otel_spans`
SET OPTIONS (partition_expiration_days = 365);- Trace-lookup search indexes (
NO_OP_ANALYZER).internal/schemaexportsSpansSearchIndexDDLandLogsSearchIndexDDLon(trace_id, span_id)for exact-token trace and span correlation lookups usingNO_OP_ANALYZER. Setauto_create.search_indexes: true(withauto_create.tables: trueandbigquery.jobs.creategranted to the exporter identity) to runCREATE SEARCH INDEX IF NOT EXISTSat startup, or provision them explicitly in BigQuery:
CREATE SEARCH INDEX IF NOT EXISTS `otel_spans_trace_lookup_idx`
ON `my-project.otel.otel_spans` (trace_id, span_id) OPTIONS (analyzer = 'NO_OP_ANALYZER');
CREATE SEARCH INDEX IF NOT EXISTS `otel_logs_trace_lookup_idx`
ON `my-project.otel.otel_logs` (trace_id, span_id) OPTIONS (analyzer = 'NO_OP_ANALYZER');
CREATE SEARCH INDEX IF NOT EXISTS `otel_metrics_trace_lookup_idx`
ON `my-project.otel.otel_metrics` (ALL COLUMNS) OPTIONS (analyzer = 'NO_OP_ANALYZER');-
Creation-time partitioning and dedup-window-bounded clustering. Set
logs.partition_granularity(DAYorHOUR, defaultDAY; Terraformlogs_partition_type) andlogs.clustering/traces.clustering(1..4columns) before first creating tables. BigQuery cannot change the time partitioning of an existing table, so a new granularity needs a new table; it can update an existing table's clustering columns in place (bq update --clustering_fields, thetables.updateAPI, or Terraform'slogs_clustering/spans_clustering), and clusters the data written after the update by the new columns. The exporter changes neither: when an existing table differs from the configured layout, startup logs aWARNthat says which of the two applies and continues without mutating the table. Clustering overrides are strictly bounded to each signal's deduplication viewPARTITION BYwindow (otel_logs_dedup:service_name,log_name,severity_number,trace_id,span_id,insert_id;otel_spans_dedup:service_name,trace_id,span_id,name), andJSONcolumns (attributes,body,resource) are rejected because BigQuery forbids clustering onJSON. -
Search index limits, create-only policy, and Day-1
BACKGROUNDreservation. BigQuery allows 1 search index and 1 analyzer (NO_OP_ANALYZER,LOG_ANALYZER, orPATTERN_ANALYZER) per base table and enforces a limit of 20 search index DDL operations per table per day. Both the exporter (auto_create.search_indexes) and Terraform (enable_search_indexes) treat search indexes as create-only (CREATE SEARCH INDEX IF NOT EXISTS); changingsearch_index.columnsorsearch_index.analyzernever drops or rebuilds an existing index. If another index already exists on the table, startup logs aWARNand succeeds, and Terraform's job skips the table after checkingINFORMATION_SCHEMA.SEARCH_INDEXES; any other job error fails the apply or the next plan, as Search Index Jobs describes. Free background search-index maintenance is capped at 20 TB logical base-table bytes per single region (or 100 TB org-wide inUS/EU); at ~20 TB/day regional ingestion, provision a dedicated BigQueryBACKGROUNDreservation (enable_background_reservation=truein Terraform) as a Day-1 prerequisite. -
Query acceleration rules for search indexes. BigQuery search indexes only accelerate
SEARCH,=,IN, prefixLIKE,STARTS_WITH, andENDS_WITHpredicates (not arbitrary wildcard/regex patterns), do not provide numeric range/equality indexing on decimal-stringint64JSON attributes, are not accelerated by BI Engine, and activate only once a base table reaches 10 GiB of logical bytes (total_logical_bytes >= 10 GiB). Furthermore, predicates on un-windowed columns such asbodyorattributesare not pushed belowotel_logs_dedup'sQUALIFY ROW_NUMBER()window—filterotel_logsdirectly or bound the scan on partition/dedup-window keys first. -
Why promoted typed columns are held out of
v1.0.0. The Storage Write API serializer (internal/protoenc/protoenc.go) binds strictly to the fixed table schema descriptor, leaving extra out-of-bandNULLABLEcolumnsNULL, whileinternal/otlpjson/anyvalue.goencodesint64attribute values as decimal JSON strings to preserve precision beyond$2^{53}-1$ . Extract promoted typed columns into downstream derived tables or scheduled views instead of modifying the ingestion table schema:
SELECT
timestamp,
service_name,
trace_id,
SAFE_CAST(JSON_VALUE(attributes, '$."http.response.status_code"') AS INT64) AS http_status_code,
JSON_VALUE(attributes, '$."http.request.method"') AS http_method
FROM `my-project.otel.otel_logs_dedup`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR);- Exporter-fallback log timestamps on retry. When a log record omits both
timestampandobserved_timestamp(0), a synthesized fallbackinsert_id(id1:...) remains deterministic across retries (the canonical record hash excludesreceive_time), but the row'stimestampcolumn falls back to the exporter's wall-clock ingestion time (timestamp_source = "exporter"). Becauseotel_logs_deduppartitions its deduplication window by(timestamp, time_unix_nano, service_name, trace_id, span_id, severity_number, log_name, insert_id), whole-batch retries that execute at a different wall-clock microsecond land in distincttimestampdeduplication groups and do not collapse—even if an explicitinsert_idattribute is set. Producers that require retry-stable log deduplication must populate OTLPtimestamporobserved_timestamp. - Trust model for
log_nameandinsert_idattributes.otel_logs_deduppartitions its deduplication window on(timestamp, time_unix_nano, service_name, trace_id, span_id, severity_number, log_name, insert_id)(addingtrace_id,span_id, andseverity_numberso filters on the base table's clustering and search-index columns push below the window). Whenlogs.log_name_attributeor the default well-knownlog_nameattributes (gcp.log_name,logging.googleapis.com/logName,log.iostream), orlogs.insert_id_attributeor the default well-knowninsert_idattributes (logging.googleapis.com/insertId,gcp.insert.id,gcp.insert_id,log.record.uid) are populated by upstream emitters, the pipeline trusts those emitters not to collide(log_name, insert_id)across distinct log records that share those window keys. Multi-tenant collectors accepting untrusted OTLP input should strip or namespace those attributes in a processor before export. - Filtering the GenAI views by time. A filter on
start_timereaches the scan ofotel_spansthroughotel_spans_dedupandv_gen_ai_spans. The rollup views readv_gen_ai_spansonce and expose the column to filter on:usage_hourforv_gen_ai_model_costsandsession_start_dateforv_gen_ai_sessions. In BigQuery dry runs over a small test table of 48 hourly partitions, the estimated bytes for oneusage_hourwere 1/48 of the estimate with no filter, and for one of the table's two UTC dates, 1/2; adding a filter onservice_nameto either, or filtering onservice_nameorsession_idalone, did not change the estimate. A dry run estimates an upper bound and does not show the benefit of clustering. Whentraces.require_partition_filteristrue, BigQuery rejects a query over a rollup that has no filter onusage_hour(orsession_start_date), and one that filters only onservice_nameorsession_id. A parent is matched with its children only within onePARTITION BYpartition of the rollup's window (see the notes under each rollup above), so a pair that straddles the boundary of a partition counts both spans. - Time-bounded
v_gen_ai_logsandv_gen_ai_metric_exemplarsspan enrichment.v_gen_ai_logsleft-joinsotel_logs_dedup(PARTITION BY DATE(timestamp)) withotel_spans_dedup(PARTITION BY TIMESTAMP_TRUNC(start_time, HOUR)) on(trace_id, span_id)and onservice_name, compared null-safely so that a log record and a span that both lack a service name still match, andv_gen_ai_metric_exemplarsjoinsUNNEST(v_gen_ai_metrics.exemplars)tootel_spans_dedupby the same rule. Both joins carry a range predicate onotel_spans.start_time: only a span that started from 24 hours before to 1 hour after the log or metric row'stimestampenriches the row. A row whose span started outside that window is still returned with its own fields, but without the columns that come from the span. The predicate comparesotel_spans.start_timewith each left-side row'stimestamp, and BigQuery does not use such a predicate to skip span partitions. In dry runs over ten daily partitions of test log rows, a one-day filter ontimestampcut the part of the estimate that comes fromotel_logsto a tenth, so the filter prunes daily partitions there, and left the part that comes fromotel_spansunchanged. No filter that BigQuery can use for partition elimination reaches the span scan, so queries overv_gen_ai_logsandv_gen_ai_metric_exemplarsare rejected whentraces.require_partition_filteristrue, with or without a filter ontimestampormetric_timestamp. So that a deployment is not left with views no query can use, startup skipsv_gen_ai_logsandv_gen_ai_metric_exemplars, as well asv_gen_ai_agent_trajectory, which joins each span to its parent and child spans on such a window, with a warning when theotel_spanstable it finds or creates requires a partition filter, and still creates every other companion view and starts exporting; setauto_create.allow_unprunable_views: true(Helm:bigquery.autoCreateAllowUnprunableViews: true) to create them anyway. The table's metadata decides, andtraces.require_partition_filtercounts only whileotel_spansdoes not exist yet, because it applies only when the table is created. Leavetraces.require_partition_filteroff for a deployment that queries those views, or joinotel_logs_dedup/v_gen_ai_metricsagainst astart_time-boundedotel_spans_dedupsubquery directly, which also lets a query choose its own window. - Real-time alerting (
deploy/continuous-queries/).deploy/continuous-queries/provides stateless per-span BigQuery Continuous Query templates (tool_error_rate_alert.sql,token_budget_burn.sql, andagent_latency_p95.sql) that evaluate newly appended rows onotel_spansviaAPPENDS(TABLE ...)and stream alerts to Cloud Pub/Sub (withINSERT INTOand PreviewTUMBLEwindow variants in that README). Each template includes a non-lossystart_time >= TIMESTAMP('1970-01-01 00:00:00+00')partition filter sotraces.require_partition_filteris satisfied without dropping late batches or quarantined replays.APPENDSreads base tables rather thanQUALIFYviews, and Continuous Queries do not support column-level security policy tags. Because exporter retries,APPENDSlookback replay, and BigQuery automatic reprocessing deliver alerts at-least-once, downstream consumers deduplicate onalert_id(<ALERT_KIND>:<trace_id>:<span_id>for per-span alerts, orTOOL_ERROR_WINDOW:<json_struct_key>viaTO_JSON_STRING(STRUCT(service_name, tool_name, window_start))in the windowed Preview example). Optional Terraform wiring indeploy/terraform(enable_continuous_queries=true) provisions theotel_alertstable, Pub/Sub topic, service account, IAM bindings, andCONTINUOUSreservation assignment.
Delivery is at-least-once. The exporter writes on the Storage Write API default stream, which is offset-free and requires no connection or stream-offset bookkeeping. A retried append can therefore land the same logical record twice.
This is a deliberate choice, not a limitation we are working around. Measured duplicate rates on the default stream are small, and read-time deduplication is the proportionate answer for the overwhelming majority of consumers. Exactly-once would mean committed-mode streams with per-stream offset management, lower default quotas and materially more server-side orchestration.
Every row carries a deterministic identity so duplicates can be collapsed at read time:
- Spans:
(start_time, start_time_unix_nano, service_name, trace_id, span_id). - Logs:
(timestamp, time_unix_nano, service_name, log_name, insert_id)whereinsert_id IS NOT NULL, resolved fromlogs.insert_id_attributeor well-known insert ID attributes (logging.googleapis.com/insertId,gcp.insert.id,gcp.insert_id,log.record.uid) and falling back to anid1:content hash whenlogs.synthesize_insert_idistrue(whenlogs.synthesize_insert_idisfalse,insert_idisNULLand retry deduplication is withdrawn for ID-less logs). - Metrics:
(timestamp, time_unix_nano, service_name, metric_name, metric_type, point_fingerprint, start_time_unix_nano).
The repo ships otel_spans_dedup, otel_logs_dedup, and otel_metrics_dedup
logical views so the default object an analyst touches is already
deduplicated. See internal/schema.
Each view's window function partitions on the row identity and also on every
column its base table is partitioned, clustered, or search-indexed on (the span
name, the log trace_id, span_id, and severity_number, and the metric
metric_type). Those columns are fixed for a record, so they never split the
retries of one, and a filter on any of them can be pushed below the window
function, so partition and cluster pruning can still apply through the view.
In BigQuery dry runs over a small test table, a filter on start_time cut the
estimate for otel_spans_dedup to the hourly partitions the filter kept. A dry
run does not show the benefit of clustering, and this repository has not
measured how many bytes pruning saves on a table of production size.
Known limit, stated rather than hidden: for logs without an explicit upstream
insert_idattribute and withlogs.synthesize_insert_id: true,insert_idfalls back to a compactid1:content hash, which includes anintra_batch_ordinalso identical records within the same batch remain distinct while retries of the batch reproduce the same IDs. Identical log records sent without an upstreaminsert_idin separate batches at the exact same nanosecond timestamp collapse into one; whenlogs.synthesize_insert_idisfalse, ID-less records emitinsert_id = NULL(otel_logs_deduppassesQUALIFY insert_id IS NULL OR ...) so retries are preserved rather than deduplicated.
- Materializing the dedup view (scheduled
MERGE). When per-query scan cost matters, use a bounded scheduledMERGErather than a non-incremental materialized view while keeping the_dedupview as the correctness backstop. See Materializing the dedup view (scheduled MERGE). - Quarantine and replay (
otel_quarantine). Whenquarantine.enabled: true, permanently rejected rows (contract violations,limits.*breaches, oversized rows, permanent offload errors) are routed toquarantine.tablefor inspection and replay viacmd/bq-quarantine-replay.
- Tier 1 — Raw
otel_logs/otel_logs_as_alllogs(high-volume search and aggregates): Query rawotel_logs(orotel_logs_as_alllogs) directly so partition, cluster, and(trace_id, span_id)search-index predicates reach base storage with zeroROW_NUMBER()sort barrier. Duplicate exposure on raw reads is bounded by uncertain stream retries and metered byotelcol_exporter_bigquery_uncertain_ack_replays. Keep fallback dedup limits in mind: separate-batch identical ID-less logs at identical nanoseconds collapse undersynthesize_insert_id: true, andtimestamp_source="exporter"rows receive a newreceive_time/timestampon retry. - Tier 2 — Bounded-window
otel_logs_dedup(exact point queries): Readotel_logs_dedupwith tighttimestamp/service_name/trace_id/log_namefilters when deduplicating retries with non-nullinsert_id. - Tier 3 — Incremental rollup materialization (
m_otel_log_volume_rollup.sqlx): For recurring dashboards and multi-TB daily volumes, materialize hourly non-GenAI counts viadeploy/dataform/definitions/marts/m_otel_log_volume_rollup.sqlxusing boundedDELETE + INSERTwindows. Never run full-tableMERGEover raw 20 TB/day log rows — rewriting full daily partitions doubles storage and slot scan costs. The rollup is opt-in (vars.enable_log_volume_rollup), carries its own Dataform tag (otel_log_marts) and run window, and reads the table named byvars.otel_logs_table; seedeploy/dataform/README.md.
timeout (exporterhelper default 5s) is the budget for one export attempt.
Within it a batch waits for the shared write stream, waits for request slots,
sends every request it splits into at write.max_request_bytes, and waits for
every acknowledgement.
- Slots.
write.stream_pool_size(default1, range1..16) opens that many default-streamManagedStreamslots per destination table, and sub-requests pick a slot whose current stream generation has open in-flight window capacity (breaking ties by fewest pending requests in round-robin order).write.max_inflight_requests(default 8) bounds unacknowledged requests per stream slot, whilewrite.max_inflight_bytes(default 64 MiB) bounds total in-flight bytes across the entire pool. The queue defaults to the same number of consumers as one slot's request window, so each can keep a request in flight without waiting. A batch that needs more requests than there are free slots drains its own earlier acknowledgements before waiting on anyone else's. Whenwrite.stream_pool_size > 1, each table's writer pool automatically usesNdedicated per-streamAppendRowsconnections and sizes the gRPC connection pool tomax(write.connection_pool_size, write.stream_pool_size). Whenwrite.enable_multiplexingisfalseandwrite.stream_pool_size > 1,write.connection_pool_sizemust be at leastwrite.stream_pool_sizeso simplexAppendRowsstreams do not share fewer gRPC channels than configured stream slots. - Adaptive backpressure (
write.backpressure.enabled). When enabled, AIMD congestion control steps the per-request byte split threshold down towardwrite.backpressure.min_request_bytesand the concurrent dispatch window down towardwrite.backpressure.min_inflight_requestsonquota_exhausted, multi-rowoversize_request,unavailable,uncertain_ack, or acknowledgements slower thanwrite.backpressure.latency_threshold, and steps them back up everywrite.backpressure.recovery_intervalon clean acknowledgements. While the reduced window is full, a batch waits for an in-flight slot for up to itstimeoutinstead of being shed early. Queued batches therefore drain during collectorShutdownand whenretry_on_failure.enabledisfalse, provided the reduced window clears the backlog withintimeout. A batch still waiting when itstimeoutexpires is handled as a timeout before dispatch, and one dispatched close to its deadline can end as a timeout after dispatch (both below). Under sustained overload, use a persistent queue (sending_queue.storage) to keep the backlog durably, including across restarts, while the window recovers. - Timeout before dispatch (waiting for the stream or a slot): nothing of that request was sent, and the stream is kept. If no earlier request of the batch was sent either, the batch goes back to exporterhelper as retryable.
- Timeout after dispatch: the rows may already be applied. The batch goes
back as
uncertain_ack(counted inuncertain_ack_replays), that slot's stream generation is retired while other slots in the pool keep serving, and the next attempt on that slot opens a new generation. Requests still in flight on the retired generation are handed back the same way. A replay can duplicate rows; the_dedupviews collapse them. - Split batches. A batch sent as several requests takes the verdict of the request that still owes the most. An uncertain request outranks a retryable one, and a retryable one outranks a permanent rejection. The batch is dropped only when every row that was not acknowledged was rejected permanently. A permanent rejection of one request does not stop the others from being sent. A replay resends every row of the batch that the service did not reject, including rows it already acknowledged.
- Bound on stuck results. At most two stream generations can be held at once: the current one, and retired ones whose results have not resolved. Both slots can hold retired generations. While both are taken, new attempts wait for either one to drain, within their own timeout, instead of opening more connections.
- Sizing & safe autoscaling. Keep
timeoutabove the acknowledgement latency of a full batch: one round trip perwrite.max_inflight_requestsrequests it splits into. Keepsending_queue.num_consumersat or belowwrite.max_inflight_requests; consumers beyond the window only wait for slots and spend their timeout doing so.retry_on_failure.max_elapsed_time(900sdefault) bounds how long a batch is replayed before it is dropped. When running Kubernetes HPA (deploy/helm/bigquery-otel-collector/templates/hpa.yamlandvalues-high-volume-logs.yaml), keepsending_queue.sizer: requests(sized for the 900s retry horizon, e.g.queue_size: 10000requests so single batches above 10,000 records are never rejected on an empty queue and 8 in-flight retry workers consume only 8 slots), enablesending_queue.batchexplicitly (off by default, withsizer: items, explicitmin_size: 0soexporterhelper's8192default never rejects a smallermax_sizewhenlogs.synthesize_insert_idisfalse, andsending_queue.batch.max_size: 0whenlogs.synthesize_insert_idistrueso pre-transform splits do not reset intra-batchinsert_idordinals), and scale out strictly on pod CPU utilization, admitted input rate (via a metrics adapter applyingrate()over the cumulative counterotelcol_receiver_accepted_log_records), and/orsending_queuedepth (otelcol_exporter_queue_size, reported insending_queue.sizerunits —requestsby default and invalues-high-volume-logs.yaml, wheretargetAverageValue: "500"is 500 requests) with a hardmaxReplicascap — never scale onotelcol_exporter_bigquery_backpressure_level(it rises on destination congestion or quota exhaustion, so scaling out adds writers against the same bottleneck; wirebackpressure_leveland quota warnings as alerts only). Pair multi-replica persistent queues with per-podStatefulSetPVCs (persistentQueue.enabled: true) and keep automatic HPA scale-down disabled (behavior.scaleDown.selectPolicy: Disabled):exporterhelperv0.165.0persistent queues stop reading immediately on podShutdownwithout draining, so confirmotelcol_exporter_queue_size == 0before manual scale-down (loweringautoscaling.maxReplicasbelow the active replica count or disablingautoscaling.enabledwhile scaled out bypassesselectPolicy: Disabledand scales down immediately, stranding unshipped PVC queue data unless replicas are drained first). In stateless mode (persistentQueue.enabled: false), automatic scale-in loses in-memory data during a destination outage: collector shutdown cancels retry backoff immediately, so a queued request whose export fails is dropped instead of retried as soon as the pod starts terminating, andterminationGracePeriodSecondsonly caps how long termination may take; it does not keep retrying until the grace period ends. Setautoscaling.behavior.scaleDown.selectPolicy: Disabledor enablepersistentQueue.enabled: truewhen zero-drop scale-in protection during outages is required.
What is validated. The internal/bigquery suite drives the real
managedwriter client against an in-process Storage Write service. With ten
concurrent senders on an eight-slot stream that never acknowledges and a 300 ms
deadline, every sender returns at its deadline, only eight requests reach the
stream, and the next append recovers on a new stream without a restart. The
cmd/bq-otel-testbench load harness (make bench-envelope) validates both
in-process bqfake pipeline scaling (write.stream_pool_size 1..16, reaching
27,896 spans/s, 28,983 logs/s, and 23,544 metric points/s at
stream_pool_size: 4, and peaking at 29,459 spans/s and 28,983 logs/s) and
live BigQuery Storage Write API throughput (10,404 spans/s / 28.72 MiB/s
encoded-row payload, 9,594 logs/s / 21.81 MiB/s encoded-row payload, and
9,185 metric points/s / 10.97 MiB/s encoded-row payload at
stream_pool_size: 4, p50 = 114.9–129.9 ms,
1.24–1.66 vCPU cores per 10,000 rows/s, 86.8 MiB process peak RSS). Full
tables and reproduction commands are recorded in
docs/benchmarks/v0.1.0-performance-envelope.md,
high-volume sizing and 5-layer byte accounting (20 TB/day = 231.5 MB/s = 600 TB/mo
vs. 300 MB/s = 25.92 TB/day) in
docs/operator-guides/load-testbench-and-sizing.md,
and monthly TCO formulas in docs/cost-estimation.md.
The least-privilege path for pre-created tables is a custom role centered on
bigquery.tables.updateData, plus bigquery.datasets.get and
bigquery.tables.get for startup validation. roles/bigquery.dataEditor is
the convenient broader alternative, not the least-privilege one. Dataset,
table, and view create permissions are needed only when the corresponding
auto_create flag is on. Both the metadata and Storage Write clients use the
same configured credentials or impersonated service-account identity.
Startup permission preflight is best-effort, not authoritative: a metadata
NotFound is authoritative for provisioning and fails fast when creation is
disabled. Other metadata failures prove neither absence nor write denial, so
they warn rather than blocking startup; the append reports the real write
authorization result.
| Mode | Required IAM permissions / role | Notes |
|---|---|---|
Steady-state export (auto_create off, least privilege) |
bigquery.datasets.get, bigquery.tables.get, bigquery.tables.updateData |
Or broader roles/bigquery.dataEditor. |
Auto-create dataset, tables, and views (auto_create.dataset, auto_create.tables, auto_create.companion_views) |
In addition to steady-state permissions: bigquery.tables.create, bigquery.tables.update (plus project-level bigquery.datasets.create only when auto_create.dataset: true) |
Upgrades metadata client scope to full BigQuery. |
Auto-create search indexes (auto_create.search_indexes) |
In addition to steady-state and table-creation permissions: bigquery.tables.createIndex, bigquery.jobs.create (project-level) |
Requires auto_create.tables: true. |
GCS payload offload (offload.enabled: true) |
storage.objects.create and storage.objects.get (roles/storage.objectCreator and roles/storage.objectViewer) on offload.bucket |
Conditional ifGenerationMatch=0 collisions (HTTP 412) read back existing object metadata to verify size/MD5 and obtain the generation; without storage.objects.get, repeated payloads fail permanently with HTTP 403. |
Non-goals by design:
- Exactly-once. See delivery semantics above.
- In-exporter aggregation or sampling. Use the Collector's processors for pre-export aggregation or sampling; the companion rollup views (
v_gen_ai_model_costs,v_gen_ai_sessions,v_gen_ai_tool_usage) are SQL views in BigQuery. - GCS batch
LOADinsidebigqueryexporter. Storage Write API is the single export path (one schema, one exporter); stock OTelgooglecloudstorageexporter+ cronbq loadcannot satisfy the flattened row,insert_id, intra-batch ordinal, orobject_refscontracts, and shared-poolLOADhas no SLA under BigQuery's1,500 jobs/table/daylimit (including failures). See the standalone post-v1specification indocs/architecture/gcs-batch-row-loader.md.
Operational limits and common misuse:
- At-least-once delivery and
_dedupviews. Appends go through Storage Write API default streams (AppendRows), so transient retries or uncertain acknowledgements can leave duplicate rows inotel_spans,otel_logs, orotel_metrics. Read the companion_dedupviews (otel_spans_dedup,otel_logs_dedup,otel_metrics_dedup) whenever query correctness requires deduplicated rows, rather than scanning the raw base tables directly (and include partition predicates so_dedupviews stay bounded). - Persistent-queue crash replays. When
sending_queue.storageis enabled, unacknowledged batches replayed from disk after a Collector crash or unclean shutdown can produce duplicates outside the in-process uncertain-ack duplicate bound because batches already committed on the server may be re-read before the queue cursor advances. - Structural rejections drop rows unless quarantine is enabled. Records rejected by
limits.*bounds (limits.max_attribute_count,limits.max_event_count,limits.max_link_count,limits.max_nesting_depth,limits.max_collection_size,limits.max_attribute_key_bytes,limits.max_value_bytes), serialized row-size bounds (write.max_row_bytes), or server row-level errors are dropped and counted viarejected_rowsunlessquarantine.enabled: trueis configured to write permanent failures toquarantine.table(best-effort withinquarantine.timeout). - Storage Write API quotas,
write.stream_pool_size, andBACKGROUNDslots.write.stream_pool_size(1..16, default1, experimental) opensNdefault-stream writers per destination signal table (otel_spans,otel_logs,otel_metrics), andquarantine.enabled: trueopens a separate client and stream forquarantine.table. Increasingwrite.stream_pool_sizeor enabling all three signals multiplies openAppendRowsstreams, connections, and throughput against Storage Write API quotas: defaults are300 MB/sin single regions (20 TB/day = 231.5 MB/sfits within default headroom) and3 GB/sinUS/EUmulti-regions, scoped to the destination project for throughput and the client project for concurrent connections (write.compression: gzipreduces network transport bytes only, never billed ingestion bytes or throughput quota). Whenwrite.enable_multiplexingisfalseandwrite.stream_pool_size > 1,write.connection_pool_sizemust be at leastwrite.stream_pool_size. For search-indexed tables exceeding20 TB(single-region) or100 TB(US/EU) org-wide logical base-table size, provision a Day-1BACKGROUNDslot reservation. - Row and append request size limits. Live BigQuery
_defaultstreamAppendRowsenforces a20 MBper-request limit, whileConfig.Validate()enforces a conservative10 MB(10,000,000-byte)apiRequestLimitBytesheadroom threshold (write.max_request_bytesdefaults to8388608/8 MiB),write.max_row_bytes <= write.max_request_bytes, andwrite.max_inflight_bytes >= write.max_request_bytes. Individual serialized rows abovewrite.max_row_bytesare rejected rather than crashing the batch; enableoffload.enabled: truewhen exporting large LLM prompts, tool responses, or log bodies. - Control-plane IAM for
auto_createvs. steady-state least privilege. Steady-state exporting withauto_create.*disabled needs only read metadata (bigquery.datasets.get,bigquery.tables.get, read scope) andbigquery.tables.updateData. Enablingauto_create.dataset,auto_create.tables,auto_create.companion_views, orauto_create.search_indexesupgrades the metadata client scope to full BigQuery (https://www.googleapis.com/auth/bigquery) and requires control-plane creation/update permissions (bigquery.datasets.create,bigquery.tables.create,bigquery.tables.update,bigquery.tables.createIndex,bigquery.jobs.create). Pre-create tables with Terraform or SQL DDL and turnauto_createoff in production. - Loopback guard on
endpoint.insecureandendpoint.without_authentication.Config.Validate()rejectsendpoint.insecure: trueandendpoint.without_authentication: trueunlessendpoint.urlis set and matcheslocalhostor a literal loopback IP (it does not resolve hostnames via DNS), and rejects combiningendpoint.without_authentication: truewithauto_create.dataset: true,auto_create.tables: true, orauto_create.search_indexes: true. - Cost drivers. Primary cost drivers are Storage Write API ingestion volume (beyond the monthly free tier), BigQuery active/long-term storage and search indexes, Cloud Storage class-A writes and object storage when
offload.enabledis on, and unpruned queries lacking partition filters (start_timeonotel_spans,timestamponotel_logsandotel_metrics,ingest_timeonotel_quarantine). See docs/cost-estimation.md for pricing formulas,PHYSICALvs.LOGICALstorage billing, and rollup views. - Redaction is off by default. Client-side PII/secret scrubbing (
redaction.enabled) defaults tofalse. Enableredaction.enabled: true(and configureredaction.redact_log_body,redaction.blocked_keys,redaction.blocked_value_patterns,redaction.disable_default_keys, andredaction.mask_replacementas needed) before shipping unscreened prompts, secrets, or personal data.
This exporter registers no component-specific feature gates.
Queue batching and persistent-queue behavior depend on the OpenTelemetry Collector core pkg.exporterhelper.queueBatchEnabled feature gate rather than an exporter-owned gate: the exporter defaults sending_queue.num_consumers to 8 (matching the default-stream write.max_inflight_requests ceiling) with sending_queue.batch (configoptional.Optional[queuebatch.BatchConfig]) left unset so batching is disabled by default unless pkg.exporterhelper.queueBatchEnabled is enabled or sending_queue.batch is configured. If you configure sending_queue.batch, check the Collector distribution's state for pkg.exporterhelper.queueBatchEnabled.
See documentation.md for the mdatagen-generated telemetry reference.
Use these content-free self-telemetry instruments (declared in metadata.yaml and internal/metadata/telemetry.go) to detect a stuck path:
| Metric | Type | Unit | Attributes | Meaning |
|---|---|---|---|---|
otelcol_exporter_bigquery_acknowledged_rows |
Counter (int) |
{row} |
none |
Rows acknowledged by BigQuery Storage Write API AppendRows. |
otelcol_exporter_bigquery_append_bytes |
Counter (int) |
By |
none |
Serialized protobuf bytes sent to AppendRows. |
otelcol_exporter_bigquery_append_result_wait |
Histogram (double) |
s |
none |
Dispatch-to-terminal wait duration for BigQuery AppendRows results, in seconds. |
otelcol_exporter_bigquery_backpressure_events |
Counter (int) |
{event} |
{signal, table, trigger} |
Count of adaptive AIMD step-down and recovery transitions. |
otelcol_exporter_bigquery_backpressure_level |
Gauge (double) |
1 |
{signal, table} |
Current adaptive AIMD backpressure level in [0.0, 1.0] (0.0 at ceiling, 1.0 at floor). |
otelcol_exporter_bigquery_effective_request_bytes |
Gauge (int) |
By |
{signal, table} |
Current effective per-request byte split threshold (write.max_request_bytes when uncongested). |
otelcol_exporter_bigquery_inflight_requests |
UpDownCounter (int) |
{request} |
none |
Dispatched AppendRows results not yet terminal. |
otelcol_exporter_bigquery_offload_dedup_hits |
Counter (int) |
{hit} |
{signal} |
Offload attempts satisfied from the intra-batch coalescing cache or cross-batch generation LRU cache (offload.cache_size) instead of uploading. |
otelcol_exporter_bigquery_offload_errors |
Counter (int) |
{error} |
{class} |
Object storage offload failures grouped by failure class. |
otelcol_exporter_bigquery_offloaded_bytes |
Counter (int) |
By |
none |
Bytes offloaded to object storage. |
otelcol_exporter_bigquery_offloaded_payloads |
Counter (int) |
{payload} |
none |
Payloads offloaded to object storage. |
otelcol_exporter_bigquery_quarantine_failed_rows |
Counter (int) |
{row} |
{reason} |
Quarantine rows that could not be written to the quarantine table. |
otelcol_exporter_bigquery_quarantined_rows |
Counter (int) |
{row} |
{reason} |
Rows written to the quarantine table (otel_quarantine). |
otelcol_exporter_bigquery_redacted_values |
Counter (int) |
{value} |
{match_type, signal} |
Attribute keys and string values masked by redaction; never records key names or values. |
otelcol_exporter_bigquery_rejected_rows |
Counter (int) |
{row} |
{reason} |
Rows rejected by BigQuery or validation, grouped by rejection reason. |
otelcol_exporter_bigquery_retries |
Counter (int) |
{failure} |
{class} |
Retried export failures grouped by retry classification. |
otelcol_exporter_bigquery_stream_retirements |
Counter (int) |
{retirement} |
none |
Unsafe stream generations retired for recovery. |
otelcol_exporter_bigquery_timeouts_after_dispatch |
Counter (int) |
{timeout} |
none |
Export attempts whose context ended after dispatch began. |
otelcol_exporter_bigquery_uncertain_ack_replays |
Counter (int) |
{replay} |
none |
Write chunks returned for caller replay after uncertain stream teardown. |
otelcol_exporter_bigquery_unresolved_results |
UpDownCounter (int) |
{result} |
none |
Results still being drained after their original attempt ended. |
Of the four non-counter current-value instruments, inflight_requests and
unresolved_results (UpDownCounter) should return to zero after recovery or
clean shutdown. When write.backpressure.enabled is true,
backpressure_level (dimensionless 1 gauge in [0.0, 1.0]) recovers toward
0.0 and effective_request_bytes (byte gauge) recovers to
write.max_request_bytes, not zero. A sustained nonzero inflight_requests or
unresolved_results count, rising timeout/retirement/uncertain-ack counters, or
queue drops indicates “accepted by the collector, not yet confirmed by
BigQuery.”
The managedwriter client and stream use an exporter-owned lifecycle context;
per-attempt timeouts are used only for dispatch/result waits. If an attempt
ends after dispatch, that stream generation is retired before exporterhelper
replays the batch, and every returned AppendResult keeps a lifecycle owner
until it resolves or the exporter shuts down. This is what lets a later append
recover without restarting the collector.
See CONTRIBUTING.md for the development workflow, layout conventions, compatibility guarantees, and Contributor License Agreement.
To run the live-BigQuery integration test suite locally against a scratch project with Application Default Credentials (see CONTRIBUTING.md for full prerequisites):
BQ_PROJECT=my-project BQ_LOCATION=US make integration-test-
make bench-enveloperunscmd/bq-otel-testbenchacrosswrite.stream_pool_sizevalues1, 2, 4, 8, 16for traces, logs, and metrics againstbqfake(or live BigQuery whenBENCH_MODE=live BQ_PROJECT=<project>is set) and prints a Markdown throughput, latency, memory, and self-telemetry summary table. -
make fuzzruns every fuzz target in the module forFUZZTIME(default30s) each, for examplemake fuzz FUZZTIME=2m. It finds the targets withgo test -list, so a new target needs no Makefile change. -
make bench-compareruns the benchmarksBENCH_COUNTtimes (default 6) atBASE_REF(defaultHEAD~1) and in the working tree, and compares the two runs withbenchstat.make bench-compare BASE=old.txt HEAD=new.txtcompares two savedgo test -benchoutputs instead. On pull requests, CI writes the same comparison against the base branch to the job summary and warns about benchmarks more than 20% slower, without failing the build. -
To add a fuzz target, list it under
jobs.fuzz.strategy.matrix.includein.github/workflows/bench-fuzz.yml, and commit at least one seed input under<package>/testdata/fuzz/<FuzzTarget>/: ago test fuzz v1header line followed by one Go literal per argument, as in the existing files.TestBenchFuzzWorkflowListsEveryFuzzTargetfails until both are in place. It also fails if the target's file has a//go:buildor// +buildline, because CI fuzzes without build tags and would not compile the target. When CI fuzzing fails, the job uploads the failing input as thefuzz-corpus-<FuzzTarget>artifact; unzip it at the repository root and rungo test ./<package> -run=<FuzzTarget>to reproduce the failure.
- SUPPORT.md: where to ask for help and what support covers.
- CONTRIBUTING.md: the development workflow and the Contributor License Agreement.
- SECURITY.md: how to report a vulnerability privately.
- CODE_OF_CONDUCT.md: the standards everyone taking part is expected to follow.
- LICENSE: the Apache License, Version 2.0.