Skip to content

feat: add victoriametrics sink - #26479

Open
missedone wants to merge 12 commits into
vectordotdev:masterfrom
neotags:feat/victoriametrics
Open

missedone wants to merge 12 commits into
vectordotdev:masterfrom
neotags:feat/victoriametrics

Conversation

@missedone

Copy link
Copy Markdown

Summary

Adds a victoriametrics sink that sends metrics using the
VictoriaMetrics remote write protocol:
a Prometheus WriteRequest protobuf compressed with zstd, as sent by vmagent. The design is
in the RFC included in this PR (rfcs/2026-09-24-1343-victoriametrics-sink.md).

prometheus_remote_write works with VictoriaMetrics today, but it falls short in several ways:

  • tenant_id sets X-Scope-OrgID, which VictoriaMetrics ignores, so cluster tenants need one
    sink per tenant.
  • Distributions are bucketed into a fixed buckets list, which is imprecise and doesn't line up
    with the vmrange histograms that VictoriaMetrics client libraries emit.
  • Incremental distributions keep every sample in the normalizer (MetricValue::add concatenates
    samples), so memory and CPU grow without bound (likely High CPU usage on prometheus_remote_write sink #19041).
  • The default healthcheck fails against VictoriaMetrics (healthcheck.uri option for prometheus_remote_write sink  #13890).

What the sink does:

  • Protocol: zstd with X-VictoriaMetrics-Remote-Write-Version: 1. The protobuf is encoded
    directly into a reused buffer, with no intermediate prost structs or grouping map, and each
    request is compressed in a single zstd call, so the frame records its content size, as vmagent
    does.

  • Request size: batch.max_bytes limits the estimated encoded size, 8 MiB by default like
    -remoteWrite.maxBlockSize, so distributions that expand into hundreds of series can't push a
    request over -maxInsertRequestSize.

  • Distributions: encoded as vmrange histograms (18 log-scale buckets per power of ten), using
    the bucket layout of VictoriaMetrics/metrics. The labels are checked byte for byte against
    golden data generated from the Go implementation. Incremental distributions are accumulated in
    bounded, sink-owned state: at most 488 counters per series, expired by expire_metrics_secs.

  • Sketches: sent as summaries with quantiles plus _sum, _count, _min and _max, the way
    VictoriaMetrics stores Datadog sketches.

  • Tenancy, set by tenant.mode:

    • none: single-node VictoriaMetrics, or vmauth.
    • path: /insert/<accountID:projectID>/prometheus/api/v1/write, one batch per tenant.
    • labels: vm_account_id/vm_project_id labels, with the tenant also set in the
      MetricMetadata fields vminsert reads.

    tenant.id is subject to template confinement, and the rendered value must be a valid numeric
    tenant, so event data can't change the URL path.

  • Per-event credentials for vmauth: an event secret victoriametrics_token is sent as a bearer
    token in place of the configured auth, following the datadog_api_key / splunk_hec_token
    pattern. Batches are split per token.

  • Retries match vmagent: requests are dropped only on 400, 409 and 415; every other failure is
    retried, including 401/403 from vmauth while credentials are rotated.

  • Healthcheck writes an empty request to the write path, which checks reachability,
    authentication and vmauth routing, not just /health.

  • Metadata is sent by default (send_metadata: true), matching -enableMetadata.

The commits are grouped for review: RFC, sink and unit tests, integration test environment and
CI, then docs.

References

Closes: #1343
Related: #13890, #19041, #23990

Vector configuration

    sources:
      statsd:
        type: statsd
        address: 0.0.0.0:8125
        mode: udp

    transforms:
      pick_token:
        type: remap
        inputs: [statsd]
        source: |
          set_secret("victoriametrics_token", get_env_var!("VM_TOKEN_" + upcase!(.tags.team)))

    sinks:
      vm_single:
        type: victoriametrics
        inputs: [statsd]
        endpoint: http://victoriametrics:8428

      vm_cluster:
        type: victoriametrics
        inputs: [statsd]
        endpoint: http://vminsert:8480
        tenant:
          mode: path
          id: "42:{{ tags.project }}"

      vm_vmauth:
        type: victoriametrics
        inputs: [pick_token]
        endpoint: http://vmauth:8427
        auth:
          strategy: bearer
          token: "${VM_TOKEN_DEFAULT}"

How did you test this PR?

  • Unit and end-to-end tests (49) in src/sinks/victoriametrics/:

    • the encoder is checked against prost decoding;
    • the vmrange labels and bucket placement against golden data from Go;
    • normalization memory stays bounded;
    • tenant parsing and routing, token override and batch splitting;
    • the size limit;
    • retry classification;
    • the healthcheck.
  • Integration tests (cargo vdev int test victoriametrics) cover:

    • single-node VictoriaMetrics;
    • a vmstorage/vminsert/vmselect cluster, with path and label tenants;
    • vmauth routing two tokens to different tenants.

    Data is read back with /api/v1/export.

  • Checked against the vmagent remote write client and the vminsert remote write parser in the
    VictoriaMetrics source (headers, block limits, status handling, tenant parsing, metadata tenant
    fields).

  • make fmt, clippy, cargo vdev check events, make check-docs, generated docs, changelog
    fragment check, typos, markdownlint.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

@missedone
missedone requested review from a team as code owners September 24, 2026 22:57
@github-actions github-actions Bot added docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: sinks Anything related to Vector's sinks domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: rfc labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@missedone

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@missedone missedone changed the title Feat/victoriametrics feat: add victoriametrics sink Sep 24, 2026
@missedone
missedone marked this pull request as draft September 25, 2026 06:37
Record the design of a `victoriametrics` sink for vectordotdev#1343: the
VictoriaMetrics remote write protocol (zstd), `vmrange` histograms for
distributions, bounded state for incremental distributions, tenancy for
single-node, vminsert, and vmauth, and retry behavior matching vmagent.
Add a sink that delivers metrics to VictoriaMetrics using the
VictoriaMetrics remote write protocol, matching the vmagent remote
write client:

- Encode `WriteRequest` protobuf directly into a reused buffer and
  compress each request with a single zstd call, so the frame records
  its content size. Labels are written sorted by name.
- Bound batches by estimated encoded size, 8 MiB by default like
  `-remoteWrite.maxBlockSize`, so requests stay below
  `-maxInsertRequestSize`.
- Encode distributions as `vmrange` histograms using the bucket layout
  of VictoriaMetrics/metrics, verified against golden data generated
  from the Go implementation, and accumulate incremental distributions
  in bounded, sink-owned state.
- Encode sketches as summaries with `_sum`, `_count`, `_min`, and
  `_max`, as VictoriaMetrics does for Datadog sketches.
- Support single-node VictoriaMetrics, cluster tenants in the vminsert
  URL path or in `vm_account_id`/`vm_project_id` labels (including the
  metadata tenant fields), and per-event vmauth credentials through the
  `victoriametrics_token` event secret.
- Drop requests only on 400, 409, and 415 and retry everything else,
  and healthcheck by writing an empty request to the write path.
- Send metric metadata by default, matching `-enableMetadata`.

Closes vectordotdev#1343
Run the `victoriametrics` sink integration tests against single-node
VictoriaMetrics, a vmstorage/vminsert/vmselect cluster, and vmauth
routing bearer tokens to tenants.
Add the component reference, generated configuration docs, example
configurations, website page, and changelog fragment for the
`victoriametrics` sink.
@missedone
missedone marked this pull request as ready for review September 29, 2026 20:06
@missedone
missedone requested a review from a team as a code owner October 17, 2026 20:53
@github-actions github-actions Bot added domain: topology Anything related to Vector's topology code meta: breaking change Anything that breaks backward compatibility. domain: sources Anything related to Vector's sources domain: transforms Anything related to Vector's transform components domain: releasing Anything related to releasing Vector domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: vdev Anything related to the vdev tooling labels Oct 17, 2026
@missedone
missedone force-pushed the feat/victoriametrics branch from f462b39 to 8b45bbf Compare October 17, 2026 20:55
@github-actions github-actions Bot removed domain: topology Anything related to Vector's topology code meta: breaking change Anything that breaks backward compatibility. domain: sources Anything related to Vector's sources labels Oct 17, 2026
@github-actions github-actions Bot removed domain: transforms Anything related to Vector's transform components domain: releasing Anything related to releasing Vector domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: vdev Anything related to the vdev tooling labels Oct 17, 2026
@missedone
missedone force-pushed the feat/victoriametrics branch from 8b45bbf to 10b895f Compare October 19, 2026 19:07
@missedone
missedone force-pushed the feat/victoriametrics branch from 10b895f to 34508e2 Compare October 19, 2026 19:09
Retry body read failures, route tenants before normalizing, reject
reserved headers, skip empty sketches, key metadata by name and type,
tighten TTL cleanup, drop the tenant from the endpoint tag, and share
histogram snapshots.
Match vmagent by dropping labels with empty values, and cover retry
behaviour and encoding edge cases end to end.
Add vmagent and a reference VictoriaMetrics to the integration
environment, and a test that the sink stores the same series, labels,
values and timestamps as vmagent for the same data.
Allow the VictoriaMetrics identifier in docs, justify the intentional
numeric casts, use let-else in tenant routing, split the vmagent
integration test, and point the generated acknowledgements schema at
the bool_or_struct definition like other sinks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: rfc domain: sinks Anything related to Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add VictoriaMetrics metrics sink

1 participant