How to send metrics from kafka to prometheus remote write?

[ernado]

Question

Hello,

I'm trying to use kafka as a transport for metrics.

Schema is as follows:

Source -[PRW]-> vector -> kafka -> vector -[PRW]-> Target

But I'm getting Configuration error for last element in chain, where vector is consumer for kafka and should push metrics via Prometheus Remote Write.

Is it even possible to achieve?

Thank you in advance.

Vector Config

sources:
  kafka:
    type: "kafka"
    bootstrap_servers: "kafka:9092"
    group_id: "vector-aggregator-group"
    topics: ["vector-metrics"]

sinks:
  prw:
    type: prometheus_remote_write
    inputs: ["kafka"]
    endpoint: http://vm:8428/api/v1/write

Vector Logs

2026-02-07T19:04:37.695229Z  INFO vector::app: Log level is enabled. level="info"
2026-02-07T19:04:37.697334Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2026-02-07T19:04:37.698571Z ERROR vector::cli: Configuration error. error=Data type mismatch between kafka (["Log"]) and prw (["Metric"]) internal_log_rate_limit=fals

[pront]

Yes. The Kafka source defaults to decoding records as logs, which causes that type mismatch. Use matching native_json encoding and decoding to preserve Vector's metric event type.

Producer Vector:

sources:
  prw:
    type: prometheus_remote_write
    address: 0.0.0.0:9090
    path: /api/v1/write

sinks:
  kafka:
    type: kafka
    inputs: [prw]
    bootstrap_servers: kafka:9092
    topic: vector-metrics
    encoding:
      codec: native_json

Consumer Vector:

sources:
  kafka:
    type: kafka
    bootstrap_servers: kafka:9092
    group_id: vector-aggregator-group
    topics: [vector-metrics]
    auto_offset_reset: earliest
    decoding:
      codec: native_json

sinks:
  prw:
    type: prometheus_remote_write
    inputs: [kafka]
    endpoint: http://vm:8428/api/v1/write

Both sides must use native_json; existing records encoded as ordinary json are not in this format. The native JSON codec supports metric events and is currently documented as experimental.

Validated on Vector 0.60.0 through Remote Write → Vector → a librdkafka mock broker → Vector → a Vector Remote Write receiver. Two gauge samples retained their names, labels, timestamps, and values. This did not use a production Kafka cluster or VictoriaMetrics.

Please don't hesitate to followup in this thread if you have further questions.