Skip to content

out_prometheus_remote_write: support zstd compression #10587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

out_prometheus_remote_write: support zstd compression #10587

wants to merge 1 commit into from
+13 −1

Conversation

johejo
Copy link

@johejo johejo commented Jul 12, 2025 •
edited
Loading

In #7318, gzip was supported. This PR adds similar change like gzip to support zstd .

Addresses #7239

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change

Run VictoriaMetrics on port 18428 and node_exporter on port 9100.

$ victoria-metrics -httpListenAddr=:18428

$ node_exporter --web.listen-address=:9100

fluent-bit.yaml

pipeline:
  inputs:
    - name: prometheus_scrape
      host: localhost
      port: 9100
  outputs:
    - name: prometheus_remote_write
      match: "*"
      host: localhost
      port: 18428
      uri: /api/v1/write
      compression: zstd
  • Debug log output from testing the change
$ ./bin/fluent-bit --config ./tmp/fluent-bit.yaml -v
Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/07/12 18:14:05] [ info] Configuration:
[2025/07/12 18:14:05] [ info]  flush time     | 1.000000 seconds
[2025/07/12 18:14:05] [ info]  grace          | 5 seconds
[2025/07/12 18:14:05] [ info]  daemon         | 0
[2025/07/12 18:14:05] [ info] ___________
[2025/07/12 18:14:05] [ info]  inputs:
[2025/07/12 18:14:05] [ info]      prometheus_scrape
[2025/07/12 18:14:05] [ info] ___________
[2025/07/12 18:14:05] [ info]  filters:
[2025/07/12 18:14:05] [ info] ___________
[2025/07/12 18:14:05] [ info]  outputs:
[2025/07/12 18:14:05] [ info]      prometheus_remote_write.0
[2025/07/12 18:14:05] [ info] ___________
[2025/07/12 18:14:05] [ info]  collectors:
[2025/07/12 18:14:05] [ info] [fluent bit] version=4.0.5, commit=22d3d60ba1, pid=111918
[2025/07/12 18:14:05] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/07/12 18:14:05] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/07/12 18:14:05] [ info] [simd    ] disabled
[2025/07/12 18:14:05] [ info] [cmetrics] version=1.0.4
[2025/07/12 18:14:05] [ info] [ctraces ] version=0.6.6
[2025/07/12 18:14:05] [ info] [input:prometheus_scrape:prometheus_scrape.0] initializing
[2025/07/12 18:14:05] [ info] [input:prometheus_scrape:prometheus_scrape.0] storage_strategy='memory' (memory only)
[2025/07/12 18:14:05] [debug] [prometheus_scrape:prometheus_scrape.0] created event channels: read=25 write=26
[2025/07/12 18:14:05] [debug] [prometheus_remote_write:prometheus_remote_write.0] created event channels: read=27 write=28
[2025/07/12 18:14:05] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] worker #0 started
[2025/07/12 18:14:05] [ info] [sp] stream processor started
[2025/07/12 18:14:05] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/07/12 18:14:05] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] worker #1 started
[2025/07/12 18:14:14] [debug] [http_client] not using http_proxy for header
[2025/07/12 18:14:14] [debug] [input coro] destroy coro_id=0
[2025/07/12 18:14:15] [debug] [task] created task=0x7fc43c05bf80 id=0 OK
[2025/07/12 18:14:15] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #0
[2025/07/12 18:14:15] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 116378
[2025/07/12 18:14:15] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-116378 payload_size=90305
[2025/07/12 18:14:15] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 90305
[2025/07/12 18:14:15] [debug] [upstream] KA connection #59 to localhost:18428 is connected
[2025/07/12 18:14:15] [debug] [http_client] not using http_proxy for header
[2025/07/12 18:14:15] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] localhost:18428, HTTP status=204
[2025/07/12 18:14:15] [debug] [upstream] KA connection #59 to localhost:18428 is now available
[2025/07/12 18:14:15] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK
[2025/07/12 18:14:15] [debug] [out flush] cb_destroy coro_id=0
[2025/07/12 18:14:15] [debug] [task] destroy task=0x7fc43c05bf80 (task_id=0)
^C[2025/07/12 18:14:21] [engine] caught signal (SIGINT)
[2025/07/12 18:14:21] [ warn] [engine] service will shutdown in max 5 seconds
[2025/07/12 18:14:21] [ info] [engine] service has stopped (0 pending tasks)
[2025/07/12 18:14:21] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] thread worker #0 stopping...
[2025/07/12 18:14:21] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] thread worker #0 stopped
[2025/07/12 18:14:21] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] thread worker #1 stopping...
[2025/07/12 18:14:21] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] thread worker #1 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
$ valgrind --leak-check=full ./bin/fluent-bit --config ./tmp/fluent-bit.yaml -v
==3051== Memcheck, a memory error detector
==3051== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==3051== Using Valgrind-3.25.1 and LibVEX; rerun with -h for copyright info
==3051== Command: ./bin/fluent-bit --config ./tmp/fluent-bit.yaml -v
==3051==
Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/07/12 18:24:19] [ info] Configuration:
[2025/07/12 18:24:19] [ info]  flush time     | 1.000000 seconds
[2025/07/12 18:24:19] [ info]  grace          | 5 seconds
[2025/07/12 18:24:19] [ info]  daemon         | 0
[2025/07/12 18:24:19] [ info] ___________
[2025/07/12 18:24:19] [ info]  inputs:
[2025/07/12 18:24:19] [ info]      prometheus_scrape
[2025/07/12 18:24:19] [ info] ___________
[2025/07/12 18:24:19] [ info]  filters:
[2025/07/12 18:24:19] [ info] ___________
[2025/07/12 18:24:19] [ info]  outputs:
[2025/07/12 18:24:19] [ info]      prometheus_remote_write.0
[2025/07/12 18:24:19] [ info] ___________
[2025/07/12 18:24:19] [ info]  collectors:
[2025/07/12 18:24:19] [ info] [fluent bit] version=4.0.5, commit=22d3d60ba1, pid=3051
[2025/07/12 18:24:19] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/07/12 18:24:19] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] worker #1 started
[2025/07/12 18:24:19] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/07/12 18:24:19] [ info] [simd    ] disabled
[2025/07/12 18:24:19] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] worker #0 started
[2025/07/12 18:24:19] [ info] [cmetrics] version=1.0.4
[2025/07/12 18:24:19] [ info] [ctraces ] version=0.6.6
[2025/07/12 18:24:19] [ info] [input:prometheus_scrape:prometheus_scrape.0] initializing
[2025/07/12 18:24:19] [ info] [input:prometheus_scrape:prometheus_scrape.0] storage_strategy='memory' (memory only)
[2025/07/12 18:24:19] [debug] [prometheus_scrape:prometheus_scrape.0] created event channels: read=25 write=26
[2025/07/12 18:24:19] [debug] [prometheus_remote_write:prometheus_remote_write.0] created event channels: read=27 write=28
[2025/07/12 18:24:19] [ info] [sp] stream processor started
[2025/07/12 18:24:19] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
==3051== Warning: client switching stacks?  SP change: 0x60de408 --> 0x55b2460
==3051==          to suppress, use: --max-stackframe=11714472 or greater
==3051== Warning: client switching stacks?  SP change: 0x55b2438 --> 0x60de408
==3051==          to suppress, use: --max-stackframe=11714512 or greater
==3051== Warning: client switching stacks?  SP change: 0x60de408 --> 0x55b2438
==3051==          to suppress, use: --max-stackframe=11714512 or greater
==3051==          further instances of this message will not be shown.
[2025/07/12 18:24:29] [debug] [http_client] not using http_proxy for header
[2025/07/12 18:24:29] [debug] [input coro] destroy coro_id=0
[2025/07/12 18:24:29] [debug] [task] created task=0x79fdda0 id=0 OK
[2025/07/12 18:24:29] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #0
[2025/07/12 18:24:29] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 116376
[2025/07/12 18:24:30] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-116376 payload_size=90071
[2025/07/12 18:24:30] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 90071
[2025/07/12 18:24:30] [debug] [upstream] KA connection #59 to localhost:18428 is connected
[2025/07/12 18:24:30] [debug] [http_client] not using http_proxy for header
[2025/07/12 18:24:30] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] localhost:18428, HTTP status=204
[2025/07/12 18:24:30] [debug] [upstream] KA connection #59 to localhost:18428 is now available
[2025/07/12 18:24:30] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK
[2025/07/12 18:24:30] [debug] [out flush] cb_destroy coro_id=0
[2025/07/12 18:24:30] [debug] [task] destroy task=0x79fdda0 (task_id=0)
^C[2025/07/12 18:24:33] [engine] caught signal (SIGINT)
[2025/07/12 18:24:33] [ warn] [engine] service will shutdown in max 5 seconds
[2025/07/12 18:24:33] [ info] [engine] service has stopped (0 pending tasks)
[2025/07/12 18:24:33] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] thread worker #0 stopping...
[2025/07/12 18:24:33] [ info] [output:prometheus_remote_write:prometheus_remote_write.0] thread worker #1 stopping...
==3051==
==3051== HEAP SUMMARY:
==3051==     in use at exit: 0 bytes in 0 blocks
==3051==   total heap usage: 45,287 allocs, 45,287 frees, 5,995,480 bytes allocated
==3051==
==3051== All heap blocks were freed -- no leaks are possible
==3051==
==3051== For lists of detected and suppressed errors, rerun with: -s
==3051== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1905

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@johejo johejo marked this pull request as ready for review July 12, 2025 09:40
@johejo johejo mentioned this pull request Jul 12, 2025
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@johejo