Q&A Dynamic uri in http sink

[SadDrummer]

Question

Hey

Prior to 0.57, I dynamically constructed the uri using env vars. This was handy for managing a lot of separate streams:

sinks:
  rsyslog:
    type: http
    ...
    uri: "${OPENOBSERVE_HOST}/api/default/rsyslog/_json"
  journalctl:
    type: http
    ...
    uri: "${OPENOBSERVE_HOST}/api/default/journalctl/_json"

The variables were stored in a .env file (added to .gitignore), load with EnvironmentFile by systemd

Is there any way to dynamically generate the uri in 0.57+?
The secrets backend doesn't seem to support string concatenation
(e.g., "SECRET[env_file.OPENOBSERVE_HOST]/api/..." and other tries fails to parse).
Templating seems not working for this case also.

Creating a multiple separate full URLs in .env for every single stream or push URI to git is not an option for me:(

Thanks!

Vector Config

No response

Vector Logs

No response

[thomasqueirozb]

In this scenario you can either set VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION/--disable-env-var-interpolation which preserves the pre-0.57 behavior or use envsubst to fill in the config file before running vector. See #26036 (specifically tests/e2e/datadog-metrics/config/compose.yaml and tests/e2e/Dockerfile)

[SadDrummer]

@thomasqueirozb thank you, I already made it.

But is there are really not exist way to dynamic build uri with use secret backend - I think, this is possible a popular way to use http_sink

[thomasqueirozb]

I'm able to create a dynamic URI with secrets - it may be that you're doing something incorrectly. Here's a small POC

vector.yaml

secret:
  file_backend:
    type: file
    path: /tmp/vector-secret-demo/secrets.json

sources:
  demo_logs:
    type: demo_logs
    format: json
    lines:
      - '{"message": "hello"}'

sinks:
  http:
    type: http
    inputs: [demo_logs]
    uri: SECRET[file_backend.database_url]/foo
    encoding:
      codec: json

secrets.json

{
  "database_url": "http://127.0.0.1:8123"
}

Running:

In one shell spawn nc -l 8123 and in the other spawn vector -c vector.yaml. The first shell should get a POST request to /foo.

If you're not able to replicate this please open a separate issue with the full config 🙂

[SadDrummer]

@thomasqueirozb, thank you for try to help!

I found issue - it is a trouble with vector validate which fail my pipeline
#26256