Handing a large amount of sampling / filtering rules based on string matches

[PrayagS]

Coming from promtail/alloy world, I have the following use case,

Given logs from an incoming source containing logs from multiple services, I want to do two things,

1. Match a pattern with the incoming stream and sample all logs that match the pattern by a given %.
2. Match a pattern with the incoming stream and filter those logs (they may be sampled later).

What's the best way to do this in vector? For use case 1, I had to do multiple things,

• One filter transform that filters out logs that match my pattern. This very well could be a remap transform if simple boolean expressions don't fit the use case. And then a sample transform for this new input.
• One remap transform that removes logs of this pattern from the primary stream to avoid duplication.
• The output of sample transform and the remap transform both go into the primary sink.

Is this the best approach? Seems very inconvenient when I have >100 such sampling rules. In promtail/alloy, I can do match and then run a set of pipeline stages on the filtered stream. The main stream and this processed stream is then merged later. Sharing a sample config below,

loki.process "foo" {
  forward_to = <RECEIVER_LIST>

  stage.foo {
    # something that applies to all the logs
  }
  
  # sample logs from `bar` service selectively
  stage.match {
    selector = "{service=\"bar\"}"

    stage.sampling {
        rate = 0.25
    }
  }
}

I'm looking at all these transforms and seeing that they only support one liner boolean expressions. Are there any plans to add support for more verbose VRL functions that can return true/false or other values according to the transform?

• One filter transform that filters out logs that match my pattern. This very well could be a remap transform if simple boolean expressions don't fit the use case. And then a sample transform for this new input.
• One remap transform that removes logs of this pattern from the primary stream to avoid duplication.

After a second look at the docs, it seems like it makes more sense to use remap transforms to filter logs since they support the reroute_dropped property. I won't need to handle duplicate logs.

[PrayagS]

I think a bigger distinction that I see here is that promtail/alloy handles processing in a tree like manner i.e. branch out to process selectively and then merge back into a single stream at the end.

But in vector, it seems like implementing this kind of branching is very verbose and leads to a ton of output streams that need to be handled carefully to maintain order and de-duplication.

[pront]

Vector pipelines are DAGs, so your pipeline can be 'tree-like' and then you can hook all the leaf outputs to a single component (transform or sink).

Is this the best approach? Seems very inconvenient when I have >100 such sampling rules

You could have complex predicates in a Remap transform and you can also decide to drop (not forward) an event based on the result of those predicates. However, I am not exactly sure how that would look like in your case.

[pront]

For the service = "bar" example, use sample.exclude: events matching exclude pass through unchanged. Conditions can also contain multiline VRL, as long as the final expression returns a boolean.

This pipeline drops messages containing /health, keeps one in four remaining bar logs, and keeps all other logs:

sources:
  logs:
    type: stdin
    decoding:
      codec: json

transforms:
  drop_health_checks:
    type: filter
    inputs: [logs]
    condition: |
      message = string(.message) ?? ""
      !contains(message, "/health")

  sample_bar:
    type: sample
    inputs: [drop_health_checks]
    rate: 4
    exclude: |
      service = string(.service) ?? ""
      service != "bar"

sinks:
  output:
    type: console
    inputs: [sample_bar]
    encoding:
      codec: json

There is one output stream, so no branching or deduplication is needed. Tested locally with Vector 0.59.0: 10 of 40 bar logs and all 40 other logs survived, all 10 health-check logs were removed, and no event IDs were duplicated.

For many rules with different sampling rates, current Vector also supports rate_field and ratio_field. A preceding remap can assign the desired rate or ratio per event, followed by a single sample transform.

Please don't hesitate to followup in this thread if you have further questions.