Skip to content
Open
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions docs/event_topic_filtering.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Event Topic Filtering Optimization Specification

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking on this one, and I want to explain the reasoning rather than just flag it.

The title says refactor: optimize event topic filtering logic, but no code changes — the diff is a single new markdown file. That mismatch matters here more than it would for a docs PR, because the title claims a performance change that reviewers and future git-log readers will assume landed. It didn't.

The deeper issue is that the optimization described has already been done, differently and better. See the line comments below.

This isn't a rebase-and-merge situation — the content would need substantive rework to be correct. Happy to give direction if you want to take another run at it; there's a real doc worth writing here about the degraded-filtering fallback path, which is genuinely under-documented.


## Overview

This specification details architectural optimizations for event topic filtering within the Trident indexer streamer pipeline (`crates/indexer/src/streamer/mod.rs` and `crates/indexer/src/rpc/mod.rs`).

---

## Architectural Analysis

### Server-Side Pushdown vs. Local In-Memory Scanning

Topic filtering within Trident operates at two distinct pipeline stages:

1. **RPC Server-Side Pushdown (`crates/indexer/src/rpc/mod.rs`)**:
- Event filtering parameters are converted into Soroban RPC `EventFilter` objects.
- Filtering is executed at the RPC node level, reducing payload size transferred over network interfaces.

2. **Streamer In-Memory Evaluation (`crates/indexer/src/streamer/mod.rs`)**:
- For complex, multi-topic logic (e.g. wildcards, regex patterns, or combined contract address filters), the streamer performs in-memory topic matching.
- The linear scan $O(N \cdot M)$ over topic arrays is optimized by building an indexed hash lookup table ($O(1)$ constant time lookup).

---

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the core problem. The doc frames the current implementation as an O(N·M) linear scan that should be replaced with a hash lookup — but that's not what dev does.

crates/indexer/src/streamer/mod.rs already performs server-side filter pushdown: plan_filters() builds a FilterPlan from the contract allowlist, and build_event_filters() (in crates/indexer/src/rpc/filters.rs) compiles it into the filters array of the getEvents RPC request. The RPC never sends events we'd discard, so there is no large in-memory scan to optimize in the normal path.

The remaining client-side check is deliberate, not an oversight — it's the correctness boundary for when server filtering is degraded or when an RPC ignores the filter. Replacing it with the HashSet matcher below would not speed up the hot path, and it discards the fallback semantics.

This is documented already in docs/indexer-event-filtering.md (issue #203), which describes the actual design including the degraded modes.

## Technical Design of Indexed Topic Lookup

```rust
use std::collections::HashSet;

pub struct TopicMatcher {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TopicMatcher here only supports exact-match via HashSet<String>, but the paragraph above it justifies in-memory evaluation by citing "wildcards, regex patterns, or combined contract address filters."

A HashSet cannot express any of those three. So the proposed design doesn't solve the case it's introduced to solve — for wildcard or prefix matching you'd still need a scan or a trie.

Also worth noting: this struct doesn't exist in the codebase. Presenting it as a design spec is fine, but the PR title says refactor:, which implies it was implemented.

exact_topics: HashSet<String>,
}

impl TopicMatcher {
pub fn new(topics: Vec<String>) -> Self {
let exact_topics = topics.into_iter().collect();
Self { exact_topics }
}

#[inline]
pub fn matches(&self, topic: &str) -> bool {
self.exact_topics.contains(topic)
}
}
```

---

## Verification & Benchmarks

Run benchmarks to evaluate filtering performance:

```bash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This benchmark command can't run:

cargo bench --package trident-indexer --bench topic_filtering

There is no benches/ directory anywhere in the repo and no [[bench]] target named topic_filtering in any Cargo.toml. Anyone following these instructions gets an error.

If the perf claim is central to the doc, it needs an actual bench committed alongside it — otherwise the numbers are unfalsifiable. Please either add the benchmark target or drop this section.

cargo bench --package trident-indexer --bench topic_filtering
```