feat: add native NetFlow monitoring with Lucene search and Sankey

.changeset/native-netflow-monitoring.md

@@ -0,0 +1,39 @@
+---
+'@hyperdx/app': minor
+'@hyperdx/api': minor
+'@hyperdx/common-utils': minor
+---
+
+Add native NetFlow sources and monitoring with configurable column mappings,
+Akvorado schema detection, sampling-aware traffic charts, top talkers, exporter
+and interface breakdowns, and flow details. NetFlow tables also support Search
+and custom charts.
+
+Support Lucene search with field autocomplete on the NetFlow page, combining
+search queries with quick filters across charts and flow records.
+
+Add clickable include/exclude filters to flow IPs, protocols, exporters, and
+interfaces, with removable selections preserved in the URL.
+
+Show table column suggestions when focusing an empty NetFlow search input.
+
+Size time-chart Y axes to their formatted labels so rate units do not wrap or
+get clipped.
+
+Avoid sending API proxy headers when querying ClickHouse directly in local mode.
+
+Add a Sankey visualization with ordered table dimensions, sampling-adjusted path
+weights, average bit rates, configurable path limits, and clickable shared
+filters. Include interface classifications in the local demo data.
+
+Recover from invalid time ranges without crashing, clear source-specific click
+filters when switching sources, and keep chart geometry aligned with automatic
+axis widths.
+
+Preserve NetFlow mappings through MCP source tools, ignore stale source inference,
+share filter keys across visualizations, retain quoted columns, and make explicit
+include/exclude actions idempotent. Format flow times using user preferences.
+
+Support NetFlow saved-search alerts with default aliases, previews, and flow
+samples in notifications. Keep unfinished search fields pending when applying
+click filters, and display the configured record limit.

docs/netflow.md

@@ -0,0 +1,174 @@
+# NetFlow monitoring
+
+The NetFlow section queries a configured ClickHouse table through HyperDX's
+existing connections and source mappings. An Akvorado `flows` table can be used
+directly. HyperDX does not collect UDP NetFlow, IPFIX, or sFlow packets: keep
+Akvorado or another collector responsible for decoding and ingestion.
+
+## Local demo
+
+Use Python 3, the repository's Node/Yarn dependencies, and a local ClickHouse
+server with HTTP access and browser CORS enabled. The demo uses
+`http://127.0.0.1:8123`, user `default`, and an empty password. Override these
+with `CLICKHOUSE_URL`, `CLICKHOUSE_USER`, and `CLICKHOUSE_PASSWORD` as needed.
+Local mode sends these credentials to the browser, so use a local development
+account.
+
+From the repository root:
+
+```sh
+yarn setup
+yarn build:common-utils
+python3 scripts/netflow-seed.py
+python3 scripts/netflow-dev.py
+```
+
+The launcher prints the URL, normally `http://127.0.0.1:3000/netflow`, and uses
+port 3001 if 3000 is occupied. It supplies the connection and native NetFlow
+source through HyperDX's supported default configuration environment variables.
+No MongoDB, API server, account registration, or browser storage edits are
+needed for this local demo. A fresh browser session uses these defaults;
+previously saved local connections and sources take precedence.
+
+Choose the last two hours to see the whole fixture, including its TCP traffic
+burst. Re-run the seeder to move the data to the current time. The seeder
+replaces only its own `netflow_demo.flows` table, checking its ownership comment
+first. It refuses remote endpoints and does not modify other databases or
+tables. Refreshes do not accumulate duplicate records. A failed insert can leave
+this disposable fixture table empty; re-run the seeder to restore it.
+
+For a fixed, reproducible time range:
+
+```sh
+python3 scripts/netflow-seed.py --end 2026-10-09T18:34:38Z
+```
+
+This produces the UTC interval `[2026-10-09 16:34:38, 2026-10-09 18:34:38)`. The
+last stored timestamp is `18:34:37`; use that absolute range when inspecting
+this fixed fixture after its timestamps leave the relative time window.
+
+## Data and sampling
+
+The data is deterministic synthetic traffic generated locally, not a packet
+capture or customer traffic. The schema follows
+[Akvorado's flow schema](https://github.com/akvorado/akvorado/blob/main/common/schema/definition.go),
+using documentation IP ranges and private AS numbers. The fixture includes
+14,400 records across three exporters, TCP, UDP, ICMP, ICMPv6, inbound and
+outbound interfaces, IPv4-mapped IPv6 addresses, and native IPv6 addresses.
+Interface classification columns (`InIfConnectivity`, `OutIfConnectivity`,
+`InIfProvider`, `OutIfProvider`) contain deterministic transit, IX, and PNI
+classifications with synthetic provider names tied to each exporter. As with
+Akvorado's default classifier, internal interfaces leave these fields empty. The
+seeder adds missing classification columns only after confirming ownership of
+the demo table; existing counters and record counts stay unchanged. Interface
+speeds are stored in Mbps, as in Akvorado. `ExporterAddress` uses plain `IPv6`
+to work without ClickHouse's optional low-cardinality IPv6 setting.
+
+`Bytes` and `Packets` hold observed counters. `SamplingRate` holds the expansion
+factor (1, 100, or 1,000). Traffic estimates are `sum(Bytes * SamplingRate)` and
+`sum(Packets * SamplingRate)`, matching
+[Akvorado's aggregation](https://github.com/akvorado/akvorado/blob/main/console/widgets.go).
+Multiply estimated bytes by eight and divide by the requested duration in
+seconds for bits per second. The record count is the number of stored records,
+not an estimate of distinct connections or sampling-expanded flows.
+
+The seeder checks actual ClickHouse query results against generated totals:
+
+| Measurement         |    Expected value |
+| ------------------- | ----------------: |
+| Records             |            14,400 |
+| Exporters           |                 3 |
+| Protocols           |                 4 |
+| Native IPv6 records |             2,572 |
+| Observed bytes      |     7,572,836,640 |
+| Observed packets    |         9,162,160 |
+| Estimated bytes     | 2,776,744,315,920 |
+| Estimated packets   |     3,359,797,180 |
+
+## Source configuration
+
+The NetFlow search bar uses the same Lucene syntax, field autocomplete, query
+history, and syntax reference as log Search. Focus the search bar to see table
+columns; type a prefix to narrow them, then click a suggestion or use the arrow
+keys and Enter/Tab. After a column and colon, matching values are suggested.
+Press Enter or Run to apply the query to all charts and flow records. Queries
+combine with the exporter, protocol, and address filters using AND; Clear
+filters resets both the search and quick filters. The query and selected
+language are saved in the URL. The shared language selector also supports SQL
+WHERE expressions.
+
+Click an IP address, protocol, exporter, or interface in a breakdown chart, flow
+row, or flow details to **Include** or **Exclude** it. These actions apply
+immediately while preserving the search query and time range. Selected values
+appear as removable filters and survive URL sharing and reloads. Multiple
+included values for the same field use OR; different fields and exclusions use
+AND. Clear filters also removes these selections.
+
+Use your table's actual field names, for example with the Akvorado schema:
+
+```text
+Proto:6 AND DstPort:443
+ExporterName:edge* AND Bytes:[1000 TO 100000]
+(DstPort:80 OR DstPort:443) AND NOT Proto:17
+```
+
+In the source editor, select NetFlow, a ClickHouse connection, and the desired
+database/table. Use these Akvorado mappings:
+
+| Source setting               | Column/expression        |
+| ---------------------------- | ------------------------ |
+| Timestamp                    | `TimeReceived`           |
+| Bytes / packets              | `Bytes` / `Packets`      |
+| Sampling rate                | `SamplingRate`           |
+| Source / destination address | `SrcAddr` / `DstAddr`    |
+| Source / destination port    | `SrcPort` / `DstPort`    |
+| Protocol                     | `Proto`                  |
+| Exporter                     | `ExporterName`           |
+| Input / output interface     | `InIfName` / `OutIfName` |
+
+For already sampling-adjusted counters, omit the sampling-rate mapping to avoid
+expanding the counters twice. Full deployments persist the source through the
+existing authenticated source API and MongoDB model.
+
+## Sankey visualization
+
+Select **Sankey** in the NetFlow visualization selector to explore traffic paths.
+Choose two to five dimensions in left-to-right order from the source mappings or
+scalar table columns. Akvorado tables default to `SrcAS` → `InIfConnectivity` →
+`InIfProvider` → exporter when those columns are available.
+
+Link widths represent sampling-adjusted bytes for the top 10, 20, or 50 paths.
+The table and tooltips show transferred bytes and average bit rate over the
+selected time range. Paths outside the limit are omitted from the diagram.
+Click a node or table value to include or exclude it using the shared filters;
+Lucene search, quick filters, and the time range also apply to this view.
+The visualization, ordered dimensions, and path limit are saved in the URL.
+
+## Query verification
+
+After building common-utils and seeding, run the real chart configurations
+against local ClickHouse:
+
+```sh
+node node_modules/tsx/dist/cli.mjs scripts/netflow-query-check.ts
+```
+
+This checks overview values, time series, breakdowns, the flow table, IPv4/IPv6
+filters, protocol filtering, and safe handling of SQL syntax in filter values.
+The seeder also verifies counts and sampled totals on every run.
+
+With the local app running and freshly seeded data, verify the browser workflow:
+
+```sh
+yarn playwright install chromium
+node scripts/netflow-browser-check.mjs
+node scripts/netflow-click-browser-check.mjs
+node node_modules/tsx/dist/cli.mjs scripts/netflow-click-query-check.ts
+node node_modules/tsx/dist/cli.mjs scripts/netflow-sankey-query-check.ts
+node scripts/netflow-sankey-browser-check.mjs
+```
+
+This checks charts, filters, sampled flow details, refresh, source creation and
+persistence, and the shared Search page. Screenshots are saved under
+`packages/app/test-results/netflow/`. Set `NETFLOW_APP_URL` if the app uses a
+different local port.

knip.json

@@ -2,7 +2,7 @@
   "$schema": "https://unpkg.com/knip@6/schema.json",
   "workspaces": {
     ".": {
-      "entry": ["scripts/__tests__/*.test.mjs"]
+      "entry": ["scripts/__tests__/*.test.mjs", "scripts/netflow-*-check.{mjs,ts}"]
     },
     "packages/app": {
       "entry": ["pages/**/*.{ts,tsx}", "scripts/*.js", "tests/e2e/**/*.ts"],

package.json

@@ -8,8 +8,10 @@
   ],
   "devDependencies": {
     "@changesets/cli": "^2.26.2",
+    "@clickhouse/client": "1.23.0-head.fae5998.1",
     "@dotenvx/dotenvx": "^1.51.1",
     "@eslint/js": "^9.39.1",
+    "@playwright/test": "^1.57.0",
     "@types/ungap__structured-clone": "^1.2.0",
     "@ungap/structured-clone": "^1.3.0",
     "babel-plugin-react-compiler": "^1.0.0",

packages/api/openapi.json

@@ -5163,6 +5163,137 @@
           }
         }
       },
+      "NetflowSource": {
+        "type": "object",
+        "description": "Network flow table with configurable SQL expressions. Traffic totals multiply bytes and packets by the optional sampling multiplier (defaults to one).",
+        "required": [
+          "name",
+          "kind",
+          "connection",
+          "from",
+          "timestampValueExpression",
+          "defaultTableSelectExpression",
+          "bytesExpression",
+          "packetsExpression",
+          "srcAddrExpression",
+          "dstAddrExpression",
+          "srcPortExpression",
+          "dstPortExpression",
+          "protocolExpression"
+        ],
+        "properties": {
+          "id": {
+            "type": "string",
+            "readOnly": true,
+            "description": "Unique source ID generated by the server.",
+            "example": "507f1f77bcf86cd799439011"
+          },
+          "name": {
+            "type": "string",
+            "description": "Display name for the source.",
+            "example": "Network flows"
+          },
+          "section": {
+            "type": "string",
+            "maxLength": 256,
+            "description": "Optional source grouping label.",
+            "example": "Network"
+          },
+          "disabled": {
+            "type": "boolean",
+            "description": "Hide this source from source selectors."
+          },
+          "kind": {
+            "type": "string",
+            "enum": [
+              "netflow"
+            ],
+            "description": "Source kind discriminator for network flow tables.",
+            "example": "netflow"
+          },
+          "connection": {
+            "type": "string",
+            "description": "ClickHouse connection ID.",
+            "example": "507f1f77bcf86cd799439012"
+          },
+          "from": {
+            "$ref": "#/components/schemas/SourceFrom",
+            "description": "Database and table containing the flow records."
+          },
+          "querySettings": {
+            "type": "array",
+            "description": "ClickHouse query settings applied to this source.",
+            "items": {
+              "$ref": "#/components/schemas/QuerySetting"
+            }
+          },
+          "timestampValueExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "TimeReceived"
+          },
+          "defaultTableSelectExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "TimeReceived, SrcAddr, DstAddr, Bytes"
+          },
+          "bytesExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "Bytes"
+          },
+          "packetsExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "Packets"
+          },
+          "srcAddrExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "SrcAddr"
+          },
+          "dstAddrExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "DstAddr"
+          },
+          "srcPortExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "SrcPort"
+          },
+          "dstPortExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "DstPort"
+          },
+          "protocolExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "Proto"
+          },
+          "samplingRateExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "SamplingRate"
+          },
+          "exporterExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "ExporterName"
+          },
+          "inIfExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "InIfName"
+          },
+          "outIfExpression": {
+            "type": "string",
+            "description": "SQL expression for the mapped flow field.",
+            "example": "OutIfName"
+          }
+        }
+      },
       "Source": {
         "oneOf": [
           {
@@ -5179,6 +5310,9 @@
           },
           {
             "$ref": "#/components/schemas/PromqlSource"
+          },
+          {
+            "$ref": "#/components/schemas/NetflowSource"
           }
         ],
         "discriminator": {
@@ -5188,7 +5322,8 @@
             "trace": "#/components/schemas/TraceSource",
             "metric": "#/components/schemas/MetricSource",
             "session": "#/components/schemas/SessionSource",
-            "promql": "#/components/schemas/PromqlSource"
+            "promql": "#/components/schemas/PromqlSource",
+            "netflow": "#/components/schemas/NetflowSource"
           }
         }
       },