test(python): add E2E ClickHouse verification script

[koxudaxi]

TL;DR

Added a new end-to-end verification script for ClickHouse sync via the Search API.

What changed?

Added a new Python script local_clickhouse_verify.py that:

1. Sends spans via OTLP exporter
2. Polls the Search API until the spans appear in ClickHouse
3. Verifies data integrity by checking various Search API endpoints
4. Reports success or failure of the E2E verification

The script creates test traces with nested spans, waits for them to be synced to ClickHouse, and then verifies the data is correctly accessible through the Search API.

How to test?

1. Start Docker: cd cloud && docker compose -f docker/compose.yml up -d
2. Start cloud server: bun run cloud:dev
3. Start sync worker: cd cloud && bun run tsx workers/clickhouseSyncLocal.ts
4. Run the script: MIRASCOPE_API_KEY=mk_xxx uv run python examples/misc/local_clickhouse_verify.py

The script will output detailed progress and verification results.

Why make this change?

This script provides a reliable way to verify the complete data flow from OTLP export to ClickHouse storage and Search API retrieval. It helps ensure that the entire pipeline is functioning correctly during local development and testing.

[koxudaxi]

• refactor(clickhouse): extract formatDateTime64 to shared utils #1817
• test(python): add E2E ClickHouse verification script #1818 👈 (View in Graphite)
• feat(api): add search API schema and handlers #1816
• chore(test): hard delete api test users #1850
• feat(clickhouse): implement ClickHouseSearchService #1815
• feat(workers): implement Cron Trigger for outbox re-enqueue #1813 : 1 other dependent PR (#1814 )
• feat(workers): implement outbox processor for ClickHouse sync #1811 : 1 other dependent PR (#1812 )
• feat(db): write to outbox on span insertion #1810
• feat(db): add spans_outbox table schema #1809
• feat(clickhouse): implement ClickHouse client service #1808
• feat(docker): add ClickHouse service for local development #1807 : 1 other dependent PR (#1819 )
• feat(clickhouse): add clickhouse migrations and migrate tooling #1849
• v2

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	v2-docs	c025880	Jan 09 2026, 04:25 AM

[willbakst]

Merge activity

• Jan 9, 1:40 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jan 9, 1:41 AM UTC: Graphite couldn't merge this pull request because a downstack PR feat(clickhouse): add clickhouse migrations and migrate tooling #1849 failed to merge.
• Jan 9, 4:25 AM UTC: @willbakst merged this pull request with Graphite.