Skip to content

Conversation

@koxudaxi
Copy link
Collaborator

@koxudaxi koxudaxi commented Jan 6, 2026

TL;DR

Added a spans outbox table to implement the Transactional Outbox pattern for reliable span synchronization to ClickHouse.

What changed?

  • Created a new spansOutbox schema with a corresponding table definition that tracks spans to be synced to ClickHouse
  • Added an outboxStatusEnum to track the sync state of spans (pending, processing, completed, failed)
  • Implemented proper indexing and constraints for efficient querying and idempotency
  • Added tests for the new schema to verify table name and column structure
  • Fixed ClickHouse client type handling to properly handle JSON responses
  • Updated schema exports to include the new spansOutbox schema

How to test?

  1. Run the schema tests: vitest cloud/db/schema/spansOutbox.test.ts
  2. Verify the table is created during migrations
  3. Test the ClickHouse client with the updated type handling

Why make this change?

This implements the Transactional Outbox pattern to ensure reliable, eventually consistent data synchronization between PostgreSQL and ClickHouse. The pattern guarantees that spans created in PostgreSQL will eventually be synced to ClickHouse, even in the presence of failures or network issues. This approach provides:

  • Atomic transactions (span + outbox entry created together)
  • Retry capability with error tracking
  • Idempotent operations through unique constraints
  • Efficient polling through specialized indexes
  • Visibility into sync status for monitoring and debugging

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Jan 6, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
v2-docs a18be51 Jan 09 2026, 03:47 AM

Copy link
Collaborator Author

koxudaxi commented Jan 6, 2026

@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 2b644cf to 0464fa6 Compare January 6, 2026 07:55
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from 379c7c3 to 94cc04c Compare January 6, 2026 08:22
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch 3 times, most recently from e7c6f63 to d3f0ae9 Compare January 6, 2026 08:32
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch 2 times, most recently from 72230a9 to db31c84 Compare January 6, 2026 08:36
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from d3f0ae9 to 6b0bc14 Compare January 6, 2026 08:36
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from db31c84 to f02b009 Compare January 6, 2026 08:41
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 6b0bc14 to aa5650a Compare January 6, 2026 08:41
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from f02b009 to f3a92e1 Compare January 6, 2026 08:44
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from aa5650a to d77eaee Compare January 6, 2026 08:44
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from f3a92e1 to 7a8dde2 Compare January 6, 2026 08:45
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from d77eaee to 3ae9d6d Compare January 6, 2026 08:45
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from e717c79 to 795a24b Compare January 8, 2026 12:21
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from b1088cf to 7654821 Compare January 8, 2026 13:25
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from 795a24b to 0a64910 Compare January 8, 2026 13:25
@koxudaxi koxudaxi changed the base branch from 01-06-feat_clickhouse_implement_clickhouse_client_service to graphite-base/1809 January 8, 2026 14:20
@koxudaxi koxudaxi force-pushed the graphite-base/1809 branch from 0a64910 to dca4196 Compare January 8, 2026 14:33
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 7654821 to 068c749 Compare January 8, 2026 14:33
@koxudaxi koxudaxi changed the base branch from graphite-base/1809 to 01-06-feat_clickhouse_implement_clickhouse_client_service January 8, 2026 14:34
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from dca4196 to ab1f5cd Compare January 8, 2026 14:36
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch 2 times, most recently from 8b53b30 to 4c8e22b Compare January 8, 2026 18:03
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch 2 times, most recently from 1bfbeaa to a4f0b7e Compare January 8, 2026 18:09
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 4c8e22b to 85abd21 Compare January 8, 2026 18:09
Copy link
Collaborator

willbakst commented Jan 9, 2026

Merge activity

@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 85abd21 to e33e679 Compare January 9, 2026 03:11
@koxudaxi koxudaxi force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from a4f0b7e to 2010b84 Compare January 9, 2026 03:11
@willbakst willbakst force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch from 2010b84 to a760968 Compare January 9, 2026 03:16
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from e33e679 to fb85ee2 Compare January 9, 2026 03:17
@willbakst willbakst force-pushed the 01-06-feat_clickhouse_implement_clickhouse_client_service branch 2 times, most recently from b005731 to 54d36f0 Compare January 9, 2026 03:24
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from fb85ee2 to bd83d90 Compare January 9, 2026 03:25
@willbakst willbakst changed the base branch from 01-06-feat_clickhouse_implement_clickhouse_client_service to graphite-base/1809 January 9, 2026 03:30
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from bd83d90 to cd0ed96 Compare January 9, 2026 03:31
@graphite-app graphite-app bot changed the base branch from graphite-base/1809 to v2 January 9, 2026 03:31
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from cd0ed96 to a18be51 Compare January 9, 2026 03:32
@willbakst willbakst merged commit dad0a95 into v2 Jan 9, 2026
10 of 11 checks passed
@willbakst willbakst deleted the 01-06-feat_db_add_spans_outbox_table_schema branch January 9, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants