Skip to content

Conversation

@koxudaxi
Copy link
Collaborator

@koxudaxi koxudaxi commented Jan 6, 2026

TL;DR

Added a spans outbox table to track spans that need to be synced to ClickHouse.

What changed?

  • Created a new database migration (0001_shiny_donald_blake.sql) that adds:
    • A new enum type outbox_status with values: 'pending', 'processing', 'completed', 'failed'
    • A new table spans_outbox to track spans that need processing
    • Added appropriate indexes and foreign key constraints
  • Updated the trace creation logic to write to the outbox when spans are inserted
  • Moved the spansOutbox.test.ts file from schema/ to the parent directory

How to test?

  1. Run database migrations to create the new table
  2. Create a new trace with spans and verify entries are created in the spans_outbox table
  3. Verify that the outbox entries have the correct status ('pending') and reference valid span IDs
  4. Run the test file to ensure the schema is correctly defined

Why make this change?

This implements an outbox pattern for reliable data synchronization between the main database and ClickHouse. When spans are inserted, they're also recorded in the outbox table, allowing a separate process to reliably sync them to ClickHouse even if the initial sync attempt fails. This pattern ensures data consistency across systems and provides retry capabilities through the status tracking and retry count fields.

Copy link
Collaborator Author

koxudaxi commented Jan 6, 2026

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Jan 6, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
v2-docs fdc0d42 Jan 09 2026, 03:52 AM

@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 2b644cf to 0464fa6 Compare January 6, 2026 07:55
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch 2 times, most recently from 6abb2ce to 0b11879 Compare January 6, 2026 08:14
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 0464fa6 to 47752e2 Compare January 6, 2026 08:14
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 0b11879 to dad5c76 Compare January 6, 2026 08:22
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch 2 times, most recently from b5d6d9c to e7c6f63 Compare January 6, 2026 08:22
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from dad5c76 to c126a39 Compare January 6, 2026 08:22
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from e7c6f63 to d3f0ae9 Compare January 6, 2026 08:32
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from c126a39 to 1d8de59 Compare January 6, 2026 08:32
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from d3f0ae9 to 6b0bc14 Compare January 6, 2026 08:36
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 1d8de59 to d9ae300 Compare January 6, 2026 08:36
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 6b0bc14 to aa5650a Compare January 6, 2026 08:41
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch 2 times, most recently from 4c878c7 to e68492f Compare January 6, 2026 08:44
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from aa5650a to d77eaee Compare January 6, 2026 08:44
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from e68492f to 754e297 Compare January 6, 2026 08:45
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from d77eaee to 3ae9d6d Compare January 6, 2026 08:45
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 9825e75 to b498dab Compare January 8, 2026 12:21
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch 2 times, most recently from b1088cf to 7654821 Compare January 8, 2026 13:25
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from b498dab to 5866958 Compare January 8, 2026 13:25
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 5866958 to 0084b89 Compare January 8, 2026 14:33
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch 2 times, most recently from 068c749 to 8b53b30 Compare January 8, 2026 14:36
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch 2 times, most recently from 2fbd108 to f2d94a8 Compare January 8, 2026 18:03
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 8b53b30 to 4c8e22b Compare January 8, 2026 18:03
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from f2d94a8 to e0e4fcc Compare January 8, 2026 18:09
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 4c8e22b to 85abd21 Compare January 8, 2026 18:09
Copy link
Collaborator

willbakst commented Jan 9, 2026

Merge activity

@koxudaxi koxudaxi force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from e0e4fcc to f9cdc37 Compare January 9, 2026 03:11
@koxudaxi koxudaxi force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from 85abd21 to e33e679 Compare January 9, 2026 03:11
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from e33e679 to fb85ee2 Compare January 9, 2026 03:17
@willbakst willbakst force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from f9cdc37 to 468bbb7 Compare January 9, 2026 03:17
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch from fb85ee2 to bd83d90 Compare January 9, 2026 03:25
@willbakst willbakst force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 468bbb7 to f6c812d Compare January 9, 2026 03:25
@willbakst willbakst force-pushed the 01-06-feat_db_add_spans_outbox_table_schema branch 2 times, most recently from cd0ed96 to a18be51 Compare January 9, 2026 03:32
@willbakst willbakst force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from f6c812d to 41d2ebc Compare January 9, 2026 03:32
@willbakst willbakst changed the base branch from 01-06-feat_db_add_spans_outbox_table_schema to graphite-base/1810 January 9, 2026 03:37
@willbakst willbakst force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 41d2ebc to 345841f Compare January 9, 2026 03:38
@graphite-app graphite-app bot changed the base branch from graphite-base/1810 to v2 January 9, 2026 03:38
@willbakst willbakst force-pushed the 01-06-feat_db_write_to_outbox_on_span_insertion branch from 345841f to fdc0d42 Compare January 9, 2026 03:38
@willbakst willbakst merged commit e29e725 into v2 Jan 9, 2026
10 checks passed
@willbakst willbakst deleted the 01-06-feat_db_write_to_outbox_on_span_insertion branch January 9, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants