feat: Implement Zero-Gap Hot-to-Cold Time-Series Migration - #1107
Merged
Mosas2000 merged 1 commit intoAug 24, 2026
Merged
Conversation
…dge#1021) - Add migration_manifests table (051 migration) with status lifecycle, SHA-256 checksum, row count, schema version, and unique constraint on (entity_type, range_start, range_end) to prevent duplicate segments - Add HotColdMigrationService with: · migrateRange() — idempotent INSERT…SELECT…ON CONFLICT DO NOTHING into archive tables with checksum stamping and status tracking · dualRead() — UNION of hot + archive rows tagged by source so readers see a complete, gap-free range during any migration phase · atomicCutover() — deletes hot rows only after archive rows are confirmed present; aborts if archive is empty · rollback() — deletes archive rows and resets manifest to rolled_back · resume() — resets a failed manifest to pending and re-runs migrateRange (idempotent due to ON CONFLICT DO NOTHING) · restoreDrill() — compares hot vs archive checksums and verifies dual-read covers the full row set (gap-free check) · verifyContinuousAggregates() — non-fatal check that hourly/daily aggregates have materialized data across the migration boundary · invalidateCaches() — Redis SCAN+DEL per entity cache key patterns - Add comprehensive unit tests covering all acceptance criteria: checksum determinism, dual-read zero-gap guarantee, deduplication, rollback guard, resume idempotency, restore drill, cache invalidation, atomicCutover empty-archive guard, manifest field mapping Closes StellaBridge#1021
Contributor
|
This is perfect as the append-only design and deterministic checksums perfectly enforce non-repudiation and provide a cryptographically verifiable trail for digital forensics. Thank you @stableprogrammer |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1021
Summary
Implements a complete zero-gap hot-to-cold time-series migration protocol for the three live hypertables (
prices,health_scores,liquidity_snapshots) andpool_events.Migration manifest (
051_hot_cold_migration_manifest.ts):migration_manifeststable tracking each archival segment: status lifecycle (pending → migrating → verifying → complete / failed → rolled_back), SHA-256 checksum, row count, schema version, error message, timestamps(entity_type, range_start, range_end)prevents duplicate segments on resumeHotColdMigrationService(hotColdMigration.service.ts):migrateRange()— idempotentINSERT … SELECT … ON CONFLICT DO NOTHINGinto archive tables; stamps checksum and row count; transitions manifest through all lifecycle statesdualRead()— UNIONs hot + archive rows (taggedsource: "hot"|"archive") ensuring readers see a complete, gap-free range during any phase of migration; deduplicates rows that exist in both tables (hot copy wins)atomicCutover()— verifies archive rows exist before deleting hot rows; aborts with an error if archive is empty to prevent data lossrollback()— deletes archive rows and resets manifest torolled_back; throws if called on acompletemanifestresume()— resets afailedmanifest topendingand re-runsmigrateRange; idempotent due toON CONFLICT DO NOTHINGrestoreDrill()— compares SHA-256 checksums between hot and archive for the range and verifies dual-read covers the full row set (gap-free assertion)verifyContinuousAggregates()— non-fatal check that hourly/daily aggregate views have materialized data across the cutover boundary; logs warnings but never blocks migrationinvalidateCaches()— uses RedisSCAN(notKEYS) to safely scan and delete cache entries per entity-specific key patternsUnit tests (
hotColdMigration.service.test.ts):