Improve snapshot compression + metrics #3296
Merged
+426
−147
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `DurabilityProvider` trait was introduced to enable the `HostController` to procure an alternative `Durability` impl from an external source. It is also useful to be able to instantiate a `SnapshotWorker` externally, in order to subscribe to snapshot creation events without access to the `RelationalDB` instance it is operating on. At a later stage, we may also use it to control the snapshot frequency externally. This patch thus reframes the trait as `PersistenceProvider`, whose job is to provide persistence-related services. Also separates snapshot creation and compression of older snapshots, and adds instrumentation to gather timing information for both.
|
Stacked on top of #3295 |
kim
force-pushed
the
kim/snapshot-compression-stats
branch
3 times, most recently
from
84b088f to
c838ed5
Compare
Report more metrics about snapshot compression, namely: - time to compress a single snapshot (histogram) - for each compression pass: - number of snapshots found to be already compressed (gauge) - number of snapshots compressed (gauge) - cumulative number of objects compressed (gauge) - cumulative number of objects hardlinked (gauge) Those metrics are collected from the `spacetimedb-snapshot` crate without imposing a prometheus dependency on it, i.e. they can be observed by the caller as ordinary Rust types. This is exploited to avoid scanning the entire snapshot repository on each pass -- only the range `(last_compressed + 1)..newest_snapshot` is visited (note that the `compress_snapshots` method now short-circuits on errors). Lastly, the snapshot worker can now be configured to disable compression. This greatly simplifies implementation of alternative post-processing strategies, e.g. involving archival, for which a more coarse-grained compression strategy may be more appropriate. Subcribers are notified of a new snapshot _after_ compression, such that any filesystem locks should be released.
kim
force-pushed
the
kim/snapshot-compression-stats
branch
from
c838ed5 to
89779b6
Compare
gefjon
approved these changes
Sep 26, 2025
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Report more metrics about snapshot compression, namely:
Those metrics are collected from the
spacetimedb-snapshotcratewithout imposing a prometheus dependency on it, i.e. they can be
observed by the caller as ordinary Rust types.
This is exploited to avoid scanning the entire snapshot repository on
each pass -- only the range
(last_compressed + 1)..newest_snapshotisvisited (note that the
compress_snapshotsmethod now short-circuits onerrors).
Lastly, the snapshot worker can now be configured to disable
compression. This greatly simplifies implementation of alternative
post-processing strategies, e.g. involving archival, for which a more
coarse-grained compression strategy may be more appropriate.
Subcribers are notified of a new snapshot after compression, such that
any filesystem locks should be released.
Expected complexity level and risk
2
Testing
May need some, I'm pondering.