Repository navigation
Compactions discard values in dropped blob files (#15645) - #15645
Closed
mikechuangmeta wants to merge 1 commit into
Closed
mikechuangmeta wants to merge 1 commit into
mikechuangmeta wants to merge 1 commit into
Conversation
|
@mikechuangmeta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131013279. |
mikechuangmeta
added a commit
to mikechuangmeta/rocksdb
that referenced
this pull request
Nov 11, 2026
Summary: Pull Request resolved: meta#15645 Changes - Every compaction, automatic or manual, whose inputs may reference a blob file recorded by `best_efforts_blob_db_repair` runs `MissingBlobRepairIterator` in a new mode limited to recorded blob files. Lost values become deletions at their sequence number; wide entities keep surviving columns. - These compactions run locally and skip garbage for recorded blob files. Unrecorded missing blob files still fail with Corruption. Why? - After D131013277 records a dropped blob file, a compaction that drops one of its values still fails: garbage for a blob file absent from the Version is rejected ("unknown blob file"), and blob GC, filters and merges read the missing payload. Reusing the `CompactRangeOptions` repair from D128836898 lets compactions proceed and removes the dangling references over time. Changes visible to existing users - None while no dropped blob files are recorded: each compaction adds an O(1) check of its input Version's empty dropped set. Design doc - https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e Reviewed By: xingbowang Differential Revision: D131013279
mikechuangmeta
force-pushed
the
export-D131013279
branch
from
November 11, 2026 00:43
c27bd59 to
5d62472
Compare
meta-codesync
Bot
changed the title
✅ clang-tidy: No findings on changed linesCompleted in 600.1s. |
mikechuangmeta
added a commit
to mikechuangmeta/rocksdb
that referenced
this pull request
Nov 11, 2026
Summary: Pull Request resolved: meta#15645 Changes - Every compaction, automatic or manual, whose inputs may reference a blob file recorded by `best_efforts_blob_db_repair` runs `MissingBlobRepairIterator` in a new mode limited to recorded blob files. Lost values become deletions at their sequence number; wide entities keep surviving columns. - These compactions run locally and skip garbage for recorded blob files. Unrecorded missing blob files still fail with Corruption. Why? - After D131013277 records a dropped blob file, a compaction that drops one of its values still fails: garbage for a blob file absent from the Version is rejected ("unknown blob file"), and blob GC, filters and merges read the missing payload. Reusing the `CompactRangeOptions` repair from D128836898 lets compactions proceed and removes the dangling references over time. Changes visible to existing users - None while no dropped blob files are recorded: each compaction adds an O(1) check of its input Version's empty dropped set. Design doc - https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e Reviewed By: xingbowang Differential Revision: D131013279
mikechuangmeta
force-pushed
the
export-D131013279
branch
from
November 11, 2026 00:55
5d62472 to
72b5fcd
Compare
mikechuangmeta
added a commit
to mikechuangmeta/rocksdb
that referenced
this pull request
Nov 11, 2026
Summary:
Changes
- Every compaction, automatic or manual, whose inputs may reference a blob
file recorded by `best_efforts_blob_db_repair` runs
`MissingBlobRepairIterator` in a new mode limited to recorded blob files.
Lost values become deletions at their sequence number; wide entities keep
surviving columns.
- These compactions run locally and skip garbage for recorded blob files.
Unrecorded missing blob files still fail with Corruption.
Why?
- After D131013277 records a dropped blob file, a compaction that drops one of
its values still fails: garbage for a blob file absent from the Version is
rejected ("unknown blob file"), and blob GC, filters and merges read the
missing payload. Reusing the `CompactRangeOptions` repair from D128836898
lets compactions proceed and removes the dangling references over time.
Changes visible to existing users
- None while no dropped blob files are recorded: each compaction adds an O(1)
check of its input Version's empty dropped set.
Design doc
- https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e
Reviewed By: xingbowang
Differential Revision: D131013279
mikechuangmeta
force-pushed
the
export-D131013279
branch
from
November 11, 2026 04:37
72b5fcd to
68f24d4
Compare
mikechuangmeta
added a commit
to mikechuangmeta/rocksdb
that referenced
this pull request
Nov 11, 2026
Summary:
Changes
- Every compaction, automatic or manual, whose inputs may reference a blob
file recorded by `best_efforts_blob_db_repair` runs
`MissingBlobRepairIterator` in a new mode limited to recorded blob files.
Lost values become deletions at their sequence number; wide entities keep
surviving columns.
- These compactions run locally and skip garbage for recorded blob files.
Unrecorded missing blob files still fail with Corruption.
Why?
- After D131013277 records a dropped blob file, a compaction that drops one of
its values still fails: garbage for a blob file absent from the Version is
rejected ("unknown blob file"), and blob GC, filters and merges read the
missing payload. Reusing the `CompactRangeOptions` repair from D128836898
lets compactions proceed and removes the dangling references over time.
Changes visible to existing users
- None while no dropped blob files are recorded: each compaction adds an O(1)
check of its input Version's empty dropped set.
Design doc
- https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e
Reviewed By: xingbowang
Differential Revision: D131013279
mikechuangmeta
force-pushed
the
export-D131013279
branch
from
November 11, 2026 04:43
68f24d4 to
78f08f5
Compare
Summary:
Changes
- Every compaction, automatic or manual, whose inputs may reference a blob
file recorded by `best_efforts_blob_db_repair` runs
`MissingBlobRepairIterator` in a new mode limited to recorded blob files.
Lost values become deletions at their sequence number; wide entities keep
surviving columns.
- These compactions run locally and skip garbage for recorded blob files.
Unrecorded missing blob files still fail with Corruption.
Why?
- After D131013277 records a dropped blob file, a compaction that drops one of
its values still fails: garbage for a blob file absent from the Version is
rejected ("unknown blob file"), and blob GC, filters and merges read the
missing payload. Reusing the `CompactRangeOptions` repair from D128836898
lets compactions proceed and removes the dangling references over time.
Changes visible to existing users
- None while no dropped blob files are recorded: each compaction adds an O(1)
check of its input Version's empty dropped set.
Design doc
- https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e
Reviewed By: xingbowang
Differential Revision: D131013279
mikechuangmeta
force-pushed
the
export-D131013279
branch
from
November 11, 2026 04:59
78f08f5 to
7bd3c67
Compare
|
This pull request has been merged in 4ff3dd2. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Changes
file recorded by
best_efforts_blob_db_repairrunsMissingBlobRepairIteratorin a new mode limited to recorded blob files.Lost values become deletions at their sequence number; wide entities keep
surviving columns.
Unrecorded missing blob files still fail with Corruption.
Why?
its values still fails: garbage for a blob file absent from the Version is
rejected ("unknown blob file"), and blob GC, filters and merges read the
missing payload. Reusing the
CompactRangeOptionsrepair from D128836898lets compactions proceed and removes the dangling references over time.
Changes visible to existing users
check of its input Version's empty dropped set.
Design doc
Reviewed By: xingbowang
Differential Revision: D131013279