[bikash-c]

Summary:

Why

GetApproximateSizes estimates blob-file size using the range's share of SST bytes. This can give similar blob estimates to ranges with very different blob value sizes. Per-SST blob reference counts provide better attribution, at the cost of visiting every SST and potentially reading table properties.

Behavior

With include_blob_files, the default estimate distributes the column family's physical blob-file size in proportion to SST bytes. Setting SizeApproximationOptions::allow_expensive_query permits more costly estimation in exchange for accuracy. Currently it enables per-SST blob attribution: every SST is visited, and table properties may be read synchronously when its blob reference count is not already in memory, regardless of the size of the requested ranges.

SSTs that record blob reference counts share the blob bytes according to those counts and the estimated fraction of each SST in the range. SSTs without a usable count share the remaining live blob bytes according to their SST sizes. Their allocation is at least their share by SST size, so references to superseded values in other SSTs cannot crowd them out. If every SST has a count, the full live-byte budget is distributed over the remaining references, including blob files retained after DeleteFilesInRange removes an SST. Garbage and file overhead are included by scaling the allocations to the physical blob footprint. If no usable weights remain, the estimate falls back to the blob-to-SST size ratio.

Implementation

Block-based SSTs record optional referenced_blob_bytes: compressed payload bytes plus each separate blob record's key and header overhead. Inlined blobs and blobs embedded in the SST are excluded.

Queries reuse immutable FileMetaData counts and capture each other SST's property result once per call. A shared classification helper and an optional BlobSizeAccounting context bind the lookup to the paired range outputs. Counts are populated only before concurrent publication; secondary input-statistics initialization does not write them. A failed read or absent count uses the same fallback classification across all requested ranges.

Compatibility and limitations

include_blob_files and allow_expensive_query both default to false. Flag-based convenience calls use the default estimation method; C scalar bindings expose the cost opt-in. No new MANIFEST field or eager-open I/O is introduced.

Older SSTs and table formats without the property use the fallback attribution. Older SST readers can ignore the additive property or expose it as a user-collected property. Prepared ingestion metadata omits the statistic because existing consumers reject unknown serialized fields. The prepared reader ignores unknown fields while preserving checksum, required-field and known-value validation. Generic TableProperties and C accessors distinguish missing from recorded zero; aggregation sums recorded inputs and remains unknown if all inputs are unknown.

Attribution remains approximate within an SST and can count superseded key versions at multiple levels. Garbage is not charged specifically to the range that created it. Memtable blob references and blobs embedded in SSTs are excluded. The expensive mode is not constant-time or I/O-free, even for a small requested range.

Internal CI resource safety

The Sandcastle crash-test driver also bounds Make compilation parallelism. Host CPU counts can otherwise launch hundreds of debug C++ compilers inside a memory-limited worker before the stress test starts. Compilation now uses at most 16 jobs, reduced further by finite cgroup v1/v2 memory limits with 4 GiB reserved for the CI runtime and 4 GiB per compiler slot. Missing or unreadable cgroup limits retain the conservative ceiling. Lower configured build parallelism remains honored; crash-test concurrency, duration, randomized options and assertions are unchanged. This is a CI-driver change only, not a runtime BlobDB memory-policy change.

Reviewed By: pdillinger

Differential Revision: D123890542

[meta-codesync]

@bikash-c has exported this pull request. If you are a Meta employee, you can view the originating Diff in D123890542.

[github-actions]

✅ clang-tidy: No findings on changed lines

Completed in 1689.9s.