feat: serialize BasicBlocks in padded representation (1/3)

[huitseeker]

As a prelude to #2448, this changes the serialization of BasicBlock to reflect the padded contents, to not need to re-batch and pad those blocks again.

The goal of this PR is twofold:

• experiment with and analyze the padded on-disk representation, unlocked by this comment 🎉 — get conversation started about how permissible it is to serialize metadata:
  • One important sub-part of this: in this PR, we're over-consuming bits in the indptr representation, not only because each index is represented with one byte (the value is in [0, 72] which fits in 7 bits) but because the groups contain at most 8 ops, so a simple delta-encoding would give us $8 \times 4$ bits $= 32$ bits with an implicit start at 0, instead of $9 \times 8 = 72$. This is not yet implemented, but would cut the ovehead numbers below about in half.
• Groundwork for Simplify MastForest serialization by directly serializing DebugInfo #2448, which needs to change the over-the-wire representation of decorator info anyway, but has a long paragraph in there on the difficulty of un-padding decorators ("The Padding Wrinkle") that disappears in a puff of simplification after this work.

As a consequence of bumping the version number for serialization of MastForest, this PR is intended to stay open until we land on the right over-the-wire format. I.e. there's a version bump in there which I don't intend to have in the PRs which will live on top of this. This stack will be three PRs (one PR delta-encoding, one PR #2448)

Test Data: Miden Standard Library

• 727 total nodes
• 439 basic block nodes (60.4%)
• 284 procedures
• 94,744 operations (with padding)

Summary

The padded format adds 4.05% size overhead:

• 0.67% from NOOP padding (633 ops)
• 3.38% from batch metadata (33,316 bytes)

Most blocks (92.7%) add ≤34 bytes.

Size comparison between unpadded and padded serialization formats for MastForest.

Size Comparison

Format	Size	Overhead
Unpadded (next)	838,400 bytes (818.75 KB)	baseline
Padded (serialize-padded-opbatches)	872,375 bytes (851.93 KB)	+33,975 bytes (+4.05%)

The unpadded format cannot guarantee exact OpBatch reconstruction after deserialization.

Overhead Sources

NOOP Padding: 633 operations (0.67%)

Metric	Count
Total operations	94,744
NOOP padding	633
Real operations	94,111

Batch Metadata: 33,316 bytes (98% of overhead)

Component	Per Unit	Count	Total
Indptr array	9 bytes	3,156 batches	28,404 bytes
Padding flags	1 byte	3,156 batches	3,156 bytes
Batch count	4 bytes	439 blocks	1,756 bytes

Total metadata: 33,316 bytes (32.54 KB)

Distribution summary:

• 92.7% of blocks: ≤34 bytes overhead (1-3 batches)
• 2.3% of blocks: >100 bytes overhead (9+ batches)
• Average: 75.89 bytes per block (skewed by outliers)

Metadata per block: 4 + (num_batches × 10) bytes

Wire Format

Each basic block stores:

┌─────────────────────────────────────────────────┐
│ Padded Operations (variable)                    │
├─────────────────────────────────────────────────┤
│ Batch Count (u32, 4 bytes)                      │
├─────────────────────────────────────────────────┤
│ Indptr Arrays (9 × u8 per batch)               │
├─────────────────────────────────────────────────┤
│ Padding Flags (1 byte per batch, bit-packed)   │
└─────────────────────────────────────────────────┘

Size: ops_size + 4 + (10 × num_batches) bytes

Script: https://gist.github.com/huitseeker/f957014b15b0ed08a36b7f936079b698

[bobbinth]

This is a very shallow review from me - but looks great! Thank you!

[bobbinth]

Question: do we need this "duality" mostly to support the slow processor? That is, is the slow processor the only reason why we need to keep track of the raw (unpadded) indexes?

[huitseeker]

No, we still support raw indexes for those reasons:

• Assembly block merging (raw ops & raw decorator indexes)
• Node creation (raw → padded conversion in initial batching)
• Fingerprinting (padded → raw conversion for stability, which I will remove, after this stack of PRs)

This stack of PRs also removes the use of raw indexes in serialization.

[plafer]

LGTM!

[plafer]

I think the stack of 3 PRs are good to merge - any reason to hold off?