feat(lgbgen): compress the embedded core bundle (default off)

[mparrett]

Stacked on #501 (base branch feat/lgb-compression).

What it does

Adds lgbgen --compress for emitting a compressed embedded-core bundle. The flag is off by default: the committed core remains uncompressed unless the maintainer explicitly chooses the size/startup tradeoff under #503.

The bundle is written atomically through a same-directory temporary file. Encoding, sync, chmod, and close must all succeed before rename, so a failed generation leaves the existing checked-in core_compiled.lgb intact.

Fresh measurements

Measured on an Apple M2 after rebasing onto the reviewed #501 format:

	uncompressed	compressed	delta
core .lgb	297,359 B	89,776 B	-207,583 B (3.31x)
lg binary	13,844,450 B	13,629,794 B	-214,656 B
decode core	2.23 ms	3.40 ms	+1.17 ms, +91,824 B/op, +303 allocs
warm process boot	7.9 ± 0.8 ms	9.8 ± 1.1 ms	about +1.9 ms

Compression primarily benefits larger bundles; for smaller programs, the framing/codec overhead can consume or exceed the byte savings. The committed core is large enough to show a clear size win, but the startup regression makes default-on a policy decision rather than a mechanical one.

Encode cost for the core rose from about 1.35 ms to 24.37 ms and from 344,159 B/op to 2,220,688 B/op. That cost is paid during generation, not process startup.

Validation

• Atomic-write tests cover preserving an existing destination after an encode failure, successful replacement, mode preservation, and temporary-file cleanup.
• Decode and encode benchmarks are checked in under pkg/bytecode.
• go test ./... passes locally.

Policy

Keep compression opt-in for now. The default-on decision remains under #503, including the stripping nuance and any target-specific defaults.

[mparrett]

Tracked under Epic #503 (Footprint & constrained targets).

[nnunley]

One note from a multi-agent pass (which found nothing wrong with this PR itself — the flag plumbing and the generated.sums change both verified correct): the bundle write in cmd/lgbgen is os.Create + direct write, so an encode error exits leaving a half-written core_compiled.lgb on disk for the next make build/make generate to trip over. That's pre-existing rather than introduced here — but since compression adds a new way for the encode to fail mid-stream and nothing else is queued to touch this path, this PR is a reasonable place to either fold in a write-to-temp + rename (a few lines), or spin it out as a fresh issue under Epic #503 so it doesn't get lost.

[nooga]

This PR's own diff looks good — small, correct, and I independently reproduced the numbers in the description exactly (244,131 → 77,039 bytes, ~2x decode cost). Not requesting changes here since there's nothing wrong in these ~65 lines; the actual blocker is upstream in #501 (which this stacks directly on) — its rebase needs to land first, since #501's boot-path code has drifted from what main now has in boot.go/run.go.

One pre-existing item worth picking up while you're in this area, since it's specifically about the write path this PR adds compression to: writeBundle (main.go:591) does os.Create + direct-write-through-Encode, so an encode failure partway leaves a truncated core_compiled.lgb on disk — and that's the file checked into the repo, so a failed go generate would corrupt the committed artifact for the next build. nnunley flagged this back on 07-15 under Epic #503; write-to-temp + rename would close it. Not blocking this PR specifically, just flagging since it's adjacent to what you're touching.

Whether to default this on for the embedded core is the open policy question under #503 — no action needed from me there, that's yours to decide whenever you're ready.

[mparrett]

Followed up on the review comments in 7e61f8da and refreshed the PR description with the post-rebase measurements.

• Rebased the stack onto the updated feat(bytecode): opt-in DEFLATE compression for .lgb bundles #501 head.
• Replaced the direct os.Create write with a same-directory temporary write followed by sync, close, mode preservation, and atomic rename. A failed encode leaves the existing checked-in bundle untouched; tests cover both failure preservation and successful replacement/cleanup.
• Kept --compress default-off. The current M2 measurement is 297,359 → 89,776 bytes for the core and 13,844,450 → 13,629,794 bytes for the binary, with warm boot moving from 7.9 ± 0.8 ms to 9.8 ± 1.1 ms. That keeps default-on as the explicit policy question under Epic: Footprint & constrained targets #503.
• Added checked-in decode and encode benchmarks; go test ./... passes locally.

@nooga / @nnunley, this should close the write-path concern. The only remaining question is the intentionally deferred default-on policy, including the stripping nuance noted for later discussion.

[mparrett]

Note: First time trying this in the repo.

Stack #695 was created successfully

[nnunley]

Reads clean on top of #501. The atomic write nooga asked for is here — CreateTemp in the
destination dir, write → Sync → Chmod → Rename, mode preserved, with tests covering
failure-preserves-destination and no leftover temps. They carry //go:build bootstrap and CI
runs that tag (go.yml:90), so they actually execute.

Confirmed the committed core is untouched: --compress defaults off, core_compiled.lgb is
not in the diff, and generated.sums moves only because it digests lgbgen's own sources. I
deflated the branch's core at level 9 and got 301,252 → 89,806 bytes against your 297,359 →
89,776; the difference is core regeneration since you measured.

Two nits: no directory fsync after the rename (durability, not atomicity), and the size is
reported from the temp file's Stat rather than the destination — same bytes, just reads odd.