feat(lg): split debug info from emitted bytecode

[mparrett]

Summary

Adds lg -strip for -c and -b, producing a smaller runtime artifact while preserving its source maps and local-variable tables in a digest-bound debug companion.

By default:

• lg -strip -c app.lgb app.lg writes app.lgb and app.lgb.debug.
• lg -strip -b app app.lg writes app and app.debug.
• -debug-output <path> selects a separate symbol-store path.

Why

Bundle-composition work in #623 showed source maps and local-variable tables are the largest removable bytecode sections, while execution does not consult them. The original draft simply discarded those sections, which made production failures impossible to reconcile with source locations.

This revision treats stripping as split debug information: deployments can ship the smaller artifact while retaining the companion in build artifacts, a symbol store, or an on-demand filesystem.

On the current fib sample:

• Full .lgb: 1,565 bytes
• Stripped runtime .lgb: 1,421 bytes (−9.2%)
• Debug companion: 201 bytes

The companion is archival/debug data; it is not required for execution.

Design

• bytecode.SplitDebug emits the stripped LGB plus a versioned companion containing source maps and local-variable tables by chunk index.
• FlagDebugSplit marks artifacts whose debug information was externalized.
• The companion stores the SHA-256 digest of the exact stripped payload. A mismatched companion is rejected rather than producing misleading tracebacks.
• lg app.lgb, standalone bundles, and lg-runtime automatically load a co-located .debug companion.
• LG_DEBUG_FILE=/path/to/app.debug loads one from another location; LG_DEBUG_FILE= explicitly disables loading.
• Without a companion, the stripped artifact still runs and reports unlocated frames.

Scope

This applies to program bytecode emitted by -c and -b. Stripping the embedded core and extending the mechanism to -w/WASI remain follow-ups.

Validation

• go test -count=1 ./pkg/bytecode
• go test -count=1 -run TestStripDebugCompanion ./test/e2e
• go test -count=1 -run TestRuntimeOnly ./test/e2e
• go vet ./...
• Repository-pinned golangci-lint: 0 issues
• End-to-end coverage for located and unlocated tracebacks, mismatched companions, custom symbol paths, standalone bundles, and lg-runtime

[nnunley]

So, this is technically fine. But from a high level, I have concerns.

I think a better high level strategy would be to have the symbolmap put in some sort of external file that could be loaded, included a on-demand wasi-filesystem for the wasm side of things, or fetchable from our distribution channels.

Stripped binaries/cores need some way of re-inflating their sources so you can get proper stack traces when errors happen. So either we preserve the sourcemap somewhere, and provide a way of fixing stack-traces after the fact, or we figure out a more compact representation (like the zstandard compression you introduced, or storing interned symbols in a prefix or suffix trie based table)

We could potentially make the stack traces more like https://github.com/theseion/Fuel (from smalltalk).

[nnunley]

I'd suggest you create a sister option to generate a sourcemap connected to the stripped .lgb file. My broader concern with this is that you end up with a non-debuggable binary that will fail with no way of reconciling error tracebacks from stack traces.

[mparrett]

Yeah I appreciate the ideas and direction you shared, that's probably the right way to go. I explored some of those options but haven't got to publishing yet. 100% valid concern.

[mparrett]

@nnunley I addressed the debuggability concern in 65a67a8.

-strip now implements split debug information instead of irreversibly discarding it:

• -c and -b emit a smaller runtime artifact plus a versioned <artifact>.debug companion containing the removed source maps and local-variable tables.
• The companion is bound to the exact stripped payload by SHA-256; a mismatched companion is rejected.
• lg, standalone bundles, and lg-runtime automatically restore source locations from a co-located companion.
• -debug-output selects a symbol-store path, and LG_DEBUG_FILE loads an archived or mounted companion at runtime.
• Omitting the companion still permits the smaller artifact to run, with unlocated frames.

This does not add network/WASI fetching yet, but the versioned companion format and explicit filesystem path provide the hook for that follow-up. I added end-to-end coverage for restored tracebacks, companion-free execution, mismatch rejection, custom paths, bundles, and lg-runtime, rebased onto current main, and marked the PR ready for review.

[nnunley]

You could add a sister wasm bundle that could be pulled in on the wasm side. A more ambitious project on the wasm side would be: https://tc39.es/ecma426/, generating an actual wasm source map. But that might actually require us to compile to wasm, which is a different can of worms, but possible.

[mparrett]

I dug into the ECMA-426 direction. It is worth investigating, but it is a different debugging layer from the companion added here.

ECMA-426 WebAssembly mappings map byte offsets in the generated Wasm binary back to original sources. Our current Wasm path embeds LGB bytecode as data inside a Go/TinyGo-compiled interpreter. The debug data in this PR maps LGB instruction pointers to .lg locations, but it has no mapping from final Wasm byte offsets to those instruction pointers. Multiple let-go forms can also execute through the same interpreter instructions, so mechanically converting the LGB companion to a .wasm.map would not produce a meaningful browser source map.

A practical follow-up is still the sister asset you suggested: for lg -w -strip, embed the stripped LGB and emit/fetch a digest-bound .lgb.debug companion before decoding it in the Wasm runtime. That would preserve let-go error locations without increasing the production Wasm payload.

A genuinely useful ECMA-426 map back to .lg likely needs compiler cooperation and possibly direct/AOT let-go-to-Wasm lowering. I opened #703 to track a research spike covering both the near-term companion and the larger source-map/DWARF question. This does not need to block #624.

[nnunley]

The split-debug design is right, and the digest-before-parse ordering is the part I most
wanted to see: a tampered digest, a truncated payload, and a companion beside an unstripped
artifact all fail or are ignored the way they should. Four things before this can land.

1. Flag-bit collision with #501, and it is not just textual. tags.go:14 takes
FlagDebugSplit = 1 << 3; #501 takes the same bit for FlagCompressed. #624 sets it on a
version 2 header, and #501's new readHeader mask rejects unknown bits per version, so an
lg built from #501 reading a -strip artifact from this branch gives:

error: decoding cross.lgb: unsupported LGB flags 0x0008 for version 2 (supported: 0x0007)

This PR is the second of the two by nine days, so it is the one that moves — but "move to
1 << 4" is the weaker half. Git will not catch the next one. I reproduced that: two branches
each adding a flag, one inside the const block and one as a standalone const further down —
which is the shape you and #501 actually have — merge clean, leaving two names on 1 << 3
with no conflict and no warning.

I have put the structural fix on #501, since that PR already owns the version/mask machinery
this depends on: positional 1 << iota with a flagsEnd sentinel so a bit value is never
written down, plus a per-version admitted-flags table. If that lands there, the change here is
to append FlagDebugSplit after FlagCompressed (giving it 1 << 4) and add it to the
version set that admits it — rather than choosing a number at all.

2. -z -strip would silently drop compression. StripDebug re-encodes through
ModuleBuilder, whose Build() recomputes Flags, so any FlagCompressed on the input is
lost with no warning. Worth an explicit decision and a test once both are in.

3. The encoder fix is unmentioned and is what makes this CONFLICTING. Commits
feee8f759/f7603bc0e change findChunkIndex to a pointer-accurate liveIdx map and
regenerate core_compiled.lgb. That is a genuine correctness fix for Func→chunk binding
when two chunks share a body, it stands on its own, and it reads as a pure -strip change
right now. Splitting it into its own PR would make both easier to review, and would drop the
binary-artifact conflict GitHub cannot resolve.

4. os.Executable()'s error is ignored at lg.go:496 and cmd/lg-runtime/main.go:126;
on failure the companion path degrades to ./.debug in the cwd, which is a plantable
filename. Digest verification makes it inert today, but an explicit check is cheap.

Measured, since the body quotes numbers: on a fib sample built from this head, -strip takes
the artifact 444 → 374 bytes (−15.8%) with a 121-byte companion. Your 1,565 → 1,421 + 201
(−9.2%) is a different sample; the direction and rough magnitude agree.

Smaller: lg.go:216's deferred Close plus the explicit one at :236 double-closes the
bundle output; compileLG now buffers and uses os.WriteFile(…, 0644) where it previously
streamed through os.Create (0666 & umask); and LoadDebugCompanion costs a stat/open on
every boot, decoding the full companion even when the artifact turns out not to be split.