Skip to content

Consolidate the spread-out native-behavior mechanisms (registration + interop emitters) #531

Description

@nnunley

What "spread out" means concretely

Defining one piece of runtime behavior — "make upper-case callable from let-go" —
currently has six mechanisms, and which one a given function uses is an accident
of when it was written:

# Mechanism Where Scale Namespace/name/privacy decided
1 Imperative ns.Def + anonymous Wrap closure pkg/rt/lang.go ~384 Def sites, ~5,800 lines at the call site, imperatively
2 //lg:native-annotated named Go funcs → generated registrar pkg/rt/native_prims.go (+_lifecycle), emitted by cmd/lginterop -primitives 14 directives so far at the definition, declaratively
3 Direct-call-eligible builtins (dual-use: interpreter registration AND AOT direct-call target) pkg/rt/builtins/ growing in lang.go, still imperatively
4 Pure .lg core, compiled to bundle + lowered tree pkg/rt/core/*.lg most of the stdlib in the .lg source
5 External-package interop: go/types scan → gogen emit cmd/lginterop main.go + scripts/lginterop.lgpkg/rt/interop_<pkg>.go xxh3 today, glplat pending generated installer
6 Reflection-based host interop (boxed values, method dispatch) pkg/rt/host_interop.go, vm.MustBox ambient n/a (dynamic)

The cost is exactly the #411 diagnosis: re-homing clojure.string primitives,
adding an io ns, or privatizing the ~82 compiler internals means hand-editing
hundreds of imperative call sites in mechanism 1, because that's where most
behavior still lives. Meanwhile mechanisms 2 and 3 already demonstrate the fix
but cover a sliver of the surface.

Correction to the epic text first

#528 lists scripts/lginterop.lg under "Superseded but still alive." It is not
superseded — it is a live runtime dependency of mechanism 5.
cmd/lginterop's
external-package mode reads scripts/lginterop.lg from the repo root on every
run (writeGenScript, cmd/lginterop/main.go:419), concatenates the exports onto
it, and shells out to lg to render the interop file. Deleting it today breaks
lginterop -packages. (The #467 corpus exclusion was about ir-stress coverage,
not liveness.) The retirement is still the right goal — but it's a port, not a
deletion, and this proposal sequences it.

Target end state

One mechanism per behavior class, chosen by a decision rule, with the
namespace/name/privacy always declared at the definition:

  • Expressible in Clojure, not perf-critical.lg core (mechanism 4).
    Aligns with Write more of let-go in let-go: capability unlocks for .lg + AOT lowering #519; every migration here shrinks the Go surface.
  • Host-touching or perf-critical internal primitive → named Go func +
    //lg: directives (mechanism 2). The registrar generates registration;
    re-homing a primitive is a one-line directive edit.
  • Direct-call-eligible uniform builtinspkg/rt/builtins/ (mechanism 3),
    registered via the same directives so 2 and 3 differ only in signature shape,
    not in registration plumbing.
  • External Go packagescmd/lginterop -packages (mechanism 5), with the
    emitter ported to Go (below).
  • Dynamic fallbackvm.MustBox/reflection (mechanism 6) stays as-is; it's
    the ambient capability the generated code leans on, not a competing registry.

Mechanism 1 — the anonymous-closure ns.Def wall — is the one that goes away,
drained into the classes above.

Slices (each behavior-preserving, PR-sized, in dependency order)

S1 — Make cmd/lginterop self-contained; the emitter STAYS in lg.
The emitter is not rewritten — lg is implemented in lg wherever possible, and
the lg-in-lg bootstrap is a feature to preserve, not debt. What retires is the
delivery mechanism: the repo-root file read, the temp-file concatenation, and
the shell-out to a separately built lg binary. Two stages:

  • S1a (works today): move scripts/lginterop.lgcmd/lginterop/ lginterop.lg with //go:embed, expose gogen.lg the same way (small embed
    var in pkg/rt/gogen), and evaluate in-process — cmd/lginterop already
    links pkg/compiler/pkg/vm, so this deletes the exec path and the
    repo-root/pre-built-lg requirements without touching the emitter. Exports
    can be handed to (lginterop/generate …) as vm values directly instead of
    via string serialization.
  • S1b (end state): generate a pure-Go executable from the lg — AOT-lower
    gogen.lg + lginterop.lg through the existing lowering pipeline (the
    core_go_lowered/ pattern), so the shipped binary is pure Go while the
    source of truth stays .lg, regenerated by make generate like every other
    lowered artifact. Gate: both namespaces lower cleanly under
    ir-stress lower-go (note chore(ir-stress): exclude scripts/lginterop.lg from corpus — stale gogen/parse-stmts #467 excluded scripts/lginterop.lg from the
    corpus as stale, so this bucket must go green first — that work also grows
    lowering coverage, which is its own payoff).

Verification for both stages: regenerating interop_xxh3.go is byte-identical.
(Folds in the already-listed dead if smart branch.) The #528 line item is
then satisfied correctly: scripts/lginterop.lg disappears from scripts/
because it moved and got lowered, not because it was rewritten.

S2 — Finish the #411 registrar and drain lang.go by namespace.
Adopt //lg:native for the clojure.string primitive family first (it's the
#411 exemplar: veneer over wrongly-homed Go prims), then the io isms, then
privatize bucket ③ wholesale via //lg:private. Each namespace is one PR:
move the closure body to a named func, annotate, regenerate, delete the
ns.Def site. Behavior gate: ns-publics diffs empty except intended
privatizations (those get their own PR per the epic's gate).

S3 — Collapse the .lg veneers that exist only to fix naming.
Once S2 re-homes primitives into their right namespaces, veneers like
clojure.string's (defn upper-case [s] (core/upper-case (str s))) reduce to
either nothing (primitive registered under the right name directly) or a real
.lg implementation (mechanism 4) where #519 wants one.

S4 — Ratchet so the spread doesn't regrow.
A test that counts anonymous vm.NativeFnType.Wrap(func occurrences in
lang.go and fails if the count rises (same shape as the bench-ratchet).
New primitives must use directives; the count only goes down.

Non-goals

Effort sketch

S1a is small and mechanical (embed + in-process eval), bounded by the golden
byte-identical test. S1b's cost is whatever it takes to get gogen.lg +
lginterop.lg green under ir-stress lower-go — that's lowering-coverage
work with value beyond this epic, and S1a doesn't block on it. S2 is
mechanical per-namespace, parallelizable across worktrees. S3 falls out of
S2. S4 is an afternoon.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    • Status
      Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions