demo(shell): WebGL renderer + post-process shader overlay (CRT, water, mode7) - DEMO/POC - #168
Draft
mparrett wants to merge 14 commits into
Draft
demo(shell): WebGL renderer + post-process shader overlay (CRT, water, mode7) - DEMO/POC#168mparrett wants to merge 14 commits into
mparrett wants to merge 14 commits into
Conversation
Two-part spike, layered so each half stands alone:
(a) Load @xterm/addon-webgl (same jsdelivr lane as xterm/fit) after
term.open(), with preserveDrawingBuffer=true and a context-loss
fallback to the DOM renderer. ?renderer=dom opts out.
(b) A WebGL2 overlay canvas re-samples the renderer's canvas each rAF
through a Shadertoy-convention wrapper (iChannel0/iResolution/iTime
+ mainImage), running Timothy Lottes' public-domain CRT filter as
ported for Ghostty (ghostty-shaders crt.glsl): scanlines, shadow
mask, tube warp, vignette, tone curve. ?crt=off opts out.
Non-obvious bits:
- preserveDrawingBuffer is required: the overlay samples outside the
addon's own rAF, and a non-preserved WebGL backbuffer reads back
blank after compositing.
- .xterm-screen holds TWO canvases; the first is xterm's transparent
link-underline layer, which samples as solid black. The overlay
selects :not(.xterm-link-layer).
- Scanline density (the shader's SCALE define) is substituted per
devicePixelRatio at compile time: 1/3 on high-DPI, 2/3 on low-DPI,
per the shader's own guidance — 1/3 at dpr1 muddies the text.
- The overlay tracks the source canvas' on-screen box per frame, so
letterboxing/refits need no extra wiring; pointer-events:none keeps
input on the terminal.
Known costs: the per-frame texImage2D from the renderer canvas is a
GPU readback (Chrome logs ReadPixels stall warnings). Fine at terminal
sizes; a shared-context or OffscreenCanvas route would remove it if
this graduates from spike.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Lottes filter is a re-scanning scaler: it fetches the input on a virtual grid of iResolution*SCALE. The previous high-DPI default (0.333, from the shader's own header) resamples a dpr-2 terminal at 2/3 of CSS resolution — below a non-retina display — which is the text softness and RGB fringing seen on a retina Mac. That advice assumes pixel-art sources, not text. New default anchors the virtual CRT at CSS resolution (SCALE = 0.5 on dpr>=2, 0.667 on dpr 1): text as sharp as a 1x display, scanline pitch ~1 CSS px. ?crtscale=0.2..1 overrides for taste (lower = chunkier scanlines, blurrier glyphs). toFixed(8) keeps the substituted define a float literal — a bare "1" would be an int and fail to compile at crtscale=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second Shadertoy-convention shader in the overlay: ghostty-shaders water.glsl, animated caustics that ripple-distort the terminal sample and layer watery highlights — first user of the wrapper's iTime. ?shader=water selects it; default stays crt. Only the selected source compiles, so registry entries can't collide in the GLSL namespace. water.glsl ships no license header upstream (it's the widely-copied Shadertoy caustic loop) — chase provenance before any upstream PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SNES Mode 7 feasibility demo: idle affine spin + zoom breathing over the terminal texture. Registry entries can now pick their sampling — mode7 uses NEAREST (chunky rotated pixels, the authentic look; LINEAR would smear them) and REPEAT wrap (tiles the screen into an endless plane instead of edge-smearing when the spin samples off-screen; WebGL2 allows REPEAT on NPOT textures). crt/water keep LINEAR+CLAMP. Idle-animated for now. The real use is game-signaled transitions (descend swirl, encounter zoom) once an OSC trigger channel exists, plus a map-rect uniform to scope the transform away from the HUD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 23s breathing period meant the zoom-in phase was easy to never see; 14s cycle now, dipping to ~4x magnification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mparrett
changed the title
mparrett
changed the title
Game side (render.lg): emit a private OSC escape ("water=1,hurt=42")
before frame flush, only when the signal set changes. Real terminals
discard unknown OSC silently (native TUI unaffected); the emission is
output-only, so determinism/replay are untouched. Signals: water =
player has the :swimming status; hurt = integer percent ramping in
below 35% health.
Shell side: an xterm OSC handler parses the pairs into fxSignals.
water swaps the active shader to the water effect for the duration;
hurt drives a Doom-style red throb the wrapper applies after
mainImage, composing over whichever shader is active. Programs now
compile lazily and cache (null on failure — no per-frame retry),
texture sampling params re-apply per shader, and uniforms are set
every frame so switches don't leave stale values. Signals are
window-exposed (with a receive counter) for manual poking and probe
assertions on the demo build.
Also adds bloom to the registry: the 24-tap golden-spiral glow from
the same qwerasd gist as the CRT adaptation, ported to ES 3.00 (array
constructor syntax; `step` renamed — it shadows the builtin).
Validated end-to-end under Playwright: the boot frame's emission
reaches the shell handler through wasm stdout (receive counter fires),
and synthetic signals produce the water swap and red pulse on screen;
?shader=bloom compiles and renders.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Pushed a follow-up commit that answers the "how do effects trigger" question from the description: a game→shell signal channel over a private OSC escape (7777), demo-wired to two signals — the 🤖 Generated with Claude Code |
Console (whitelist, still read-only — the emissions are output-only terminal escapes, the world is untouched): (crt)/(water)/(bloom)/ (mode7) toggle a shader override via a new shader= OSC key (the shell treats a repeated override as toggle-off, unknown/empty clears); (hurt N) forces the red pulse; (fx) clears everything. No-ops in a real terminal. Contrast: water's caustic multiply bottoms out ~0.6x and left swimming notably dark. The wrapper now compiles a mid-anchored contrast stretch + brightness lift per shader (registry defaults; water gets 1.2/+0.1), applied after mainImage and before the hurt tint. ?contrast= / ?bright= override all shaders for tasting. Override precedence in the frame loop: console shader override, then the swim signal, then the URL base. Validated under Playwright driving the real dev console: (hurt 80) sets 0.8, (bloom) toggles on and off, (fx) clears, receive counter ticks per command. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR-preview smoke gate asserts the title sentinel appears in .xterm-rows textContent — DOM-renderer output. With the WebGL renderer active, xterm draws glyphs to a canvas atlas and never populates DOM rows, so the sentinel exists only as pixels and the wait times out. (The CI runner's WebGL working is what made it fail.) The shell now exposes window.xsofyTermText, a renderer-independent readout of the terminal buffer via xterm's buffer API; the smoke prefers it and falls back to DOM scraping for builds without it. Verified locally with the exact gate: node smoke.mjs --dir dist → OK, sentinel at ~7s on the WebGL build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Fills the PLAY pane's first empty cell (col 1, row 3) with a tile that sends backtick — the dev-console key, otherwise unreachable on touch devices with no physical keyboard. The console is dev-gated game-side (?mode=dev), so the key is a no-op for regular players. Verified under Playwright in a portrait viewport: the tile renders and tapping it opens the console. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
read-string stops at the first form, so a bare "hurt 20" parsed as just the symbol hurt and the 20 silently fell away (the command then used its default 70). The line is now read wrapped in parens — bare words, bare-with-args, and user-typed (...) all arrive as one list, with the latter's double nesting unwrapped. Verified natively (run-command probe: all three shapes dispatch with args intact) and in the browser console end-to-end (bare "hurt 20" drives the shell signal to 0.2). A note for probe authors: the wasm console's 30ms poll loop drops keystrokes from Playwright's default typing rate — type with delay>=150ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keypad (to extract later): WAIT and ENTER swap — wait is the high-frequency key, so it moves up to the PLAY pane's top row and enter takes the old wait corner; y/n swap to n/y in both PLAY variants; the a-z pane gains ( and ) left of z (portrait: they replace the stagger spacer, backspace slims 3->2; landscape: thin span-1 keys, esc/backspace slim to fit) — parens make the dev console's command syntax typeable on touch. Shaders now default OFF: no ?shader= means no base shader, and the overlay sits dormant — hidden, no texture upload, no per-frame GPU readback — until ?shader=, a game signal (swim), or a console override activates one. A dormant-then-activated overlay was already supported by the lazy program cache; this just makes null a valid base. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mparrett
added
deploy-preview
14 reads small on laptop screens, not just phones. Portrait keeps its phone-tuned 22; a stored user choice (the font chip's localStorage value) still wins over both defaults, so nobody's setting changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Console-first wiring for the pickup pulses; the in-game emits (on actual rune/item pickup) are the follow-up slice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pulse=<kind> on the OSC channel fires a transient additive wash: instant attack, ~500ms quadratic decay, composed after mainImage so it rides over any shader. Palette: rune = purple 0.65 peak, item = white 0.4 peak. Every occurrence retriggers (the envelope runs shell-side off pulseAt), unlike the level-style signals. Also adds a plain passthrough registry entry the frame loop wakes automatically while any transient effect is live (pulse or hurt) — with shaders defaulting OFF, effects previously had no canvas to show on. The overlay still goes fully dormant once the effect ends. Verified under Playwright through the real console: (pulse rune) wakes the dormant overlay, decays back to dormant in ~1s, bare "pulse" defaults to item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a proof of concept that came from the thought: "since xterm has an opengl canvas, can we do fun stuff with opengl?"
I turns out we can. I am not sure we should. Some concepts implemented here we've discussed before game state emits to js and triggers (e.g. sound effects), and might be worth extracting separately. But this is fun to look at. Let me know what you think!
A few demo shaders:
water
Screen.Recording.2026-07-21.at.6.57.44.PM.mov
mode7
Screen.Recording.2026-07-21.at.6.59.12.PM.mov
crt
Screen.Recording.2026-07-21.at.6.58.36.PM.mov
Draft/spike for discussion — the terminal gets a WebGL renderer and a shader post-process pass, in two layers that stand alone:
(a) WebGL renderer.
@xterm/addon-webglfrom the same jsdelivr lane as xterm/fit, loaded afterterm.open(), with a context-loss fallback to the DOM renderer.?renderer=domopts out.(b) Shader overlay. A WebGL2 canvas re-samples the renderer's canvas every frame through a Shadertoy-convention wrapper (
iChannel0/iResolution/iTime+mainImage), so any Shadertoy-style shader drops in. Aligned to the terminal canvas,pointer-events: none, tracking letterbox refits per frame.?crt=offopts out while keeping (a).Four shaders in the registry, selected by
?shader=; programs compile lazily and can switch at runtime. Default is none — the page starts with no shader applied (and the overlay sits dormant, costing nothing) until?shader=, a game signal, or a console override activates one:crt— Timothy Lottes' CRT filter (public domain / Unlicense, as adapted for Ghostty): scanlines, shadow mask, tube warp, vignette. The filter re-scans the input atiResolution × SCALE, so SCALE below 1/dpr costs text sharpness; the default anchors at CSS resolution (0.5 on retina, 2/3 at dpr 1),?crtscale=0.2..1to taste.water— animated caustic ripple.bloom— 24-tap golden-spiral glow, from the same qwerasd gist as the CRT adaptation.mode7— SNES-style affine spin/zoom demo, sampled nearest-neighbor on a tiled plane (registry entries choose their sampling: NEAREST and REPEAT here, LINEAR and CLAMP for the others). Idle-animated; wiring it to game transitions (descend swirl) would ride the signal channel below plus a map-rect uniform so the HUD stays put (not built).Game-signaled effects. The game emits a private OSC escape (
7777,k=vpairs) before frame flush, only when the signal set changes. Real terminals discard unknown OSC (native TUI unaffected), and the emission is output-only, so determinism/replay are untouched. Wired so far: the:swimmingstatus swaps the active shader towaterfor the duration, and health below 35% drives a Doom-style red pulse the wrapper composes over whichever shader is active. The dev console can drive the same channel:(crt)/(water)/(bloom)/(mode7)toggle a shader override (repeat toggles off),(hurt N)forces the pulse,(pulse rune)/(pulse item)fire one-shot pickup flashes (purple / white — transient additive washes with a ~500ms decay; a plain passthrough wakes the dormant overlay so they show even with no shader selected; in-game pickup emits are a follow-up),(fx)clears. The console stays read-only: these are output-only escapes. Precedence: console override, then swim, then the URL base.To try it on the deploy preview: open xsofy.quest/pr-preview/pr-168/?mode=dev (the
mode=devparam gates the console), then press`(backtick) in-game to open it and type(help)for the command list.Contrast. Some effects darken the scene (water's caustic multiply bottoms out around 0.6x), so the wrapper compiles a per-shader mid-anchored contrast stretch + brightness lift after
mainImage(water ships 1.2/+0.1);?contrast=/?bright=override any shader for tasting.Implementation notes:
preserveDrawingBuffer: true— the overlay samples outside the addon's rAF, and a non-preserved backbuffer reads back blank after compositing..xterm-screenholds two canvases; the first is the transparent link-underline layer, which samples as solid black. The overlay selects:not(.xterm-link-layer).texImage2Dfrom the renderer canvas is a GPU readback (Chrome logs ReadPixels stall warnings). Fine at terminal sizes in testing; a shared-context or OffscreenCanvas route would remove it if this graduates..xterm-rowswith DOM text, which silently blinds DOM-scraping assertions; the PR-preview smoke gate failed exactly this way. The shell now exposeswindow.xsofyTermText(a renderer-independent readout via xterm's buffer API) and the smoke prefers it, falling back to DOM scraping for builds without it.For discussion
water.glslcame from the ghostty-shaders collection without a license header (it's the widely-copied Shadertoy caustic loop) — provenance needs chasing or the shader swapped/dropped.crtandbloomare clean (same Unlicense gist).Validated with Playwright on the WASM bundle: renderer active, overlay tracking, all four shaders compiling and rendering, input unaffected; the game→shell signal path exercised end-to-end (boot emission reaches the handler; console commands drive the overrides); the CI smoke gate run locally against the WebGL build passes. Deploy preview is live for on-device poking.
🤖 Generated with Claude Code