diff --git a/AGENTS.md b/AGENTS.md index 7b82d1c..8efc715 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,12 @@ it against a local emulator (Docker). A directory that is not a chant project gets a structured `{code: "no-project"}` error from `/api/graph`, not a blank graph. +In a checkout, `behold demo --list` prints a second block under the bundled +one: the eleven workbench entries from `workbench.json`, this repo's own +catalog of the internal estates behold is developed against — the same +`behold demo ` load, `just example name=""` to serve one, and +"The workbench catalog" below for what each is and what it boots. + A running server offers the same catalog over HTTP (#268): `GET /api/demos` lists every bundled demo with `{name, description, requires, satisfiable, reason?, fetches, repo?, target, loaded}` — `satisfiable` is doctor's PATH probe @@ -368,7 +374,9 @@ until the table existed. A new kind is: 1. A `registerMemberKind({ kind, probe, expects, via })` call in `src/member-kind.ts` — the `probe` is sync, read-only and runs no code (a - file's presence, a regex over a root file); `via.tool` stamps the binary and + file's presence, a regex over a root file, or either: the choudoufu probe + takes the `estate.chdf.hcl` sidecar, choudoufu's leading form, OR a `live {` + block in a root *.tf, #387); `via.tool` stamps the binary and version that would answer, which is the version half of `memberIr`'s cache key; `via.read` is the one uncached read, source or live per `opts`. The module imports nothing from the read path at runtime (src/chant.ts @@ -396,9 +404,110 @@ until the table existed. A new kind is: string is `chant`. Anything behold boots for a kind goes through `assertScratch` first. +### The workbench catalog + +`demos.json` is the catalog that ships. `workbench.json`, read beside it and +deliberately absent from package.json's `files`, is the catalog of internal +estates this checkout is developed against (#386, #388). Four rules, and +src/demos.ts holds them: + +1. A third source, `local`, and a second file. A local entry's `path` is + relative to the directory of the catalog file that named it — the intentius + checkouts are siblings, so the workbench writes `../choudoufu`, + `../waterpark`, `../chant`, and the file stays committed and reproducible. + An entry whose path is not checked out is unsatisfiable exactly as a + missing binary is: `--list` and `/api/demos` say so, CI stays clean. + `BEHOLD_WORKBENCH=` names a catalog elsewhere; a workbench name that + collides with a bundled one is dropped with a stderr line. +2. In place or copied, said explicitly. `inPlace: true` serves `path` where it + sits; the default copies to `behold-demos/` with the bundled filter. + Anything whose setup writes into the tree — `init`, `apply`, a rendered + generator — is copied or generated into the target, never run in a + checkout, which is how #366's "behold never runs `choudoufu init` in a + served project" survives. An `inPlace` entry with a `setup` says so in its + description. +3. A generator is a source. An entry with no `path` at all renders its estate + into an empty target through its own `setup`, which runs with cwd = the + target and two extra variables: `BEHOLD_WORKBENCH_DIR` (the catalog file's + directory, so a script can reach `../choudoufu`) and `BEHOLD_DEMO_NAME`. + The up script writes the matching down script into the target, the way the + bundled choudoufu demo ships `scripts/choudoufu-down.sh`. +4. `CHOUDOUFU_BIN`. `choudoufuBinary()` (src/choudoufu-member.ts) names the + binary and every spawn, the doctor probe and the demo requirement check go + through it; the doctor line prints which binary answered. Workbench scripts + spell the same fallback, `${CHOUDOUFU_BIN:-choudoufu}`. + +`just example name=""` serves one. Scratch discipline is unchanged: an +emulator a workbench entry boots is `behold-wb-` on its own port, and +the up scripts (`workbench//up.sh`, helpers in `workbench/lib/`) assert +the `behold-wb-` prefix and the not-4566 rule themselves, in bash, because they +are the boot site. Each writes the matching `scripts/down.sh` into its target. + +**The eleven entries**, in catalog order. The counts are measured, not +estimated — every entry the e2e below can reach on a developer machine prints +its own graph and overlay counts as it runs. + +| entry | what it serves | needs | +|---|---|---| +| `chant-getting-started` | chant's own getting-started example from `../chant`, in place: the source graph, 8 nodes, no substrate. The one-second answer to "did I break plain chant reading?" | that example's own `node_modules` — nothing is installed in your chant tree | +| `chant-local-cloud-trio` | chant's local-cloud-trio, in place: one project declaring across aws, azure and gcp, 8 nodes and 2 edges of source | the same | +| `fountain-ops` | `../fountain-ops` in place with `--env local` — the mature estate on your working checkout | docker, k3d, kubectl, jq, just. **The one entry whose setup runs in your working copy**: the checkout's own `just up`, a five-minute k3d cluster, and it switches your kubectl context. `just down` there removes it; behold never does | +| `choudoufu-workbench` | the live-mv workbench's four estates copied out of `../choudoufu`, composed with `--env live`: 42 cards — 24 bound, the 12 team cards reading `owned by tlmig-sample-monolith`, 6 neutral | docker, choudoufu | +| `choudoufu-cohort-s3` | `estate-gen`'s `s3` cohort rendered into the target — a sidecar-declared estate (#387): 6 cards, all neutral, because the apply stops where floci answers S3 Control tag reads on a hostname that does not resolve | docker, choudoufu, go, terraform | +| `choudoufu-cohort-iam-ecr` | the same for `iam-ecr`, the one cohort floci implements end to end: 6 cards, all 6 bound | the same | +| `choudoufu-cohort-ec2-networking` | the same for `ec2-networking`: 49 cards, the widest roster, all neutral (floci refuses a transit gateway call and the apply stops) | the same | +| `terralith-1` | `terralith-gen` at scale 1 plus the `estate.chdf.hcl` sidecar it omits, applied by choudoufu from nothing: 79 cards, all 79 bound | docker, choudoufu, go | +| `terralith-4` | the same at scale 4: 301 cards, all 301 bound. The estate behold is sized against | the same | +| `terralith-4-adopt` | scale 4 again, but STOCK terraform applies it first, so nothing wears a marker: 301 cards — 85 UNOWNED, 84 bound by derived identity alone, 132 neutral. One `choudoufu live-import` line, which the up script prints and you run, and all 301 read bound | docker, choudoufu, go, terraform | +| `waterpark` | `../waterpark/access` in place as a bare Terraform directory (#384): five roots as boxes, 58 cards, 49 edges, nothing written under the estate | `@intentius/chant-lexicon-terraform` + `@cdktf/hcl2json` beside behold — optional peers behold declares and does not install | + +**The scratch, by name and port.** Each emulator is the entry's own, booted by +its up script and removed by the `scripts/down.sh` that script wrote into the +target — never by pattern. + +| entry | container | host port | +|---|---|---| +| `choudoufu-workbench` | `behold-wb-choudoufu-workbench` | 4651 | +| `choudoufu-cohort-s3` | `behold-wb-choudoufu-cohort-s3` | 4652 | +| `choudoufu-cohort-iam-ecr` | `behold-wb-choudoufu-cohort-iam-ecr` | 4653 | +| `choudoufu-cohort-ec2-networking` | `behold-wb-choudoufu-cohort-ec2-networking` | 4654 | +| `terralith-1` | `behold-wb-terralith-1` | 4655 | +| `terralith-4` | `behold-wb-terralith-4` | 4656 | +| `terralith-4-adopt` | `behold-wb-terralith-4-adopt` | 4657 | +| `chant-*`, `waterpark` | none — source reads, no substrate | — | +| `fountain-ops` | the k3d cluster `fountain-local`, which is the checkout's, not behold's | — | + +The port is the floci's host binding and the entry's `serve.spawnEnv` +`AWS_ENDPOINT_URL` at once, so the estate's own applies and the choudoufu +behold spawns talk to the same emulator. + +**The e2e** is `just e2e-workbench` (`e2e/workbench-e2e.sh`, #391): every entry +loaded through `behold demo --port

` on its own port +from 4720 up, `/api/graph` asserted to carry nodes and a live entry's +`/api/overlay?env=live` to answer with its bound/unowned/neutral split, timings +per entry; eight entries and 475s on the machine this was written on. +`terralith-4-adopt` is asserted twice — 85 UNOWNED, then the +`live-import` line the up script printed, run by the script itself in the +target the way a person would, then 301 bound — because that write is the +person's, never behold's (#372). `waterpark` is checked for an untouched +checkout afterwards. `missingRequirements` decides what runs: a missing binary, +an unchecked-out sibling, an uninstalled chant example or the absent Terraform +lexicon each print `skip: : needs …`, so in CI every entry skips and the +run exits 0. `fountain-ops` is skipped by name everywhere, for the reason in +the table. A `behold-wb-*` container left standing at the end fails the run. + +The seeded catalog (#389) found one thing missing in src/: a LONE choudoufu +estate — every generated entry is one — served the no-project card, because the +single-project read is `chant graph

` and such a directory has no lexicon +for it to read. `servesAsEstate` (src/member-kind.ts) is the predicate that +routes one directory of a non-chant kind through the estate compose path, where +the member's own kind reads it. One member composes exactly as four do, ids +namespaced under the member's short name, so the graph, the pane and the morph +agree on what a node is called. + ### Rendering a Terraform estate -A Terraform estate reaches behold through chant, not through a member kind. +A Terraform estate reaches behold through chant. `@intentius/chant-lexicon-terraform` reads the HCL an estate already has and emits one entity per block, so **a Terraform estate is a chant project whose only lexicon is a reader** and `chant graph --format ir` serves it like any @@ -430,8 +539,71 @@ resources, four fifths of it not infrastructure. Nothing is dropped silently: `terraformElisionNote` says what is not drawn and where to see it, the way `edgelessNote` says why a view has no edges. -**Do not invent edges.** A stock Terraform estate has none until chant#2265 -resolves a block's `"${…}"` references. The one relationship that looked -derivable — a cross-root read by name — was measured and refused (#381): both +**Do not invent edges.** They arrive from chant or not at all: the fixtures here +were recorded when a Terraform IR carried none, and lexicon 0.61.0 (chant#2265, +which resolves a block's `"${…}"` references) draws 390 over water park's five +roots with no change on this side. The one relationship that looked derivable +without it — a cross-root read by name — was measured and refused (#381): both ends carry the same unresolved interpolation, so a match would be a coincidence of variable naming. A data source says what it reads as a row instead. + +**Serving a directory that declares nothing (#384).** Every estate this lane +exists to draw is a directory of `.tf` files and nothing else, and #378 chose +not to ask one for a `chant.config.ts` of its own — INTENTIUS/waterpark#88 was +withdrawn because the estate is more useful untouched. So `behold serve +` generates the reader config itself, outside the estate, and +points chant at it. The Invariant's one in-project write stays +`.behold/layout.json`; nothing is written under the served directory, and +`src/terraform-member.test.ts` asserts the estate's source stamp is unchanged +across a read. Three decisions, each a trade #384 left open and each measured on +water park's `access/` before it was taken: + +1. **The lexicon is opt-in.** chant resolves the lexicon from the config file's + own location, so the generated project has to see it — and making it a + dependency would put `@cdktf/hcl2json`, a ~1.8 MB wasm blob, in the install + of every user who serves a chant project. `@intentius/chant-lexicon-terraform` + and `@cdktf/hcl2json` are therefore **optional peers**: declared in + package.json (the only place their versions are named — the refusal reads + them from there), never installed by behold, probed at serve and doctor time, + and refused with the one install line and where behold looked. The same gate + `behold demo` puts on a binary it does not ship. The lexicon's own chant peer + is what moved behold's `@intentius/chant` floor to `^0.61.0`: chant 0.54 + loads no published version of it (`applyLineage is not a function`). +2. **Roots are discovered, and the skips are reported.** A root is a directory + with a `.tf` declaring a line-start `terraform {` or `provider "` block + beside a `resource`, `data` or `module` block — a regex probe at the depth + the choudoufu probe uses, no HCL parsed. #384 proposed the first half alone + ("what a root has and a called module does not") and the estate refuted it: + water park's `modules/persona` is a shared module called by three roots and + its `versions.tf` is `baseline`'s byte for byte. So two exclusions stand + beside the probe — a directory under a `modules/` segment (Terraform's own + standard module structure; the roots that call it draw its blocks already) + and one with nothing to draw (`access/backends` is two backend fragments) — + and both are named in the graph's note with their reason, the way + `terraformElisionNote` names what a zoom left out. Measured on `access/`: + five roots (`envs/prod`, `identity`, `github`, `baseline`, + `satellites/waterpark-runner`), two skipped, `envs/dev` neither drawn nor + reported because it holds only a README. +3. **A `terraform` member kind, after chant and choudoufu.** #378 said there is + no such kind and meant it about *reading*: behold parses no HCL and the + render goes through chant. A kind whose `read` shells `chant graph` against a + generated config is a scaffold, not a second reader, and it inherits the + probe, the cache stamp, the doctor line and estate composition (#368) for + free — so a Terraform root composes in an estate beside a chant project and a + choudoufu estate at no extra cost. `src/terraform-member.ts` is the whole of + it; `detectProjectShape` answers a directory that is its own member with + `membersFrom: "probe"`, which is what retired #384's `no chant.config.ts + here` dead end. + +The scratch project is `/behold-tf-`: +`behold-*` and cleared through `assertScratch` (src/scratch.ts), one directory +per estate reused across runs, asserted to be outside the estate before a byte +is written. It holds the generated `chant.config.ts` and two symlinks — +`node_modules` to behold's own, which is how the config resolves the lexicon, +and `estate` to the served directory, which is how each root's `dir` is spelled. +The second is not decoration: the lexicon sets a root's module boundary to the +root's own directory when its `dir` resolves outside the project root, so +absolute paths cost every `../modules/x` call the estate makes — 72 nodes and 6 +resources on water park, against 247 and 43 through the symlink. Nothing in the +answer mentions the scratch path (a terraform entity carries `attrs.file` +relative to its root, and no `sourceLoc`). diff --git a/README.md b/README.md index 9505bca..4652962 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,21 @@ Argo CD estate, declared only, so it runs where Docker doesn't; `behold demo carve` is the Terraform peel walkthrough, below). Every loaded demo lands in the panel's recents, so switching between them is the Scope tab. +Working in a checkout rather than an npm install? There is a second catalog, +`workbench.json`, which is this checkout's and is not shipped — it is +deliberately absent from package.json's `files`, so an npm install has +`demos.json` and nothing else. It holds the eleven internal estates behold is +developed against, named by relative path from this repo's root, so an entry +whose sibling checkout you don't have says so in `--list` instead of failing +halfway: chant's own two examples and water park's `access/` Terraform roots +served in place, `../fountain-ops` on its own k3d cluster, the live-mv +workbench's four estates, three `estate-gen` cohorts, and the generated +terralith at scale 1 and 4 — greenfield, and adopted from a stock `terraform +apply` by a `live-import` you run by hand. `just example terralith-4` serves +one (`just example name=""`); each that boots a substrate brings up its +own scratch emulator and leaves a `scripts/down.sh` in the target that removes +it. `just e2e-workbench` loads the whole catalog and asserts each graph. + The catalog is in the panel too (#268): the Scope tab's switcher lists every bundled demo under your recents, one click to copy, install and serve it — demos whose prerequisites are missing stay visible, disabled, saying what to diff --git a/e2e/choudoufu-estate-floci-e2e.sh b/e2e/choudoufu-estate-floci-e2e.sh index 9ead7ed..e83b5c1 100755 --- a/e2e/choudoufu-estate-floci-e2e.sh +++ b/e2e/choudoufu-estate-floci-e2e.sh @@ -35,9 +35,12 @@ cd "$(dirname "$0")/.." PORT="${BEHOLD_E2E_PORT:-4699}" TARGET="${BEHOLD_E2E_TARGET:-$(mktemp -d "${TMPDIR:-/tmp}/behold-choudoufu-e2e.XXXXXX")/estate}" +# behold#388: the binary behold itself spawns, so the run and the tool under +# test never diverge. +CHOUDOUFU="${CHOUDOUFU_BIN:-choudoufu}" if ! docker info >/dev/null 2>&1; then echo "skip: Docker is not running"; exit 0; fi -if ! command -v choudoufu >/dev/null 2>&1; then echo "skip: choudoufu is not on PATH"; exit 0; fi -if ! choudoufu version -json | grep -q choudoufu_version; then echo "skip: choudoufu predates version -json's choudoufu_version (needs 0.16.0 or a build from main)"; exit 0; fi +if ! command -v "$CHOUDOUFU" >/dev/null 2>&1; then echo "skip: ${CHOUDOUFU} is not on PATH (set CHOUDOUFU_BIN to name a build)"; exit 0; fi +if ! "$CHOUDOUFU" version -json | grep -q choudoufu_version; then echo "skip: choudoufu predates version -json's choudoufu_version (needs 0.16.0 or a build from main)"; exit 0; fi if docker ps -a --format '{{.Names}}' | grep -qx behold-choudoufu-floci; then echo "behold-choudoufu-floci already exists — a previous run didn't tear down; \`docker rm -f behold-choudoufu-floci\` and re-run" >&2 exit 1 @@ -101,7 +104,7 @@ code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:$PORT/ap echo "→ 5. by hand: the role moves to team-a; the receipt reads it back" LINE=$(printf '%s' "$M" | jq -r '.moves[] | select(.address == "aws_iam_role.team_a") | .command') ( cd "$TARGET/team-a" && AWS_ENDPOINT_URL=http://127.0.0.1:4650 AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1 \ - ${LINE/choudoufu live-mv/choudoufu live-mv -json} | jq -e '.written == true and .verified == true' >/dev/null ) + ${LINE/choudoufu live-mv/$CHOUDOUFU live-mv -json} | jq -e '.written == true and .verified == true' >/dev/null ) echo " ✓ $LINE (written, verified)" R=$(api "/api/choudoufu/moves?plan=carve.json&receipt=1") jq_assert "$R" '[.receipt.moves[] | select(.address == "aws_iam_role.team_a")] | .[0].state == "moved"' "the receipt says moved" diff --git a/e2e/workbench-e2e.sh b/e2e/workbench-e2e.sh new file mode 100755 index 0000000..a6f7141 --- /dev/null +++ b/e2e/workbench-e2e.sh @@ -0,0 +1,237 @@ +#!/usr/bin/env bash +# behold#391 — the workbench catalog's acceptance run (M5 of #386). +# +# Every entry in `workbench.json` — the catalog this checkout carries and the +# npm package does not — loaded through `behold demo ` and +# asserted over HTTP, in catalog order, one behold per entry on its own port +# from 4720 up: +# +# - `missingRequirements` (src/demos.ts) decides first. A binary that is not +# on PATH, a sibling checkout nobody has, an optional peer behold does not +# install — each prints `skip: : needs …` and the run moves on. CI +# has no Docker, no choudoufu, no siblings, so every entry skips there and +# the script exits 0, the way e2e/choudoufu-estate-floci-e2e.sh does. +# - what survives is served: wait for /healthz, then /api/graph must carry +# nodes (the count is printed), and a live entry's /api/overlay?env= +# must answer with its bound/unowned/neutral split. +# - `terralith-4-adopt` is the one entry asserted twice: stock terraform +# applied that estate, so every card serves UNOWNED. The script then runs +# the `choudoufu live-import` line the up script PRINTED, by hand, in the +# target — behold never runs that write (#372's boundary) — and the same +# overlay reads bound afterwards. +# - `waterpark` is served in place out of a working checkout, so +# `git status --porcelain` there must be as empty after as before (#384: +# nothing is written under a served Terraform estate). +# +# Scratch discipline (src/scratch.ts, and workbench/lib/common.sh's bash half): +# each entry's emulator is `behold-wb-` on the entry's own port, and it +# is torn down through the `scripts/down.sh` its own up script wrote into the +# target — never `docker rm` by pattern. The run refuses to start if a +# behold-wb-* container is already there, and fails if one is left behind. +# +# just e2e-workbench +# BEHOLD_E2E_PORT=4740 just e2e-workbench # a different port block +# BEHOLD_E2E_ONLY="terralith-1 waterpark" just e2e-workbench +set -euo pipefail +cd "$(dirname "$0")/.." + +PORT_BASE="${BEHOLD_E2E_PORT:-4720}" +ONLY="${BEHOLD_E2E_ONLY:-}" +# The longest entry is the terralith at scale 4 adopted from a stock apply: +# a terraform apply and a choudoufu init over 301 resources before behold +# answers at all. +WAIT="${BEHOLD_E2E_WAIT:-600}" + +# The one entry excluded BY NAME rather than by a missing requirement: its +# setup is ../fountain-ops's own `just up`, which boots a five-minute k3d +# cluster in that working copy and switches your kubectl context. Everything +# else here is scratch; that one is not, so it stays a by-hand run. +excluded_reason() { + case "$1" in + fountain-ops) echo "excluded by name — its setup is ../fountain-ops's own \`just up\`: a five-minute k3d cluster built in that working copy, and it switches your kubectl context. Run it by hand: just example name=\"fountain-ops\"" ;; + *) echo "" ;; + esac +} + +command -v jq >/dev/null 2>&1 || { echo "skip: jq is not on PATH (every assertion here reads JSON)"; exit 0; } +[ -x ./node_modules/.bin/tsx ] || { echo "run npm install first — the catalog is read through src/demos.ts, with tsx" >&2; exit 1; } +if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then + LEFT="$(docker ps -a --format '{{.Names}}' | grep '^behold-wb-' || true)" + if [ -n "$LEFT" ]; then + echo "workbench containers already exist — a previous run didn't tear down:" >&2 + echo "$LEFT" | sed 's/^/ /' >&2 + echo "run \`bash scripts/down.sh\` in each target, then re-run" >&2 + exit 1 + fi +fi + +echo "→ build behold" +npm run build --silent + +# The catalog, read through the same functions behold reads it with: the +# registry merge, `missingRequirements`, `demoLocalPath`, and — for an entry +# served in place — the two preconditions that are not a binary on PATH: an +# uninstalled chant example (an in-place entry is never installed, #390) and +# the Terraform lexicon behold declares as an optional peer and does not ship. +CATALOG="$(./node_modules/.bin/tsx -e "$( + cat <<'JS' +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { loadDemoRegistry, missingRequirements, demoLocalPath } from "./src/demos.ts"; +import { hasTerraformRoots, terraformReaderState } from "./src/terraform-member.ts"; + +const out = []; +for (const e of loadDemoRegistry(process.cwd())) { + if (e.catalog !== "workbench") continue; + const local = demoLocalPath(e) ?? ""; + const missing = missingRequirements(e); + let reason = missing.length ? `needs ${missing.join(", ")}` : ""; + if (!reason && e.inPlace && local) { + if (existsSync(join(local, "package.json")) && !existsSync(join(local, "node_modules"))) { + reason = `needs its own node_modules — an in-place entry is served exactly as it sits and behold installs nothing there (npm install in ${local})`; + } else if (hasTerraformRoots(local)) { + const refusal = terraformReaderState().refusal; + if (refusal) reason = `needs the Terraform lexicon behold does not install — ${refusal.remedy}`; + } + } + // The unit separator, not a tab: `read` in bash folds a run of IFS + // WHITESPACE into one delimiter, so an entry with no env would shift every + // field after it by one. + out.push([e.name, e.serve.env ?? "", e.inPlace ? "1" : "0", local, reason].join("\u001f")); +} +process.stdout.write(out.join("\n")); +JS +)")" + +[ -n "$CATALOG" ] || { echo "no workbench catalog in this checkout (workbench.json missing?)" >&2; exit 1; } + +PID=""; TARGET=""; INPLACE="0"; ENTRY="" +teardown() { # tear the entry currently up down, through its own down script + [ -n "$PID" ] && kill "$PID" 2>/dev/null || true + PID="" + # An in-place entry's "target" is somebody's checkout, and a checkout may + # well have a scripts/down.sh of its own — the workbench never wrote it and + # this run never runs it. + if [ "$INPLACE" = "0" ] && [ -n "$TARGET" ] && [ -f "$TARGET/scripts/down.sh" ]; then + bash "$TARGET/scripts/down.sh" || true + fi + TARGET="" +} +trap teardown EXIT + +api() { curl -sf "http://localhost:$1$2"; } +jq_assert() { # + if ! printf '%s' "$1" | jq -e "$2" >/dev/null; then echo "FAIL: $ENTRY: $3" >&2; printf '%s' "$1" | jq -c '.' | head -c 1200 >&2; echo >&2; exit 1; fi + echo " ✓ $3" +} +# The live half's three words for a card, from src/choudoufu-live.ts: bound +# (the plan matched an identity), UNOWNED (live, and this estate does not own +# it), neutral (looked, and could not answer). +counts() { printf '%s' "$1" | jq -r '[([.ir.nodes[]|select(.attrs._status=="good")]|length), ([.ir.nodes[]|select(.attrs.omission=="UNOWNED")]|length), ([.ir.nodes[]|select(.attrs._status=="neutral")]|length)] | @tsv'; } + +TMPROOT="$(mktemp -d "${TMPDIR:-/tmp}/behold-workbench-e2e.XXXXXX")" +RAN=0; SKIPPED=0; i=-1 +RUN_T0=$SECONDS + +while IFS=$'\x1f' read -r name env inplace local reason <&3; do + [ -n "$name" ] || continue + i=$((i + 1)) + port=$((PORT_BASE + i)) + if [ -n "$ONLY" ] && ! printf ' %s ' "$ONLY" | grep -q " $name "; then continue; fi + excluded="$(excluded_reason "$name")" + if [ -n "$excluded" ]; then reason="$excluded"; fi + if [ -n "$reason" ]; then + echo "skip: $name: $reason" + SKIPPED=$((SKIPPED + 1)) + continue + fi + + ENTRY="$name"; INPLACE="$inplace" + # An in-place entry is served out of a working checkout, which may well be + # dirty before we touch it. What must not change is what the serve leaves + # behind, so the check is before against after, not "is this tree clean". + before="" + if [ "$inplace" = "1" ] && git -C "$local" rev-parse --git-dir >/dev/null 2>&1; then + before="$(git -C "$local" status --porcelain)" + fi + t0=$SECONDS + TARGET="$TMPROOT/$name" + log="$TMPROOT/$name.log" + echo + echo "→ $name (:$port${env:+, --env $env})" + node ./bin/behold.js demo "$name" "$TARGET" --port "$port" >"$log" 2>&1 /dev/null 2>&1 && { ready=1; break; } + kill -0 "$PID" 2>/dev/null || break + sleep 1 + done + if [ -z "$ready" ]; then + echo "FAIL: $name: behold never answered on :$port" >&2 + tail -40 "$log" >&2 + exit 1 + fi + echo " up in $((SECONDS - t0))s" + + G="$(api "$port" "/api/graph")" + jq_assert "$G" '(.ir.nodes | length) > 0' "$(printf '%s nodes, %s edges in /api/graph' "$(printf '%s' "$G" | jq -r '.ir.nodes|length')" "$(printf '%s' "$G" | jq -r '.ir.edges|length')")" + + if [ -n "$env" ]; then + O="$(api "$port" "/api/overlay?env=$env")" + jq_assert "$O" '(.ir.nodes | length) > 0' "/api/overlay?env=$env answers" + IFS=$'\t' read -r bound unowned neutral <<<"$(counts "$O")" + echo " ✓ overlay: bound $bound, unowned $unowned, neutral $neutral" + fi + + # #391: the adopt entry, twice — unowned before the by-hand import, bound + # after it. The line run below is the one the up script printed into the + # log; behold never runs it, and neither does the entry's setup. + if [ "$name" = "terralith-4-adopt" ]; then + # UNOWNED is the word for "live, and this estate does not own it" — what a + # stock apply leaves behind, since nothing wears the marker tag. (The cards + # that do read bound before the import are bound by DERIVED identity, off + # the configuration alone; ownership is the thing the import changes.) + [ "$unowned" -gt 0 ] || { echo "FAIL: $name: nothing reads UNOWNED before the import — the estate was supposed to arrive unowned" >&2; exit 1; } + echo " ✓ before the import: $unowned cards UNOWNED" + endpoint="$(grep -o 'AWS_ENDPOINT_URL=http://127\.0\.0\.1:[0-9]*' "$log" | tail -1)" + line="$(grep -o '[^ ]* live-import -state=[^ ]* -estate=[^ ]* -approve' "$log" | tail -1)" + [ -n "$line" ] && [ -n "$endpoint" ] || { echo "FAIL: $name: the up script printed no live-import line to run" >&2; tail -30 "$log" >&2; exit 1; } + echo " → by hand, in the target: $line" + ( cd "$TARGET" && env "$endpoint" AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1 sh -c "$line" >"$log.import" 2>&1 ) || + { echo "FAIL: $name: live-import failed" >&2; tail -20 "$log.import" >&2; exit 1; } + O2="$(api "$port" "/api/overlay?env=$env")" + IFS=$'\t' read -r bound2 unowned2 neutral2 <<<"$(counts "$O2")" + echo " ✓ after the import: bound $bound2, unowned $unowned2, neutral $neutral2" + [ "$bound2" -gt "$bound" ] || { echo "FAIL: $name: the import bound nothing" >&2; exit 1; } + [ "$unowned2" -eq 0 ] || { echo "FAIL: $name: $unowned2 cards still read UNOWNED after the import" >&2; exit 1; } + jq_assert "$O2" '([.ir.nodes[]|select(.attrs._status=="good")]|length) == (.ir.nodes|length)' "every card is bound after the import — the whole estate, adopted" + fi + + # #384/#390: an in-place entry is served where it sits — a Terraform estate's + # scratch project lives outside the estate, and no in-place entry is ever npm + # installed into somebody's checkout. So the checkout reads the same after. + if [ "$inplace" = "1" ] && git -C "$local" rev-parse --git-dir >/dev/null 2>&1; then + after="$(git -C "$local" status --porcelain)" + if [ "$after" != "$before" ]; then + echo "FAIL: $name: $local changed under the serve — behold wrote into a checkout" >&2 + diff <(printf '%s\n' "$before") <(printf '%s\n' "$after") | sed 's/^/ /' >&2 || true + exit 1 + fi + echo " ✓ $local is untouched (git status --porcelain unchanged)" + fi + + teardown + echo " $name: $((SECONDS - t0))s" + RAN=$((RAN + 1)) +done 3<<<"$CATALOG" + +ENTRY="" +if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then + LEFT="$(docker ps -a --format '{{.Names}}' | grep '^behold-wb-' || true)" + [ -z "$LEFT" ] || { echo "FAIL: workbench containers left behind: $LEFT" >&2; exit 1; } + echo + echo "✓ no behold-wb-* container left" +fi +echo "workbench e2e: $RAN entries green, $SKIPPED skipped, $((SECONDS - RUN_T0))s" diff --git a/example-choudoufu-estate/scripts/choudoufu-up.sh b/example-choudoufu-estate/scripts/choudoufu-up.sh index de00b64..a5db7e0 100755 --- a/example-choudoufu-estate/scripts/choudoufu-up.sh +++ b/example-choudoufu-estate/scripts/choudoufu-up.sh @@ -27,8 +27,13 @@ if ! docker info >/dev/null 2>&1; then echo "choudoufu-estate demo: Docker is not running — start Docker and re-run." >&2 exit 1 fi -if ! command -v choudoufu >/dev/null 2>&1; then - echo "choudoufu-estate demo: choudoufu is not on PATH — https://github.com/INTENTIUS/choudoufu (0.16.0 or newer)." >&2 +# behold#388: the same binary behold itself spawns — the Homebrew release is +# below the 0.16.0 floor, so a build from main is named by CHOUDOUFU_BIN and +# every script that runs choudoufu has to honour it or the demo's `init` and +# behold's own reads would be two different tools. +CHOUDOUFU="${CHOUDOUFU_BIN:-choudoufu}" +if ! command -v "$CHOUDOUFU" >/dev/null 2>&1; then + echo "choudoufu-estate demo: ${CHOUDOUFU} is not on PATH — https://github.com/INTENTIUS/choudoufu (0.16.0 or newer), or set CHOUDOUFU_BIN." >&2 exit 1 fi @@ -48,10 +53,10 @@ export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1 for estate in monolith team-a team-b team-c; do echo "→ choudoufu init in ${estate} (provider schemas — what makes the rungs real)" - (cd "$estate" && choudoufu init -input=false -no-color >/dev/null) + (cd "$estate" && "$CHOUDOUFU" init -input=false -no-color >/dev/null) done echo "→ choudoufu apply in monolith (the terralith: 21 resources, three teams in one estate)" -(cd monolith && choudoufu apply -auto-approve -input=false -no-color | tail -1) +(cd monolith && "$CHOUDOUFU" apply -auto-approve -input=false -no-color | tail -1) echo "choudoufu-estate demo: up. behold serves it next; the plan in monolith/carve.json moves team-a's resources out." diff --git a/justfile b/justfile index 6eddb9c..e39b381 100644 --- a/justfile +++ b/justfile @@ -55,6 +55,15 @@ example-install: serve project="example" env="": npm run dev -- serve {{project}} {{ if env != "" { "--env " + env } else { "" } }} +# Serve one catalog entry from this checkout (#388) — the workbench estates in +# workbench.json as readily as the bundled demos. `behold demo --list` names +# them and says which are ready; an entry whose sibling checkout is missing +# says so instead of failing halfway. +# just example # terralith-4 +# just example name="waterpark" +example name="terralith-4": + npm run dev -- demo {{name}} + # End-to-end: install the example's chant, build behold, serve it, assert the API. # Auto-detects AWS creds — exercises /api/overlay when present, /api/graph when not. e2e: @@ -154,3 +163,21 @@ e2e-argo-estate: # just e2e-ci-github e2e-ci-github: bash e2e/ci-executor-github-e2e.sh + +# behold#391 (M5 of #386): the workbench catalog itself. Every entry in +# workbench.json — this checkout's, not the package's — loaded through +# `behold demo ` on its own port from 4720 up, asserted +# over /api/graph and, for a live entry, /api/overlay; the terralith adopted +# from a stock apply asserted before and after a `choudoufu live-import` the +# script runs BY HAND in the target, the way a person would. Each entry's +# scratch emulator is torn down through the scripts/down.sh its own up script +# wrote — never `docker rm` by pattern — and a left-behind behold-wb-* fails +# the run. An entry whose requirement is missing (a binary, a sibling +# checkout, the optional Terraform lexicon) prints its reason and is skipped, +# so in CI every entry skips and this exits 0. `fountain-ops` is skipped by +# name everywhere: its setup boots a five-minute k3d cluster in your working +# copy and switches your kubectl context. +# just e2e-workbench +# BEHOLD_E2E_ONLY="terralith-1 waterpark" just e2e-workbench +e2e-workbench: + bash e2e/workbench-e2e.sh diff --git a/package-lock.json b/package-lock.json index 2edf943..073934e 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,8 +9,8 @@ "version": "0.17.0", "dependencies": { "@hono/node-server": "^2.0.5", - "@intentius/chant": "^0.54.0", - "@intentius/chant-k8s-client": "^0.54.0", + "@intentius/chant": "^0.61.0", + "@intentius/chant-k8s-client": "^0.61.0", "@intentius/pinhole": "^0.3.8", "hono": "^4.12.34" }, @@ -24,6 +24,18 @@ "tsx": "^4.19.0", "typescript": "^7.0.2", "vitest": "^4.1.9" + }, + "peerDependencies": { + "@cdktf/hcl2json": "^0.21.0", + "@intentius/chant-lexicon-terraform": "^0.61.0" + }, + "peerDependenciesMeta": { + "@cdktf/hcl2json": { + "optional": true + }, + "@intentius/chant-lexicon-terraform": { + "optional": true + } } }, "node_modules/@dagrejs/dagre": { @@ -470,9 +482,9 @@ } }, "node_modules/@intentius/chant": { - "version": "0.54.0", - "resolved": "https://registry.npmjs.org/@intentius/chant/-/chant-0.54.0.tgz", - "integrity": "sha512-ETYVPvDWqJiEsaaaYK+1U2lGrZZqHPU6/bOH797oQNw5gd7ESkEoYZxlh0qO90FQGVdx1pQMvmlCWskWD4xIlA==", + "version": "0.61.0", + "resolved": "https://registry.npmjs.org/@intentius/chant/-/chant-0.61.0.tgz", + "integrity": "sha512-aIumdd8TO3y4uDHDTUJRUAaKi3fN3x8XCpf0CmT+5/LHvNwVmcePXMmx3Gzppm+F/4JVbhM0H0oT/LklFKYO5g==", "license": "Apache-2.0", "dependencies": { "@dagrejs/dagre": "^3.0.0", @@ -488,9 +500,9 @@ } }, "node_modules/@intentius/chant-k8s-client": { - "version": "0.54.0", - "resolved": "https://registry.npmjs.org/@intentius/chant-k8s-client/-/chant-k8s-client-0.54.0.tgz", - "integrity": "sha512-LzoPfOQ4tIy7QR3ohLi/OqRgoZU/NXZ5TU6WeXFVqzffSjEoZu9yonjkaALy30Dqqa77XGFMTcP/SMZkNSEdsQ==", + "version": "0.61.0", + "resolved": "https://registry.npmjs.org/@intentius/chant-k8s-client/-/chant-k8s-client-0.61.0.tgz", + "integrity": "sha512-MkMzBH/yQnpGXq6FmL2Hqh5BLJnDj9GqUoO7fTMEA7z5BJWyjUhQQfeI2W146gUAe9FexYi9snE1ixAnnp9iyA==", "license": "Apache-2.0", "dependencies": { "@kubernetes/client-node": "^2.0.0" diff --git a/package.json b/package.json index 9c4bded..976d8df 100644 --- a/package.json +++ b/package.json @@ -36,11 +36,23 @@ }, "dependencies": { "@hono/node-server": "^2.0.5", - "@intentius/chant": "^0.54.0", - "@intentius/chant-k8s-client": "^0.54.0", + "@intentius/chant": "^0.61.0", + "@intentius/chant-k8s-client": "^0.61.0", "@intentius/pinhole": "^0.3.8", "hono": "^4.12.34" }, + "peerDependencies": { + "@intentius/chant-lexicon-terraform": "^0.61.0", + "@cdktf/hcl2json": "^0.21.0" + }, + "peerDependenciesMeta": { + "@intentius/chant-lexicon-terraform": { + "optional": true + }, + "@cdktf/hcl2json": { + "optional": true + } + }, "devDependencies": { "@types/node": "^26.4.1", "esbuild": "^0.28.0", diff --git a/src/__fixtures__/terraform-estate/backends/backend.local.tf b/src/__fixtures__/terraform-estate/backends/backend.local.tf new file mode 100644 index 0000000..3c533e6 --- /dev/null +++ b/src/__fixtures__/terraform-estate/backends/backend.local.tf @@ -0,0 +1,5 @@ +terraform { + backend "local" { + path = "terraform.tfstate" + } +} diff --git a/src/__fixtures__/terraform-estate/baseline/iam_policy.boundary.tf b/src/__fixtures__/terraform-estate/baseline/iam_policy.boundary.tf new file mode 100644 index 0000000..b73c3ee --- /dev/null +++ b/src/__fixtures__/terraform-estate/baseline/iam_policy.boundary.tf @@ -0,0 +1,3 @@ +resource "aws_iam_policy" "boundary" { + name = var.boundary_name +} diff --git a/src/__fixtures__/terraform-estate/baseline/versions.tf b/src/__fixtures__/terraform-estate/baseline/versions.tf new file mode 100644 index 0000000..07008c0 --- /dev/null +++ b/src/__fixtures__/terraform-estate/baseline/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.9.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + } +} diff --git a/src/__fixtures__/terraform-estate/envs/dev/README.md b/src/__fixtures__/terraform-estate/envs/dev/README.md new file mode 100644 index 0000000..f5882f1 --- /dev/null +++ b/src/__fixtures__/terraform-estate/envs/dev/README.md @@ -0,0 +1,2 @@ +The same shapes as prod with no traffic. Empty until a lesson needs it — no +`.tf` here, which is why discovery neither draws it nor reports it. diff --git a/src/__fixtures__/terraform-estate/envs/prod/provider.tf b/src/__fixtures__/terraform-estate/envs/prod/provider.tf new file mode 100644 index 0000000..dc58d9a --- /dev/null +++ b/src/__fixtures__/terraform-estate/envs/prod/provider.tf @@ -0,0 +1,3 @@ +provider "aws" { + region = var.region +} diff --git a/src/__fixtures__/terraform-estate/envs/prod/s3_bucket.artifacts.tf b/src/__fixtures__/terraform-estate/envs/prod/s3_bucket.artifacts.tf new file mode 100644 index 0000000..07c0516 --- /dev/null +++ b/src/__fixtures__/terraform-estate/envs/prod/s3_bucket.artifacts.tf @@ -0,0 +1,9 @@ +resource "aws_s3_bucket" "artifacts" { + bucket = "${var.name}-artifacts" +} + +module "desk_operator" { + source = "../../modules/persona" + + name = "desk-operator" +} diff --git a/src/__fixtures__/terraform-estate/envs/prod/versions.tf b/src/__fixtures__/terraform-estate/envs/prod/versions.tf new file mode 100644 index 0000000..07008c0 --- /dev/null +++ b/src/__fixtures__/terraform-estate/envs/prod/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.9.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + } +} diff --git a/src/__fixtures__/terraform-estate/modules/persona/iam_role.this.tf b/src/__fixtures__/terraform-estate/modules/persona/iam_role.this.tf new file mode 100644 index 0000000..af3185a --- /dev/null +++ b/src/__fixtures__/terraform-estate/modules/persona/iam_role.this.tf @@ -0,0 +1,4 @@ +resource "aws_iam_role" "this" { + name = var.name + permissions_boundary = var.boundary_arn +} diff --git a/src/__fixtures__/terraform-estate/modules/persona/versions.tf b/src/__fixtures__/terraform-estate/modules/persona/versions.tf new file mode 100644 index 0000000..07008c0 --- /dev/null +++ b/src/__fixtures__/terraform-estate/modules/persona/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.9.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + } +} diff --git a/src/choudoufu-live.test.ts b/src/choudoufu-live.test.ts index a30e233..a91e527 100644 --- a/src/choudoufu-live.test.ts +++ b/src/choudoufu-live.test.ts @@ -242,9 +242,9 @@ describe("readChoudoufuLive — the three spawns (#370)", () => { expect(consistent.find((a) => a[0] === "live-ls")).toEqual(["live-ls", "-estate=tlmig-sample-monolith", "-json", "-consistent", "."]); }); - it("refuses a member with no live block: there is no estate tag to list under", async () => { - const noEstate = raw("choudoufu-live-check-estate-references.json"); // choudoufu's own fixture, no live block - await expect(readChoudoufuLive("/est/refs", {}, runner({ "live-check": { code: 0, stdout: noEstate, stderr: "" } }))).rejects.toThrow("declares no `live { estate = … }` block"); + it("refuses a member that names no estate: there is no estate tag to list under", async () => { + const noEstate = raw("choudoufu-live-check-estate-references.json"); // choudoufu's own fixture, no estate named + await expect(readChoudoufuLive("/est/refs", {}, runner({ "live-check": { code: 0, stdout: noEstate, stderr: "" } }))).rejects.toThrow("names no estate"); }); it("throws with the plan's own error when live-plan carries an error diagnostic, and with the refusal when a read is refused", async () => { diff --git a/src/choudoufu-live.ts b/src/choudoufu-live.ts index 8e17a12..e99c9bb 100644 --- a/src/choudoufu-live.ts +++ b/src/choudoufu-live.ts @@ -41,7 +41,7 @@ import { captureChoudoufu, ChoudoufuReadError, choudoufuVersion, - hasLiveBlock, + isChoudoufuEstate, liveCheckToIr, readLiveCheck, stripAnsiLines, @@ -376,7 +376,11 @@ export async function readChoudoufuLive(dir: string, opts: GraphOptions, run: Ru const estate = check.doc.estate; if (!estate) { throw new ChoudoufuReadError( - { error: `${dir} declares no \`live { estate = … }\` block, so there is no estate tag to list the account under.`, code: "choudoufu-live-check", remedy: "Add the live block (choudoufu's `live-check` says what else the configuration needs), then reload." }, + { + error: `${dir} names no estate, so there is no estate tag to list the account under.`, + code: "choudoufu-live-check", + remedy: "Declare it — `estate = \"…\"` in an `estate.chdf.hcl` sidecar, or a `live { estate = … }` block in a root *.tf (choudoufu's `live-check` says what else the configuration needs) — then reload.", + }, dir, ); } @@ -419,8 +423,8 @@ export const choudoufuVia: MemberVia = { /** The kind, as src/member-kind.ts registers it. */ export const choudoufuSpec: MemberKindSpec = { kind: "choudoufu", - probe: hasLiveBlock, - expects: "a `live { estate = … }` block in a root *.tf file", + probe: isChoudoufuEstate, + expects: "an `estate.chdf.hcl` sidecar or a `live { estate = … }` block in a root *.tf file", via: choudoufuVia, }; diff --git a/src/choudoufu-member.test.ts b/src/choudoufu-member.test.ts index 204d935..9230bc0 100644 --- a/src/choudoufu-member.test.ts +++ b/src/choudoufu-member.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect, afterAll } from "vitest"; -import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; @@ -8,11 +8,13 @@ import { RUNGS, SCHEMA_SOURCES, addChoudoufuReferenceEdges, + captureChoudoufu, + choudoufuBinary, choudoufuCardFields, choudoufuMeetsFloor, - + choudoufuVersion, dataSourceKind, - hasLiveBlock, + isChoudoufuEstate, isDevBuild, liveCheckToIr, parseChoudoufuVersion, @@ -22,7 +24,7 @@ import { type LiveCheckDocument, } from "./choudoufu-member.ts"; import { choudoufuSpec } from "./choudoufu-live.ts"; -import { choudoufuSpawnEnv, setChoudoufuSpawnEnv } from "./choudoufu-member.ts"; +import { choudoufuSpawnEnv, resetChoudoufuVersionCache, setChoudoufuSpawnEnv } from "./choudoufu-member.ts"; // Fixture provenance (#369). Every document below was printed by // `choudoufu live-check -json` from a choudoufu built from main at @@ -257,6 +259,32 @@ describe("the spawn environment seam (#372)", () => { setChoudoufuSpawnEnv({}); expect(choudoufuSpawnEnv(base).AWS_ENDPOINT_URL).toBe("https://real"); }); + + // #388: `choudoufuSpawnEnv`'s sibling. The Homebrew release is below the + // floor and the build that carries the floor's fields is one somebody left + // outside PATH, so CHOUDOUFU_BIN has to reach the argv — not just the + // helper. Spawned for real against a script that prints its own name. + it("CHOUDOUFU_BIN is what captureChoudoufu spawns, and what the version probe reads", async () => { + const dir = mkdtempSync(join(tmpdir(), "behold-choudoufu-bin-")); + const fake = join(dir, "choudoufu-from-main"); + writeFileSync(fake, `#!/bin/sh\necho "{\\"choudoufu_version\\":\\"v0.16.0-37-g7d2f1b0b9e\\",\\"terraform_version\\":\\"1.13.0-dev\\",\\"argv\\":\\"$0 $*\\"}"\n`); + chmodSync(fake, 0o755); + const before = process.env.CHOUDOUFU_BIN; + try { + process.env.CHOUDOUFU_BIN = fake; + expect(choudoufuBinary()).toBe(fake); + const run = await captureChoudoufu(["version", "-json"], dir); + expect(run.code).toBe(0); + expect(JSON.parse(run.stdout).argv).toBe(`${fake} version -json`); + resetChoudoufuVersionCache(); + expect(choudoufuVersion()).toMatchObject({ bin: fake, version: "v0.16.0-37-g7d2f1b0b9e", forkField: true }); + } finally { + if (before === undefined) delete process.env.CHOUDOUFU_BIN; + else process.env.CHOUDOUFU_BIN = before; + resetChoudoufuVersionCache(); + rmSync(dir, { recursive: true, force: true }); + } + }); }); describe("the probe and the floor (#369)", () => { @@ -272,13 +300,31 @@ describe("the probe and the floor (#369)", () => { return dir; }; - it("hasLiveBlock: a root *.tf with a live block, by regex, root files only", () => { - expect(hasLiveBlock(make({ "main.tf": 'terraform {\n live {\n estate = "x"\n }\n}\n' }))).toBe(true); - expect(hasLiveBlock(make({ "main.tf": 'resource "aws_vpc" "x" {}\n' }))).toBe(false); - expect(hasLiveBlock(make({ "sub/main.tf": "live {\n}\n" }))).toBe(false); - expect(hasLiveBlock(make({ "notes.txt": "live {" }))).toBe(false); - expect(hasLiveBlock("/nonexistent/dir")).toBe(false); - expect(choudoufuSpec.probe).toBe(hasLiveBlock); + it("isChoudoufuEstate: a root *.tf with a live block, by regex, root files only", () => { + expect(isChoudoufuEstate(make({ "main.tf": 'terraform {\n live {\n estate = "x"\n }\n}\n' }))).toBe(true); + expect(isChoudoufuEstate(make({ "main.tf": 'resource "aws_vpc" "x" {}\n' }))).toBe(false); + expect(isChoudoufuEstate(make({ "sub/main.tf": "live {\n}\n" }))).toBe(false); + expect(isChoudoufuEstate(make({ "notes.txt": "live {" }))).toBe(false); + expect(isChoudoufuEstate("/nonexistent/dir")).toBe(false); + expect(choudoufuSpec.probe).toBe(isChoudoufuEstate); + }); + + // #387: the sidecar is choudoufu's leading form, and every `tools/estate-gen` + // cohort writes one — a *.tf set with no `live` block anywhere in it. + it("isChoudoufuEstate: the estate.chdf.hcl sidecar, on its own, is enough", () => { + const cohort = { + "estate.chdf.hcl": 'estate = "s3-cohort"\n\nrecord_store "local" {\n path = ".tofu-records"\n}\n', + "s3.tf": 'resource "aws_s3_bucket" "one" {\n bucket = "one"\n}\n', + "versions.tf": 'terraform {\n required_version = ">= 1.5.0"\n}\n', + }; + expect(isChoudoufuEstate(make(cohort))).toBe(true); + // The sidecar is not read, only found — and only in the directory itself. + expect(isChoudoufuEstate(make({ "sub/estate.chdf.hcl": 'estate = "x"\n', "main.tf": 'resource "aws_vpc" "x" {}\n' }))).toBe(false); + // Neither form: still not an estate. + expect(isChoudoufuEstate(make({ "s3.tf": cohort["s3.tf"], "README.md": "an estate.chdf.hcl would go here" }))).toBe(false); + // Both forms at once is choudoufu's error to report, not the probe's. + expect(isChoudoufuEstate(make({ "estate.chdf.hcl": 'estate = "x"\n', "main.tf": "live {\n}\n" }))).toBe(true); + expect(choudoufuSpec.expects).toBe("an `estate.chdf.hcl` sidecar or a `live { estate = … }` block in a root *.tf file"); }); it("parseChoudoufuVersion: the field's presence is the floor's real check (choudoufu#968)", () => { diff --git a/src/choudoufu-member.ts b/src/choudoufu-member.ts index f446fd1..8585158 100644 --- a/src/choudoufu-member.ts +++ b/src/choudoufu-member.ts @@ -3,8 +3,10 @@ * * choudoufu is an OpenTofu fork whose ownership lives on the resource as two * AWS tags, `tofu-estate` and `tofu-address`; the state file is a cache that - * is allowed to be stale. Each `live { estate = "..." }` block is one estate, - * and one estate is one member box in behold's estate compose. + * is allowed to be stale. An estate is declared once per directory — in the + * `estate.chdf.hcl` sidecar, choudoufu's leading form, or in a `live { estate + * = "..." }` block in a root *.tf — and one estate is one member box in + * behold's estate compose. * * This module reads the DECLARED half: `choudoufu live-check -json`, which * makes no cloud call and reads no state. The document is the roster (every @@ -32,7 +34,7 @@ * header for the cycle that rule exists for); the spawn helper is its own. */ import { spawn, spawnSync } from "node:child_process"; -import { readdirSync, readFileSync } from "node:fs"; +import { existsSync, readdirSync, readFileSync } from "node:fs"; import { join } from "node:path"; import type { GraphIR } from "@intentius/chant"; import { stripAnsi } from "./ansi.ts"; @@ -104,7 +106,7 @@ export type LiveCheckParse = { ok: true; doc: LiveCheckDocument } | { ok: false; const refuse = (error: string, remedy: string): LiveCheckParse => ({ ok: false, refusal: { error, code: "choudoufu-live-check", remedy } }); -const INSTALL = `Install choudoufu ${CHOUDOUFU_FLOOR} or newer (https://github.com/INTENTIUS/choudoufu) and put it on PATH.`; +const INSTALL = `Install choudoufu ${CHOUDOUFU_FLOOR} or newer (https://github.com/INTENTIUS/choudoufu) and put it on PATH, or point CHOUDOUFU_BIN at a build from main.`; const INIT = (dir: string): string => `Run \`choudoufu init -input=false\` in ${dir} so the rungs come from the provider's schemas.`; const isRecord = (v: unknown): v is Record => typeof v === "object" && v !== null && !Array.isArray(v); @@ -184,10 +186,24 @@ function tfFiles(dir: string): string[] { const LIVE_BLOCK = /^\s*live\s*\{/m; -/** Does `dir` look like a choudoufu estate: a root `*.tf` with a `live { … }` - * block. A regex, not a parse — behold reads no HCL — and over root files +/** choudoufu's leading estate declaration (#387): a sidecar beside the root + * `*.tf` files carrying `estate = "…"` and the record store. choudoufu's own + * reference calls it the leading form; `tools/estate-gen` writes one into + * every cohort and a migrated terralith carries one, which is why a probe + * that knew only the `live` block missed the estates behold is developed + * against. */ +export const ESTATE_SIDECAR = "estate.chdf.hcl"; + +/** Does `dir` look like a choudoufu estate: the `estate.chdf.hcl` sidecar in + * the directory, or — the other spelling, still supported — a root `*.tf` + * with a `live { … }` block. Either is enough; both at once is an error + * choudoufu itself reports, and not this probe's business. + * + * The sidecar half is the file's presence, nothing read. The `live` half is + * unchanged: a regex, not a parse — behold reads no HCL — over root files * only, because that is where choudoufu requires the block to be. */ -export function hasLiveBlock(dir: string): boolean { +export function isChoudoufuEstate(dir: string): boolean { + if (existsSync(join(dir, ESTATE_SIDECAR))) return true; for (const f of tfFiles(dir)) { try { if (LIVE_BLOCK.test(readFileSync(join(dir, f), "utf8"))) return true; @@ -340,6 +356,22 @@ export function choudoufuCardFields(node: { attrs: Record }): A // The binary. // --------------------------------------------------------------------------- +/** + * The choudoufu behold spawns (#388, decision 4 of #386): `CHOUDOUFU_BIN` when + * it names one, else `choudoufu` from PATH. One helper, so the version probe, + * the doctor line, every `-json` read and the demo requirement check all mean + * the same binary. + * + * It exists because the Homebrew release is 0.15.0, below behold's floor, and + * the build that carries the floor's fields is one somebody built from main + * and left outside PATH. Read on every call, not captured: a served-project + * switch or a test may change it under a running process. Workbench scripts + * spell the same fallback, `${CHOUDOUFU_BIN:-choudoufu}`. + */ +export function choudoufuBinary(): string { + return process.env.CHOUDOUFU_BIN || "choudoufu"; +} + /** What `choudoufu version -json` said. `forkField` is whether the document * carried `choudoufu_version` at all — absent means v0.15.0 or older * (choudoufu#968 designed the key to be always present, empty on a dev build, @@ -354,7 +386,7 @@ export interface ChoudoufuVersion { } /** Parse `version -json`'s stdout. Exported for testing. */ -export function parseChoudoufuVersion(stdout: string, bin = "choudoufu"): ChoudoufuVersion | undefined { +export function parseChoudoufuVersion(stdout: string, bin: string = choudoufuBinary()): ChoudoufuVersion | undefined { let json: unknown; try { json = JSON.parse(stdout); @@ -388,10 +420,10 @@ export function choudoufuMeetsFloor(v: ChoudoufuVersion): boolean { const versionCache = new Map(); -/** The choudoufu on PATH, once per process: undefined when there is none. - * Sync because it is the version half of `memberIr`'s cache key, which is - * computed on every read. */ -export function choudoufuVersion(bin = "choudoufu"): ChoudoufuVersion | undefined { +/** The choudoufu `choudoufuBinary()` names, once per binary per process: + * undefined when there is none. Sync because it is the version half of + * `memberIr`'s cache key, which is computed on every read. */ +export function choudoufuVersion(bin: string = choudoufuBinary()): ChoudoufuVersion | undefined { if (versionCache.has(bin)) return versionCache.get(bin); const run = spawnSync(bin, ["version", "-json"], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }); const v = run.error || run.status !== 0 ? undefined : parseChoudoufuVersion(run.stdout, bin); @@ -424,7 +456,7 @@ export function choudoufuSpawnEnv(base: NodeJS.ProcessEnv = process.env): NodeJS * once at close, as `runChantRaw` does — coercing per chunk corrupts a * multi-byte character straddling the 64KB highWaterMark. Never rejects: a * missing binary is code 127, a failing exit is data. */ -export function captureChoudoufu(args: string[], cwd: string, bin = "choudoufu"): Promise { +export function captureChoudoufu(args: string[], cwd: string, bin: string = choudoufuBinary()): Promise { return new Promise((resolvePromise) => { const out: Buffer[] = []; const err: Buffer[] = []; diff --git a/src/choudoufu-moves.test.ts b/src/choudoufu-moves.test.ts index ea1bf8a..d57b2c6 100644 --- a/src/choudoufu-moves.test.ts +++ b/src/choudoufu-moves.test.ts @@ -144,7 +144,7 @@ describe("the handoff (#371)", () => { stdout: cwd.endsWith("mono") ? raw("choudoufu-live-check-monolith.json") : raw("choudoufu-live-check-estate-references.json"), }); const got = await moveMembers([{ name: "mono", dir: "/est/mono" }, { name: "refs", dir: "/est/refs" }], run); - // The second declares no live block — not a destination, not an error. + // The second names no estate — not a destination, not an error. expect(got).toEqual([{ name: "mono", dir: "/est/mono", estate: "tlmig-sample-monolith" }]); }); }); diff --git a/src/cli.ts b/src/cli.ts index e271ae4..833d836 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -14,6 +14,7 @@ import { runExport } from "./export.ts"; import { diagnose, formatReport } from "./doctor.ts"; import { isAutoSyncMode, type AutoSyncMode } from "./autosync.ts"; import { detectProjectShape } from "./project.ts"; +import { servesAsEstate } from "./member-kind.ts"; import { setChoudoufuSpawnEnv } from "./choudoufu-member.ts"; import { readCarveReport } from "./carve-lens.ts"; import { @@ -57,7 +58,9 @@ Usage: demo The five-minute path from npm — no chant project needed. A catalog of demo estates (behold demo --list): bundled ones copy out of the package into a directory that's yours to edit; git ones shallow- - clone a public estate. Bare \`behold demo\` is the AWS example — an + clone a public estate; and in a checkout, the workbench block lists + whatever workbench.json names on this machine (#388). Bare + \`behold demo\` is the AWS example — an S3 bucket + policy on a local emulator: blue = declared, click Deploy, watch it turn green. \`behold demo k8s\` stands a workload up on a throwaway k3d cluster instead. \`behold demo carve\` is the @@ -228,7 +231,11 @@ export async function run(argv: string[]): Promise { for (const d of dirs) warnIfNotChantProject(d); await startServer({ projectDir: dirs[0], // primary — ops/overlay/rollback act on it - ...(dirs.length > 1 ? { projectDirs: dirs } : {}), + // #389: more than one directory composes, and so does one that is a member + // of a kind chant cannot read — a lone choudoufu estate has no chant to + // shell, so it is served as a one-member estate rather than through the + // single-project read that would answer "no lexicon detected". + ...(servesAsEstate(dirs) ? { projectDirs: dirs } : {}), port, ...(env ? { env } : {}), ...(pollSecs !== undefined ? { pollSecs } : {}), @@ -249,6 +256,10 @@ export async function run(argv: string[]): Promise { function warnIfNotChantProject(dir: string): void { const shape = detectProjectShape(dir); if (shape.kind === "project") return; + // #387: the one-member "the directory is itself a member" shape is servable + // as it stands (a choudoufu estate with its sidecar), so there is nothing to + // warn about — the member list would only name the directory again. + if (shape.kind === "estate" && shape.membersFrom === "itself") return; if (shape.kind === "estate") { process.stderr.write( `behold: warning — ${dir} is an estate root, not a chant project itself.\n` + @@ -345,9 +356,10 @@ async function runDoctor(rest: string[]): Promise { * watching the graph react is part of the demo), install its deps, and serve * it. #209 grew this into a CATALOG (demos.json, shipped in the package): * `--list` prints it with per-demo requirement checks; `demo ` loads a - * bundled (tarball copy) or git (shallow clone) entry. Idempotent: an - * existing target is reused (and an already-installed one skips npm - * install), so a second `behold demo` is just "start the demo again". */ + * bundled (tarball copy), git (shallow clone) or — in a checkout — local + * (#388: copied, served in place, or rendered by its own setup) entry. + * Idempotent: an existing target is reused (and an already-installed one skips + * npm install), so a second `behold demo` is just "start the demo again". */ async function runDemo(rest: string[]): Promise { const pkgRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); const registry = loadDemoRegistry(pkgRoot); @@ -364,11 +376,23 @@ async function runDemo(rest: string[]): Promise { process.stdout.write("behold demo: no catalog in this install (demos.json missing)\n"); return; } - for (const e of registry) { - const missing = missingRequirements(e); - const ready = missing.length ? `needs ${missing.join(", ")}` : "ready"; - process.stdout.write(` ${e.name.padEnd(14)} ${ready.padEnd(20)} ${e.description}\n`); - } + // #388: two catalogs, two blocks. The bundled ones ship in the tarball + // and are the same everywhere; the workbench ones are this checkout's, + // and an entry whose sibling is not checked out says so in the same + // place a missing binary does. + const block = (heading: string, entries: typeof registry): void => { + if (!entries.length) return; + process.stdout.write(`${heading}\n`); + for (const e of entries) { + const missing = missingRequirements(e); + const ready = missing.length ? `needs ${missing.join(", ")}` : "ready"; + process.stdout.write(` ${e.name.padEnd(14)} ${ready.padEnd(20)} ${e.description}\n`); + } + }; + block("bundled", registry.filter((e) => (e.catalog ?? "demos") === "demos")); + const workbench = registry.filter((e) => e.catalog === "workbench"); + if (workbench.length) process.stdout.write("\n"); + block("workbench (this checkout)", workbench); process.stdout.write("\nRun one: behold demo (bare `behold demo` = writes)\n"); return; } else if (a === "-h" || a === "--help") return void process.stdout.write(USAGE); diff --git a/src/demos-route.test.ts b/src/demos-route.test.ts index 64aa361..7016ad4 100644 --- a/src/demos-route.test.ts +++ b/src/demos-route.test.ts @@ -153,6 +153,42 @@ describe("GET /api/demos — the catalog, with per-entry satisfiability (#268)", // The flag is absent on everything else, not false. expect(demos.filter((d) => d.switchable === false).map((d) => d.name)).toEqual(["carve"]); }); + + // #388: the workbench catalog reaches the panel through the same route, and + // each row says which catalog listed it so the panel can group the two. + it("carries `catalog` per entry, and a workbench local entry's satisfiability", async () => { + const catalogDir = mkdtempSync(join(tmpdir(), "behold-workbench-route-")); + writeFileSync( + join(catalogDir, "workbench.json"), + JSON.stringify({ + demos: [ + { name: "wb-ready", description: "an estate in a sibling checkout", source: "local", path: ".", requires: [], serve: {} }, + { name: "wb-needs", description: "wants a tool this machine lacks", source: "local", path: ".", requires: ["terraform"], serve: {} }, + { name: "wb-here", description: "served where it sits", source: "local", path: ".", inPlace: true, requires: [], serve: {} }, + ], + }), + ); + process.env.BEHOLD_WORKBENCH = join(catalogDir, "workbench.json"); + try { + const { demos } = (await (await makeApp().request("/api/demos")).json()) as { demos: Array }; + expect(demos.find((d) => d.name === "writes")!.catalog).toBe("demos"); + const ready = demos.find((d) => d.name === "wb-ready")!; + expect(ready.catalog).toBe("workbench"); + expect(ready.satisfiable).toBe(true); + expect(ready.fetches).toBe(false); // a directory on this machine is not a fetch + expect(ready.target).toBe(join(sandbox, "behold-demos", "wb-ready")); + const needs = demos.find((d) => d.name === "wb-needs")!; + expect(needs.satisfiable).toBe(false); + expect(needs.reason).toMatch(/needs terraform on PATH/); + // An in-place entry IS its path, so it reads as already loaded. + const here = demos.find((d) => d.name === "wb-here")!; + expect(here.target).toBe(catalogDir); + expect(here.loaded).toBe(true); + } finally { + delete process.env.BEHOLD_WORKBENCH; + rmSync(catalogDir, { recursive: true, force: true }); + } + }); }); describe("POST /api/demos/open — a catalog name, never a path (#268)", () => { diff --git a/src/demos.test.ts b/src/demos.test.ts index e100ee0..db0c11c 100644 --- a/src/demos.test.ts +++ b/src/demos.test.ts @@ -6,7 +6,8 @@ import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync import { tmpdir } from "node:os"; import { join, dirname } from "node:path"; import { fileURLToPath } from "node:url"; -import { loadDemoRegistry, missingRequirements, fetchesFromNetwork, demoTargetDir, loadDemo, type DemoEntry } from "./demos.ts"; +import { loadDemoRegistry, missingRequirements, fetchesFromNetwork, demoTargetDir, demoLocalPath, loadDemo, type DemoEntry } from "./demos.ts"; +import { choudoufuBinary } from "./choudoufu-member.ts"; const REPO = join(dirname(fileURLToPath(import.meta.url)), ".."); @@ -26,6 +27,24 @@ describe("demos.json — the committed catalog (#209)", () => { expect(files, `${e.name}: ${e.dir} not shipped`).toContain(e.dir!); } expect(files).toContain("demos.json"); + // #388: the workbench catalog is a checkout's, not the tarball's — an + // entry naming `../choudoufu` means nothing inside an npm install. + expect(files).not.toContain("workbench.json"); + }); + + // #388: the committed workbench.json has to load — a typo there is a + // catalog nobody in this repo can run — and its entries carry the catalog + // they came from, which is what the two `--list` blocks and the panel group on. + it("reads workbench.json beside demos.json, stamping every entry with its catalog", () => { + expect(existsSync(join(REPO, "workbench.json"))).toBe(true); + for (const e of registry) expect(e.catalog === "demos" || e.catalog === "workbench", e.name).toBe(true); + expect(registry.filter((e) => e.catalog === "demos").map((e) => e.name)).toContain("writes"); + // Every workbench entry (M3 seeds them) is a local one, and names its + // path relative to the file that listed it. + for (const e of registry.filter((e) => e.catalog === "workbench")) { + expect(e.source, e.name).toBe("local"); + expect(e.catalogDir, e.name).toBe(REPO); + } }); // #254: the walkthrough is a catalog entry like any other, and every path it @@ -103,6 +122,68 @@ describe("loadDemoRegistry — malformed input degrades, never throws", () => { for (const d of shipped.serve.dirs!) expect(existsSync(join(REPO, shipped.dir!, d, "main.tf")), d).toBe(true); }); + // #388: the local source. A path, a path served in place, or no path at all + // and a setup that renders one — anything else names nothing to serve. + it("validates the local source: path, no path + setup, inPlace; and drops what names nothing", () => { + const local = (over: Record) => ({ description: "d", source: "local", requires: [], serve: {}, ...over }); + const dir = tmpRoot( + JSON.stringify({ + demos: [ + local({ name: "path", path: "../choudoufu" }), + local({ name: "generator", setup: "bash render.sh" }), + local({ name: "in-place", path: "../chant", inPlace: true, description: "served where it sits; its setup writes nothing the repo does not gitignore" }), + local({ name: "nothing" }), + local({ name: "in-place-no-path", inPlace: true, setup: "x" }), + local({ name: "path-not-string", path: 4 }), + local({ name: "in-place-not-bool", path: "../x", inPlace: "yes" }), + ], + }), + ); + expect(loadDemoRegistry(dir).map((e) => e.name)).toEqual(["path", "generator", "in-place"]); + // A relative path resolves against the CATALOG file's directory, never the + // cwd behold started in — the workbench names its siblings. + const entry = loadDemoRegistry(dir).find((e) => e.name === "path")!; + expect(demoLocalPath(entry)).toBe(join(dir, "..", "choudoufu")); + expect(demoLocalPath({ ...entry, path: "/opt/estates/net" })).toBe("/opt/estates/net"); + expect(demoLocalPath(loadDemoRegistry(dir).find((e) => e.name === "generator")!)).toBeUndefined(); + rmSync(dir, { recursive: true, force: true }); + }); + + // #388: two files, one registry. A workbench name that collides with a + // bundled one is dropped — `behold demo writes` means the same demo in every + // checkout, and a silent override is how it would stop meaning it. + it("merges workbench.json and BEHOLD_WORKBENCH, dropping collisions", () => { + const dir = tmpRoot(JSON.stringify({ demos: [{ name: "writes", description: "the bundled one", source: "bundled", dir: "x", requires: [], serve: {} }] })); + writeFileSync( + join(dir, "workbench.json"), + JSON.stringify({ + demos: [ + { name: "writes", description: "an impostor", source: "local", path: "../elsewhere", requires: [], serve: {} }, + { name: "terralith-4", description: "205 resources", source: "local", setup: "render.sh", requires: [], serve: {} }, + ], + }), + ); + const elsewhere = mkdtempSync(join(tmpdir(), "behold-workbench-")); + writeFileSync(join(elsewhere, "extra.json"), JSON.stringify({ demos: [{ name: "waterpark", description: "the access roots", source: "local", path: "../waterpark", requires: [], serve: {} }] })); + + const before = loadDemoRegistry(dir); + expect(before.map((e) => e.name)).toEqual(["writes", "terralith-4"]); + expect(before.map((e) => e.catalog)).toEqual(["demos", "workbench"]); + expect(before[0]!.description).toBe("the bundled one"); // the bundled entry wins the collision + + process.env.BEHOLD_WORKBENCH = join(elsewhere, "extra.json"); + try { + const after = loadDemoRegistry(dir); + expect(after.map((e) => e.name)).toEqual(["writes", "terralith-4", "waterpark"]); + // The named file's own directory is what ITS entries resolve against. + expect(demoLocalPath(after.find((e) => e.name === "waterpark")!)).toBe(join(elsewhere, "..", "waterpark")); + } finally { + delete process.env.BEHOLD_WORKBENCH; + } + rmSync(dir, { recursive: true, force: true }); + rmSync(elsewhere, { recursive: true, force: true }); + }); + it("a half-wired carve entry drops rather than serving a walkthrough that can't act (#254)", () => { const carve = (c: unknown) => ({ name: "c", description: "d", source: "bundled", dir: "x", requires: [], serve: { carve: c } }); const good = { report: "r.json", from: "tf", state: "tf/s.tfstate", project: "app", out: "app/carveout" }; @@ -202,6 +283,77 @@ describe("loadDemo — the copy/install/setup the CLI and the route share (#268) expect(res.ok).toBe(false); expect(!res.ok && res.error).toContain("exit 3"); }); + + // #388, the local source. Three shapes, one loader. + const localEntry = (over: Partial): DemoEntry => ({ name: "wb", description: "d", source: "local", requires: [], serve: {}, ...over }); + + /** A checkout to point a local entry at: a sibling of the catalog file. */ + const sibling = (name: string): { catalogDir: string; path: string } => { + const catalogDir = scratch("behold-workbench-"); + const path = join(catalogDir, "..", name); + mkdirSync(join(path, "src"), { recursive: true }); + mkdirSync(join(path, "node_modules", "left-behind"), { recursive: true }); + writeFileSync(join(path, "chant.config.ts"), "export default {};"); + roots.push(path); + return { catalogDir, path }; + }; + + it("copies a local entry into the target by default — it's yours, edit it", async () => { + const { catalogDir, path } = sibling("choudoufu-copy"); + const target = join(scratch("behold-demos-target-"), "wb"); + const res = await loadDemo(localEntry({ path: `../choudoufu-copy`, catalogDir }), { pkgRoot: catalogDir, target }); + expect(res.ok && res.serveDirs).toEqual([target]); + expect(existsSync(join(target, "chant.config.ts"))).toBe(true); + expect(existsSync(join(target, "node_modules"))).toBe(false); // the bundled filter, unchanged + expect(readFileSync(join(path, "chant.config.ts"), "utf8")).toBe("export default {};"); // the checkout is untouched + }); + + it("serves an inPlace entry where it sits — no copy, and dirs resolve under the path", async () => { + const { catalogDir, path } = sibling("chant-in-place"); + mkdirSync(join(path, "examples", "one"), { recursive: true }); + const entry = localEntry({ path: "../chant-in-place", inPlace: true, catalogDir, serve: { dirs: ["examples/one"] } }); + const target = join(scratch("behold-demos-target-"), "wb"); + const res = await loadDemo(entry, { pkgRoot: catalogDir, target }); + expect(res.ok && res.serveDirs).toEqual([join(path, "examples", "one")]); + expect(existsSync(target)).toBe(false); // nothing copied, nothing created + // And `demoTargetDir` agrees: an in-place entry IS its path, so the panel + // reads it as already loaded rather than offering a copy nobody serves. + expect(demoTargetDir(entry)).toBe(join(path)); + }); + + it("#390: never runs npm install in an inPlace entry — a checkout is read as it sits", async () => { + const { catalogDir, path } = sibling("chant-uninstalled"); + rmSync(join(path, "node_modules"), { recursive: true, force: true }); // the sibling helper's leftover — this checkout is NOT installed + writeFileSync(join(path, "package.json"), JSON.stringify({ name: "x", dependencies: { "@intentius/no-such-package-ever": "1.0.0" } })); + const entry = localEntry({ path: "../chant-uninstalled", inPlace: true, catalogDir }); + const res = await loadDemo(entry, { pkgRoot: catalogDir, target: join(scratch("behold-demos-target-"), "wb") }); + expect(res).toEqual({ ok: true, serveDirs: [path] }); // an install of that dependency would have failed + expect(existsSync(join(path, "node_modules"))).toBe(false); + expect(existsSync(join(path, "package-lock.json"))).toBe(false); + }); + + it("makes a generator entry's target and runs the setup in it, with the workbench env", async () => { + const catalogDir = scratch("behold-workbench-"); + const target = join(scratch("behold-demos-target-"), "terralith-4"); + const res = await loadDemo( + localEntry({ + name: "terralith-4", + catalogDir, + setup: 'printf "%s\\n%s\\n" "$BEHOLD_WORKBENCH_DIR" "$BEHOLD_DEMO_NAME" > rendered.txt && echo "estate = \\"t4\\"" > estate.chdf.hcl', + }), + { pkgRoot: catalogDir, target }, + ); + expect(res.ok && res.serveDirs).toEqual([target]); + expect(readFileSync(join(target, "rendered.txt"), "utf8")).toBe(`${catalogDir}\nterralith-4\n`); + expect(existsSync(join(target, "estate.chdf.hcl"))).toBe(true); // the setup rendered the estate + }); + + it("refuses an inPlace entry whose checkout is not there, rather than serving an empty directory", async () => { + const catalogDir = scratch("behold-workbench-"); + const res = await loadDemo(localEntry({ path: "../not-checked-out", inPlace: true, catalogDir }), { pkgRoot: catalogDir, target: join(scratch("behold-demos-target-"), "wb") }); + expect(res.ok).toBe(false); + expect(!res.ok && res.error).toContain("../not-checked-out"); + }); }); describe("demoTargetDir", () => { @@ -240,4 +392,35 @@ describe("missingRequirements", () => { const entry = { name: "x", description: "", source: "git" as const, repo: "https://x/y", requires: [], serve: {} }; expect(missingRequirements(entry)).toEqual([]); // git is installed here }); + + // #388: a sibling nobody checked out reads exactly like a missing binary — + // the entry stays listed, disabled, saying what is absent, and CI (where no + // sibling is checked out) stays clean. + it("reports a local entry's unchecked-out path as missing, and says so in its own words", () => { + const catalogDir = mkdtempSync(join(tmpdir(), "behold-workbench-")); + const here = { name: "x", description: "", source: "local" as const, path: ".", requires: [], serve: {}, catalogDir }; + expect(missingRequirements(here)).toEqual([]); + expect(missingRequirements({ ...here, path: "../nope" })).toEqual(["../nope (not checked out)"]); + // A generator entry names no path, so there is nothing to be missing. + expect(missingRequirements({ ...here, path: undefined, setup: "render.sh" })).toEqual([]); + rmSync(catalogDir, { recursive: true, force: true }); + }); + + // #388, decision 4: the binary behold spawns is CHOUDOUFU_BIN's when it + // names one — the Homebrew release is below behold's floor, and the build + // that carries the floor's fields is one somebody left outside PATH. + it("CHOUDOUFU_BIN satisfies the choudoufu requirement, and is the binary the helper names", () => { + const entry = { name: "x", description: "", source: "bundled" as const, dir: "x", requires: ["choudoufu"], serve: {} }; + const before = process.env.CHOUDOUFU_BIN; + try { + delete process.env.CHOUDOUFU_BIN; + expect(choudoufuBinary()).toBe("choudoufu"); + process.env.CHOUDOUFU_BIN = process.execPath; // an existing file, which is all the check asks + expect(choudoufuBinary()).toBe(process.execPath); + expect(missingRequirements(entry)).toEqual([]); + } finally { + if (before === undefined) delete process.env.CHOUDOUFU_BIN; + else process.env.CHOUDOUFU_BIN = before; + } + }); }); diff --git a/src/demos.ts b/src/demos.ts index 791eeb6..7035fce 100644 --- a/src/demos.ts +++ b/src/demos.ts @@ -4,9 +4,20 @@ * bundled (copied out of the tarball) or git (shallow-cloned). The registry * is data so growing the catalog (fountain #210, flux-estate #211) is a JSON * entry, not CLI surgery. + * + * #388 adds the second catalog and the third source. `workbench.json`, read + * beside `demos.json` and NOT in the npm `files` list, is a checkout's own + * catalog: the internal estates behold is developed against, named by relative + * path from the file that lists them (`../choudoufu`, `../waterpark`), so the + * file is committed and reproducible and an entry nobody has checked out is + * unsatisfiable exactly as a missing binary is. `BEHOLD_WORKBENCH=` + * names a catalog somewhere else. A `local` entry serves a directory on this + * machine: copied to `behold-demos/` (the bundled semantics — it's + * yours, edit it), served where it is (`inPlace`), or rendered into an empty + * target by its own `setup` when it names no path at all. */ -import { readFileSync, existsSync, cpSync } from "node:fs"; -import { join, relative, resolve, sep } from "node:path"; +import { readFileSync, existsSync, cpSync, mkdirSync } from "node:fs"; +import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { spawn, spawnSync } from "node:child_process"; /** @@ -46,84 +57,170 @@ export interface DemoServe { spawnEnv?: Record; } +/** Which catalog file an entry came from (#388): `demos.json`, shipped in the + * package, or `workbench.json`, this checkout's own. */ +export type DemoCatalog = "demos" | "workbench"; + export interface DemoEntry { name: string; description: string; - source: "bundled" | "git"; + source: "bundled" | "git" | "local"; /** bundled: the directory inside the package to copy out. */ dir?: string; /** git: the public repo to shallow-clone. */ repo?: string; + /** local (#388): the directory on this machine, relative to the directory + * of the catalog file that named it (the intentius checkouts are siblings, + * so `../choudoufu` is what a workbench entry writes) or absolute. Optional: + * an entry with no `path` and a `setup` is a GENERATOR entry, whose setup + * renders the estate into the empty target — estate-gen, terralith-gen. */ + path?: string; + /** local (#388): serve `path` where it is, no copy. Default false, which is + * the bundled semantics: copy to `behold-demos/` first, because + * anything whose setup writes into the tree (`init`, `apply`, a rendered + * generator) is copied or generated into the target, never run in a + * checkout. An `inPlace` entry with a `setup` must say so in its + * description — the setup is running in somebody's working copy. */ + inPlace?: boolean; /** Binaries that must be on PATH before this demo can run. */ requires: string[]; /** Optional post-install shell command, run in the target (posix shell). */ setup?: string; serve: DemoServe; + /** Which catalog listed this entry (#388). Stamped by `loadDemoRegistry`; + * absent on an entry a test built by hand, which reads as `demos`. */ + catalog?: DemoCatalog; + /** The directory of the catalog file that listed it — what `path` resolves + * against, and what `setup` gets as `BEHOLD_WORKBENCH_DIR`. Stamped by + * `loadDemoRegistry`. */ + catalogDir?: string; } -/** Read + validate the registry. A malformed entry is dropped (a registry - * typo must never break `behold demo` wholesale); a missing/corrupt file - * reads as an empty catalog. */ -export function loadDemoRegistry(pkgRoot: string): DemoEntry[] { +/** Read + validate one catalog file, stamping every surviving entry with the + * catalog it came from and the directory it was read from. A malformed entry + * is dropped (a registry typo must never break `behold demo` wholesale); a + * missing/corrupt file reads as an empty catalog. */ +function readCatalog(file: string, catalog: DemoCatalog): DemoEntry[] { let raw: unknown; try { - raw = JSON.parse(readFileSync(join(pkgRoot, "demos.json"), "utf8")); + raw = JSON.parse(readFileSync(file, "utf8")); } catch { return []; } const list = (raw as { demos?: unknown })?.demos; if (!Array.isArray(list)) return []; - return list.filter((e): e is DemoEntry => { - const d = e as DemoEntry; - if (!d || typeof d.name !== "string" || !d.name || typeof d.description !== "string") return false; - if (d.source === "bundled") { - if (typeof d.dir !== "string" || !d.dir) return false; - } else if (d.source === "git") { - if (typeof d.repo !== "string" || !d.repo) return false; - } else { - return false; - } - if (!Array.isArray(d.requires) || d.requires.some((r) => typeof r !== "string")) return false; - if (!d.serve || typeof d.serve !== "object") return false; - if (d.serve.dirs !== undefined && (!Array.isArray(d.serve.dirs) || d.serve.dirs.some((x) => typeof x !== "string") || !d.serve.dirs.length)) - return false; - // #372: a spawn environment is a flat string map or nothing. - if ( - d.serve.spawnEnv !== undefined && - (typeof d.serve.spawnEnv !== "object" || d.serve.spawnEnv === null || Array.isArray(d.serve.spawnEnv) || Object.values(d.serve.spawnEnv).some((v) => typeof v !== "string")) - ) - return false; - // #254: a carve entry names four relative paths, and a missing one would - // mean a walkthrough whose Emit step has nowhere to write — drop the entry - // rather than serve a half-wired demo. Every path stays relative: it is - // joined onto the COPY, and an absolute one there would escape it. - if (d.serve.carve !== undefined) { - const c = d.serve.carve as Partial | null; - const rel = (v: unknown): boolean => typeof v === "string" && !!v && !v.startsWith("/") && !v.split("/").includes(".."); - if (!c || typeof c !== "object") return false; - if (!rel(c.report) || !rel(c.from) || !rel(c.project) || !rel(c.out)) return false; - if (c.state !== undefined && !rel(c.state)) return false; - // The emitted source imports the project's lexicon; Node resolves that - // from the file's own directory upward, so an output dir outside the - // project would never lint. - if (!`${c.out}/`.startsWith(`${c.project}/`)) return false; + const catalogDir = dirname(resolve(file)); + return list + .filter((e): e is DemoEntry => validEntry(e)) + .map((e) => ({ ...e, catalog, catalogDir })); +} + +function validEntry(e: unknown): e is DemoEntry { + const d = e as DemoEntry; + if (!d || typeof d.name !== "string" || !d.name || typeof d.description !== "string") return false; + if (d.source === "bundled") { + if (typeof d.dir !== "string" || !d.dir) return false; + } else if (d.source === "git") { + if (typeof d.repo !== "string" || !d.repo) return false; + } else if (d.source === "local") { + // #388: a path, or a setup that renders one into the empty target. An + // entry with neither names nothing to serve, and `inPlace` with no path + // has no place to serve in. + if (d.path !== undefined && (typeof d.path !== "string" || !d.path)) return false; + if (d.inPlace !== undefined && typeof d.inPlace !== "boolean") return false; + if (!d.path && !d.setup) return false; + if (d.inPlace && !d.path) return false; + } else { + return false; + } + if (!Array.isArray(d.requires) || d.requires.some((r) => typeof r !== "string")) return false; + if (!d.serve || typeof d.serve !== "object") return false; + if (d.serve.dirs !== undefined && (!Array.isArray(d.serve.dirs) || d.serve.dirs.some((x) => typeof x !== "string") || !d.serve.dirs.length)) return false; + // #372: a spawn environment is a flat string map or nothing. + if ( + d.serve.spawnEnv !== undefined && + (typeof d.serve.spawnEnv !== "object" || d.serve.spawnEnv === null || Array.isArray(d.serve.spawnEnv) || Object.values(d.serve.spawnEnv).some((v) => typeof v !== "string")) + ) + return false; + // #254: a carve entry names four relative paths, and a missing one would + // mean a walkthrough whose Emit step has nowhere to write — drop the entry + // rather than serve a half-wired demo. Every path stays relative: it is + // joined onto the COPY, and an absolute one there would escape it. + if (d.serve.carve !== undefined) { + const c = d.serve.carve as Partial | null; + const rel = (v: unknown): boolean => typeof v === "string" && !!v && !v.startsWith("/") && !v.split("/").includes(".."); + if (!c || typeof c !== "object") return false; + if (!rel(c.report) || !rel(c.from) || !rel(c.project) || !rel(c.out)) return false; + if (c.state !== undefined && !rel(c.state)) return false; + // The emitted source imports the project's lexicon; Node resolves that + // from the file's own directory upward, so an output dir outside the + // project would never lint. + if (!`${c.out}/`.startsWith(`${c.project}/`)) return false; + } + return true; +} + +/** + * The whole catalog: `demos.json`, then `workbench.json` beside it, then the + * file `BEHOLD_WORKBENCH` names (#388). One registry, so `--list`, the panel + * and `demo ` all read the same thing. + * + * A workbench name that collides with one already in the catalog is dropped + * with a stderr line rather than shadowing it: `behold demo writes` must mean + * the same demo in every checkout, and a silent override is how it would stop + * meaning it. + */ +export function loadDemoRegistry(pkgRoot: string): DemoEntry[] { + const merged = readCatalog(join(pkgRoot, "demos.json"), "demos"); + const seen = new Set(merged.map((e) => e.name)); + const named = process.env.BEHOLD_WORKBENCH; + for (const e of [...readCatalog(join(pkgRoot, "workbench.json"), "workbench"), ...(named ? readCatalog(resolve(named), "workbench") : [])]) { + if (seen.has(e.name)) { + process.stderr.write(`behold: workbench entry "${e.name}" collides with a demo already in the catalog — dropped\n`); + continue; } - return true; - }); + seen.add(e.name); + merged.push(e); + } + return merged; } -/** Which of an entry's required binaries are NOT on PATH. `git` is an - * implicit requirement of every git-sourced entry. */ +/** A `local` entry's directory, absolute: `path` as given when it is + * absolute, else resolved against the catalog file's own directory — the + * workbench names its siblings (`../choudoufu`), and a path relative to + * whatever cwd behold was started in would mean a different estate per + * terminal. Undefined for every other source, and for a generator entry + * (no `path`), whose estate does not exist until its setup renders it. */ +export function demoLocalPath(entry: DemoEntry): string | undefined { + if (entry.source !== "local" || !entry.path) return undefined; + return isAbsolute(entry.path) ? entry.path : resolve(entry.catalogDir ?? process.cwd(), entry.path); +} + +/** Which of an entry's required binaries are NOT on PATH — plus, for a `local` + * entry (#388), a `path` nobody has checked out. `git` is an implicit + * requirement of every git-sourced entry. `choudoufu` is satisfied by a + * `CHOUDOUFU_BIN` that names an existing file, since the build that carries + * behold's floor is one somebody built from main, not one on PATH. */ export function missingRequirements(entry: DemoEntry): string[] { const bins = entry.source === "git" && !entry.requires.includes("git") ? [...entry.requires, "git"] : entry.requires; const finder = process.platform === "win32" ? "where" : "which"; - return bins.filter((bin) => spawnSync(finder, [bin], { stdio: "ignore" }).status !== 0); + const missing = bins.filter((bin) => { + if (bin === "choudoufu" && process.env.CHOUDOUFU_BIN && existsSync(process.env.CHOUDOUFU_BIN)) return false; + return spawnSync(finder, [bin], { stdio: "ignore" }).status !== 0; + }); + // An unchecked-out sibling reads exactly like a missing binary: the entry + // stays in the listing, disabled, saying what is absent, and CI stays clean. + const local = demoLocalPath(entry); + if (local && !existsSync(local)) missing.push(`${entry.path} (not checked out)`); + return missing; } /** Does loading this entry reach the network? A git entry is cloned, which is * the one demo step that leaves the machine — the CLI says so as it runs and * the panel's button (#268) says so BEFORE it runs. A bundled entry's copy is - * local; its `setup` may still pull images, which the description carries. */ + * local; its `setup` may still pull images, which the description carries. A + * `local` entry is a directory already on this machine (#388), so it never + * fetches either. */ export function fetchesFromNetwork(entry: DemoEntry): boolean { return entry.source === "git"; } @@ -132,8 +229,12 @@ export function fetchesFromNetwork(entry: DemoEntry): boolean { * the CLI and the panel route agree on "is this demo already loaded?". * `behold-demo` (the pre-catalog #193 default) is reused for the writes demo * when it exists, so an existing copy keeps working; everything else lands - * under `behold-demos/`. */ + * under `behold-demos/`. An `inPlace` entry (#388) has no target of its + * own: it IS its path, which is what makes "already loaded" true for it and + * keeps `behold-demos/` free of a copy nothing serves. */ export function demoTargetDir(entry: DemoEntry, cwd: string = process.cwd()): string { + const inPlace = entry.inPlace ? demoLocalPath(entry) : undefined; + if (inPlace) return inPlace; const legacy = resolve(cwd, "behold-demo"); if (entry.name === "writes" && existsSync(legacy)) return legacy; return resolve(cwd, "behold-demos", entry.name); @@ -163,23 +264,40 @@ export type DemoLoadResult = { ok: true; serveDirs: string[] } | { ok: false; er export async function loadDemo(entry: DemoEntry, opts: DemoLoadOptions): Promise { const { pkgRoot, target } = opts; const say = (line: string): void => opts.log?.(`behold demo ${entry.name} → ${line}`); - if (!existsSync(target)) { - if (entry.source === "bundled") { - const bundled = join(pkgRoot, entry.dir!); - if (!existsSync(bundled)) return { ok: false, error: `this install has no bundled ${entry.dir}` }; + const local = demoLocalPath(entry); + // #388: an in-place entry is served where it sits, so nothing below writes a + // copy and `work` is the checkout itself. Everything else works in `target`. + const inPlace = entry.source === "local" && entry.inPlace; + const work = inPlace ? local! : target; + if (inPlace) { + if (!existsSync(work)) return { ok: false, error: `${entry.path} is not checked out (looked in ${work})` }; + say(`serving ${work} where it is — nothing copied`); + } else if (!existsSync(target)) { + if (entry.source === "bundled" || (entry.source === "local" && local)) { + const from = entry.source === "bundled" ? join(pkgRoot, entry.dir!) : local!; + if (!existsSync(from)) return { ok: false, error: entry.source === "bundled" ? `this install has no bundled ${entry.dir}` : `${entry.path} is not checked out (looked in ${from})` }; say(`copying to ${target} (it's yours — edit it)`); // Skip only node_modules INSIDE the example. The filter must test the - // path relative to the bundled root: in an npm install the example + // path relative to the copied root: in an npm install the example // itself lives under node_modules/@intentius/behold/, so a bare // `src.includes("node_modules")` matched every file and copied nothing. try { - cpSync(bundled, target, { + cpSync(from, target, { recursive: true, - filter: (src) => !relative(bundled, src).split(sep).includes("node_modules"), + filter: (src) => !relative(from, src).split(sep).includes("node_modules"), }); } catch (err) { return { ok: false, error: `copy failed: ${err instanceof Error ? err.message : String(err)}` }; } + } else if (entry.source === "local") { + // #388's third decision: a generator IS a source. The entry names no + // path; its setup renders the estate into a target that starts empty. + say(`rendering into ${target} (empty — the setup writes the estate)`); + try { + mkdirSync(target, { recursive: true }); + } catch (err) { + return { ok: false, error: `could not make ${target}: ${err instanceof Error ? err.message : String(err)}` }; + } } else { say(`cloning ${entry.repo} to ${target}`); if ((await runStep("git", ["clone", "--depth", "1", entry.repo!, target])) !== 0) { @@ -189,29 +307,42 @@ export async function loadDemo(entry: DemoEntry, opts: DemoLoadOptions): Promise } else { say(`reusing ${target}`); } - if (existsSync(join(target, "package.json")) && !existsSync(join(target, "node_modules"))) { + // #390: an in-place entry is somebody's working copy, and `npm install` + // there would leave a node_modules and a package-lock.json in a checkout + // behold was only asked to read. It is served exactly as it sits; a project + // that is not installed says so on its own card, the way `behold serve` does. + if (!inPlace && existsSync(join(work, "package.json")) && !existsSync(join(work, "node_modules"))) { say("npm install…"); - if ((await runStep("npm", ["install"], { cwd: target, shell: process.platform === "win32" })) !== 0) { - return { ok: false, error: `npm install failed in ${target}` }; + if ((await runStep("npm", ["install"], { cwd: work, shell: process.platform === "win32" })) !== 0) { + return { ok: false, error: `npm install failed in ${work}` }; } } if (entry.setup) { say(entry.setup); - if ((await runStep(entry.setup, [], { cwd: target, shell: true })) !== 0) { + // #388: a workbench setup reaches its siblings through the catalog file's + // own directory (`$BEHOLD_WORKBENCH_DIR/../choudoufu`), never through the + // cwd behold happened to start in; the name is there so one script can + // serve several entries and name its scratch after the right one. + const env = { + ...process.env, + BEHOLD_DEMO_NAME: entry.name, + ...(entry.catalogDir ? { BEHOLD_WORKBENCH_DIR: entry.catalogDir } : {}), + }; + if ((await runStep(entry.setup, [], { cwd: work, shell: true, env })) !== 0) { return { ok: false, error: `setup failed (${entry.setup})` }; } } // #211: an estate demo serves several member projects composed; the first // listed is the primary, same as `behold serve a b c…`. - return { ok: true, serveDirs: entry.serve.dirs?.length ? entry.serve.dirs.map((d) => join(target, d)) : [target] }; + return { ok: true, serveDirs: entry.serve.dirs?.length ? entry.serve.dirs.map((d) => join(work, d)) : [work] }; } /** One child process, output inherited (git/npm/setup narrate themselves into * behold's own terminal). Resolves to the exit code; a spawn error is a * non-zero code, never a rejection. */ -function runStep(cmd: string, args: string[], opts: { cwd?: string; shell?: boolean } = {}): Promise { +function runStep(cmd: string, args: string[], opts: { cwd?: string; shell?: boolean; env?: NodeJS.ProcessEnv } = {}): Promise { return new Promise((res) => { - const child = spawn(cmd, args, { stdio: "inherit", cwd: opts.cwd, shell: opts.shell ?? false }); + const child = spawn(cmd, args, { stdio: "inherit", cwd: opts.cwd, shell: opts.shell ?? false, env: opts.env }); child.on("error", () => res(-1)); child.on("close", (code) => res(code ?? 1)); }); diff --git a/src/doctor.test.ts b/src/doctor.test.ts index f933ba7..ba16b79 100644 --- a/src/doctor.test.ts +++ b/src/doctor.test.ts @@ -5,6 +5,7 @@ import { join, dirname } from "node:path"; import { diagnose, formatReport, type DoctorProbes, type DoctorReport, type DoctorCheck } from "./doctor.ts"; import type { Kubeconfig } from "./k8s-target.ts"; import type { Substrate } from "./substrates.ts"; +import { discoverTerraformRoots as realRoots } from "./terraform-member.ts"; // Fixtures are built in the OS tmpdir rather than pointed at the bundled // examples: example-writes' node_modules is intentionally absent in a fresh @@ -40,7 +41,7 @@ function installed(name: string, version: string, bin?: string): Record = {}): Kubeconfig => ({ contexts: new Map(), @@ -69,7 +70,7 @@ describe("diagnose", () => { "src/main.ts": "", "ops/deploy.op.ts": `export const op = { name: "prod-apply", kind: ApplyOp };`, ...CHANT, - ...installed("@intentius/chant-lexicon-aws", "0.54.0"), + ...installed("@intentius/chant-lexicon-aws", "0.61.0"), }); const report = await diagnose(dir, probes({ detectSubstrates: async () => up })); @@ -77,8 +78,8 @@ describe("diagnose", () => { expect(report.ok).toBe(true); expect(report.kind).toBe("project"); expect(report.checks.map((c) => c.status)).toEqual(["pass", "pass", "pass", "pass", "pass", "pass", "pass"]); - expect(by(report, "chant").detail).toContain("chant 0.54.0"); - expect(by(report, "lexicons").detail).toContain("aws 0.54.0"); + expect(by(report, "chant").detail).toContain("chant 0.61.0"); + expect(by(report, "lexicons").detail).toContain("aws 0.61.0"); expect(by(report, "envs").detail).toContain("prod"); expect(by(report, "ops").detail).toContain("prod-apply (apply)"); expect(report.checks.every((c) => c.fix === undefined)).toBe(true); @@ -116,7 +117,7 @@ describe("diagnose", () => { const dir = fixture({ "chant.config.ts": `export default { lexicons: ["k8s"], k8s: { profiles: { local: { context: "k3d-demo" } } } };`, ...CHANT, - ...installed("@intentius/chant-lexicon-k8s", "0.54.0"), + ...installed("@intentius/chant-lexicon-k8s", "0.61.0"), }); const report = await diagnose(dir, probes()); @@ -135,7 +136,7 @@ describe("diagnose", () => { const dir = fixture({ "chant.config.ts": `export default { lexicons: ["k8s"], k8s: { profiles: { local: { context: "k3d-demo" } } } };`, ...CHANT, - ...installed("@intentius/chant-lexicon-k8s", "0.54.0"), + ...installed("@intentius/chant-lexicon-k8s", "0.61.0"), }); const kc = kubeconfig({ contexts: new Map([["k3d-demo", "k3d-demo-cluster"]]), @@ -156,7 +157,7 @@ describe("diagnose", () => { const dir = fixture({ "chant.config.ts": `export default { lexicons: ["k8s"], k8s: { profiles: { local: { context: "k3d-demo" } } } };`, ...CHANT, - ...installed("@intentius/chant-lexicon-k8s", "0.54.0"), + ...installed("@intentius/chant-lexicon-k8s", "0.61.0"), }); const kc = kubeconfig({ contexts: new Map([ @@ -182,7 +183,7 @@ describe("diagnose", () => { const dir = fixture({ "chant.config.ts": `export default { lexicons: ["aws"], environments: ["prod"] };`, ...CHANT, - ...installed("@intentius/chant-lexicon-aws", "0.54.0"), + ...installed("@intentius/chant-lexicon-aws", "0.61.0"), }); const down: Substrate[] = [ { name: "docker", label: "Docker", status: "up", detail: "daemon running" }, @@ -223,14 +224,14 @@ describe("diagnose", () => { "a/ops/apply.op.ts": `export const op = { name: "a-apply", kind: ApplyOp };`, "b/chant.config.ts": `export default { lexicons: ["aws"], environments: ["staging"] };`, ...CHANT, - ...installed("@intentius/chant-lexicon-aws", "0.54.0"), + ...installed("@intentius/chant-lexicon-aws", "0.61.0"), }); const report = await diagnose(dir, probes({ detectSubstrates: async () => up })); expect(report.kind).toBe("estate"); expect(by(report, "project").detail).toContain(".behold.json members): a (chant), b (chant)"); - expect(by(report, "chant").detail).toBe("a: chant 0.54.0, b: chant 0.54.0 (behold's floor 0.54.0)"); + expect(by(report, "chant").detail).toBe("a: chant 0.61.0, b: chant 0.61.0 (behold's floor 0.61.0)"); expect(by(report, "envs").detail).toContain(`behold serve ${dir}/a ${dir}/b --env prod`); expect(by(report, "ops").detail).toContain("a: a-apply (apply)"); }); @@ -255,7 +256,7 @@ describe("diagnose", () => { // `.behold.json` is said out loud before the estate serves without it. it("fails the project line on a declared member with an unknown kind, naming the kinds it reads", async () => { const dir = fixture({ - ".behold.json": JSON.stringify({ members: ["a", { dir: "b", kind: "terraform" }] }), + ".behold.json": JSON.stringify({ members: ["a", { dir: "b", kind: "pulumi" }] }), "a/chant.config.ts": `export default { lexicons: [], environments: ["local"] };`, "b/main.tf": "", ...CHANT, @@ -267,10 +268,10 @@ describe("diagnose", () => { expect(report.ok).toBe(false); const project = by(report, "project"); expect(project.status).toBe("fail"); - expect(project.detail).toContain("a (chant); invalid: b — unknown member kind \"terraform\""); + expect(project.detail).toContain("a (chant); invalid: b — unknown member kind \"pulumi\""); expect(project.fix).toContain("Member kinds this behold reads: chant, choudoufu"); // The chant line asks only the chant members. - expect(by(report, "chant").detail).toBe("a: chant 0.54.0 (behold's floor 0.54.0)"); + expect(by(report, "chant").detail).toBe("a: chant 0.61.0 (behold's floor 0.61.0)"); }); // #369: the choudoufu line, only on an estate with a choudoufu member. @@ -296,12 +297,20 @@ describe("diagnose", () => { it("passes when the binary meets the floor and every member has provider schemas", async () => { const report = await diagnose(estate(), probes({ choudoufu: { version: () => current, liveCheck: async () => document("provider") } })); expect(report.ok).toBe(true); - expect(by(report, "choudoufu")).toEqual({ name: "choudoufu", status: "pass", detail: "choudoufu v0.16.0 (on OpenTofu 1.13.0); net: schemas provider" }); + // #388: the line names the binary that answered — `choudoufu` from PATH + // or whatever CHOUDOUFU_BIN pointed at. + expect(by(report, "choudoufu")).toEqual({ name: "choudoufu", status: "pass", detail: "choudoufu v0.16.0 (on OpenTofu 1.13.0) at /usr/local/bin/choudoufu; net: schemas provider" }); }); - it("fails with no choudoufu on PATH, and with one too old to carry choudoufu_version", async () => { + it("fails with no choudoufu to spawn, and with one too old to carry choudoufu_version", async () => { const none = await diagnose(estate(), probes({ choudoufu: { version: () => undefined, liveCheck: async () => document("provider") } })); - expect(by(none, "choudoufu")).toMatchObject({ status: "fail", detail: expect.stringContaining("no choudoufu on PATH"), fix: expect.stringContaining("0.16.0") }); + // The detail names the binary that did not answer, and the fix names + // both ways to give it one (#388). + expect(by(none, "choudoufu")).toMatchObject({ + status: "fail", + detail: expect.stringContaining("no choudoufu: `choudoufu` does not answer"), + fix: expect.stringContaining("CHOUDOUFU_BIN"), + }); const old = await diagnose(estate(), probes({ choudoufu: { version: () => ({ bin: "/x/choudoufu", version: "", forkField: false }), liveCheck: async () => document("provider") } })); expect(by(old, "choudoufu")).toMatchObject({ status: "fail", detail: expect.stringContaining("v0.15.0 or older") }); expect(old.ok).toBe(false); @@ -322,6 +331,63 @@ describe("diagnose", () => { }); }); + // #384: the terraform line, on the estate that is a directory of `.tf` files + // and nothing else. The reader is two optional peers behold does not install, + // so its absence is a fail with the one install line. + describe("the terraform line", () => { + const estate = () => + fixture({ + "envs/prod/versions.tf": "terraform {\n required_providers {\n aws = {}\n }\n}\n", + "envs/prod/main.tf": 'resource "aws_s3_bucket" "artifacts" {}\n', + "modules/persona/versions.tf": "terraform {\n required_providers {\n aws = {}\n }\n}\n", + "modules/persona/main.tf": 'resource "aws_iam_role" "this" {}\n', + }); + const present = { + lexicon: { pkg: "@intentius/chant-lexicon-terraform", range: "^0.61.0", version: "0.61.0" }, + parser: { pkg: "@cdktf/hcl2json", range: "^0.21.0", version: "0.21.0" }, + from: "/opt/behold/dist", + }; + const absent = { + ...present, + lexicon: { pkg: "@intentius/chant-lexicon-terraform", range: "^0.61.0" }, + parser: { pkg: "@cdktf/hcl2json", range: "^0.21.0" }, + refusal: { + error: "Reading a Terraform estate needs chant's terraform lexicon, which behold does not install: … are not resolvable from /opt/behold/dist.", + code: "terraform-lexicon" as const, + remedy: "Install @intentius/chant-lexicon-terraform@^0.61.0 @cdktf/hcl2json@^0.21.0 beside behold, then reload.", + }, + }; + + it("is absent from a plain chant project", async () => { + const plain = fixture({ "chant.config.ts": `export default { lexicons: [], environments: ["local"] };`, ...CHANT }); + expect((await diagnose(plain, probes())).checks.some((c) => c.name === "terraform")).toBe(false); + }); + + it("passes with the reader's versions and what discovery found, skips included", async () => { + const report = await diagnose(estate(), probes({ terraform: { reader: () => present, roots: (d) => realRoots(d) } })); + expect(report.kind).toBe("estate"); + expect(by(report, "project").detail).toBe("a terraform member — the directory itself, no member list"); + expect(by(report, "terraform")).toEqual({ + name: "terraform", + status: "pass", + detail: "@intentius/chant-lexicon-terraform 0.61.0 (@cdktf/hcl2json 0.21.0); .: 1 root (prod), 1 skipped (modules/persona)", + }); + // The chant line asks the chant members, and there are none. + expect(by(report, "chant").status).toBe("pass"); + }); + + it("fails with the one install line when the reader is not there, and still says what it would draw", async () => { + const report = await diagnose(estate(), probes({ terraform: { reader: () => absent, roots: (d) => realRoots(d) } })); + expect(report.ok).toBe(false); + expect(by(report, "terraform")).toMatchObject({ + status: "fail", + detail: expect.stringContaining("behold does not install"), + fix: absent.refusal.remedy, + }); + expect(by(report, "terraform").detail).toContain("1 root (prod)"); + }); + }); + it("is none, and says why, when the only declared member fails its kind's probe", async () => { const dir = fixture({ ".behold.json": JSON.stringify({ members: [{ dir: "b", kind: "chant" }] }), "b/README.md": "" }); const report = await diagnose(dir, probes()); @@ -336,7 +402,7 @@ describe("--json", () => { "chant.config.ts": `export default { lexicons: ["aws"], environments: ["prod"] };`, "ops/apply.op.ts": `export const op = { name: "prod-apply", kind: ApplyOp };`, ...CHANT, - ...installed("@intentius/chant-lexicon-aws", "0.54.0"), + ...installed("@intentius/chant-lexicon-aws", "0.61.0"), }); const report = await diagnose(dir, probes({ detectSubstrates: async () => up })); diff --git a/src/doctor.ts b/src/doctor.ts index 0f970b6..0825cc8 100644 --- a/src/doctor.ts +++ b/src/doctor.ts @@ -35,8 +35,16 @@ import { type ChantResolution, } from "./chant.ts"; import { registeredMemberKinds } from "./member-kind.ts"; -import { CHOUDOUFU_FLOOR, choudoufuMeetsFloor, choudoufuVersion, readLiveCheck, type ChoudoufuVersion, type LiveCheckParse } from "./choudoufu-member.ts"; +import { CHOUDOUFU_FLOOR, choudoufuBinary, choudoufuMeetsFloor, choudoufuVersion, readLiveCheck, type ChoudoufuVersion, type LiveCheckParse } from "./choudoufu-member.ts"; import { detectProject, detectProjectShape, type ProjectKind } from "./project.ts"; +import { + HCL_PARSER_PKG, + TERRAFORM_LEXICON_PKG, + discoverTerraformRoots, + terraformReaderState, + type TerraformReaderState, + type TerraformRootScan, +} from "./terraform-member.ts"; import { loadKubeconfig, resolveK8sTarget, type K8sProfiles, type Kubeconfig } from "./k8s-target.ts"; import { detectSubstrates, type Substrate } from "./substrates.ts"; import { discoverEstateOps, type OpInfo } from "./ops.ts"; @@ -48,7 +56,7 @@ export type CheckStatus = "pass" | "warn" | "fail"; * on it); `detail` is what behold found; `fix` is the single next step, set * whenever the status isn't a pass. */ export interface DoctorCheck { - name: "project" | "chant" | "lexicons" | "envs" | "kube" | "substrates" | "ops" | "choudoufu"; + name: "project" | "chant" | "lexicons" | "envs" | "kube" | "substrates" | "ops" | "choudoufu" | "terraform"; status: CheckStatus; detail: string; fix?: string; @@ -77,6 +85,12 @@ export interface DoctorProbes { version: () => ChoudoufuVersion | undefined; liveCheck: (dir: string) => Promise; }; + /** The terraform reader's two optional peers and the root walk (#384), + * injectable so a test needs neither installed. */ + terraform?: { + reader: () => TerraformReaderState; + roots: (dir: string) => TerraformRootScan; + }; } const list = (xs: readonly string[]): string => xs.join(", "); @@ -276,7 +290,8 @@ function opsCheck(root: string, ops: OpInfo[], chantSource: ChantResolution["sou /** * The choudoufu line (#369), only on an estate with a choudoufu member: the - * binary on PATH and at the floor — checked by the FIELD `version -json` + * binary — `choudoufu` from PATH or the one `CHOUDOUFU_BIN` names (#388), and + * the line prints which answered — at the floor, checked by the FIELD `version -json` * carries, since the floor's fields landed on choudoufu main before a release * did — then each member's own `live-check -json`, offline, for whether its * rungs came from provider schemas or from choudoufu's built-in table. The @@ -290,8 +305,8 @@ async function choudoufuCheck(root: string, members: { dir: string; abs: string return { name: "choudoufu", status: "fail", - detail: `${members.length} choudoufu member${members.length === 1 ? "" : "s"} (${list(members.map((m) => m.dir))}), and no choudoufu on PATH`, - fix: `Install choudoufu ${CHOUDOUFU_FLOOR} or newer (https://github.com/INTENTIUS/choudoufu) and put it on PATH.`, + detail: `${members.length} choudoufu member${members.length === 1 ? "" : "s"} (${list(members.map((m) => m.dir))}), and no choudoufu: \`${choudoufuBinary()}\` does not answer \`version -json\``, + fix: `Install choudoufu ${CHOUDOUFU_FLOOR} or newer (https://github.com/INTENTIUS/choudoufu) and put it on PATH, or point CHOUDOUFU_BIN at a build from main.`, }; } if (!choudoufuMeetsFloor(v)) { @@ -305,7 +320,10 @@ async function choudoufuCheck(root: string, members: { dir: string; abs: string } const reads = await Promise.all(members.map(async (m) => ({ m, parsed: await probe.liveCheck(m.abs) }))); const failed = reads.filter((r) => !r.parsed.ok); - const which = `choudoufu ${v.version || "dev build"}${v.upstream ? ` (on OpenTofu ${v.upstream})` : ""}`; + // #388: which binary answered, always — `choudoufu` from PATH, or whatever + // CHOUDOUFU_BIN named, which is how a build from main is used before a + // release carries the floor's fields. + const which = `choudoufu ${v.version || "dev build"}${v.upstream ? ` (on OpenTofu ${v.upstream})` : ""} at ${v.bin}`; if (failed.length) { return { name: "choudoufu", @@ -327,6 +345,34 @@ async function choudoufuCheck(root: string, members: { dir: string; abs: string return { name: "choudoufu", status: "pass", detail: `${which}; ${perMember}` }; } +/** + * The terraform line (#384), only on an estate with a terraform member: is the + * reader here at all, and what did root discovery find. + * + * The reader is two optional peers behold deliberately does not install (see + * src/terraform-member.ts's header — an HCL parser in every user's install is + * the cost, and most users serve chant projects), so its absence is a fail with + * the one install line, exactly as a missing choudoufu binary is. The roots are + * a pass that says what will be drawn and what was skipped, because a root + * missing from the picture should be findable here rather than by counting + * boxes. + */ +function terraformCheck(members: { dir: string; abs: string }[], probe: NonNullable): DoctorCheck { + const state = probe.reader(); + const scans = members.map((m) => ({ m, scan: probe.roots(m.abs) })); + const perMember = list( + scans.map(({ m, scan }) => { + const skipped = scan.skipped.length ? `, ${scan.skipped.length} skipped (${list(scan.skipped.map((s) => s.dir))})` : ""; + return `${m.dir}: ${scan.roots.length} root${scan.roots.length === 1 ? "" : "s"} (${list(scan.roots.map((r) => r.name))})${skipped}`; + }), + ); + if (state.refusal) { + return { name: "terraform", status: "fail", detail: `${state.refusal.error} ${perMember}`, fix: state.refusal.remedy }; + } + const reader = `${TERRAFORM_LEXICON_PKG} ${state.lexicon.version || "dev"} (${HCL_PARSER_PKG} ${state.parser.version || "dev"})`; + return { name: "terraform", status: "pass", detail: `${reader}; ${perMember}` }; +} + /** * Diagnose a directory. Read-only; resolves every fact through the module the * server reads it from. A directory that is neither a chant project nor an @@ -379,15 +425,21 @@ export async function diagnose(dir: string, probes: DoctorProbes = {}): Promise< const memberList = list(members.map((m) => `${m.dir} (${m.kind})`)); const membersFrom = shape.membersFrom === "behold-config" ? ".behold.json members" : "npm workspaces"; + // #387: a directory that is itself a member was named by nothing, so the + // line says what it is rather than "estate of 1 members (npm workspaces)". + const estateDetail = + shape.membersFrom === "itself" + ? `a ${members[0]!.kind} member — the directory itself, no member list` + : `estate of ${members.length} members (${membersFrom}): ${memberList}`; const projectCheck: DoctorCheck = estate ? invalid.length ? { name: "project", status: "fail", - detail: `estate of ${members.length} members (${membersFrom}): ${memberList}; invalid: ${list(invalidDetail)}`, + detail: `${estateDetail}; invalid: ${list(invalidDetail)}`, fix: kindsFix, } - : { name: "project", status: "pass", detail: `estate of ${members.length} members (${membersFrom}): ${memberList}` } + : { name: "project", status: "pass", detail: estateDetail } : { name: "project", status: "pass", detail: `chant project (${relative(root, shape.configFile!)})` }; // One config read per chant target, shared by the lexicon/env/kube lines — @@ -413,8 +465,9 @@ export async function diagnose(dir: string, probes: DoctorProbes = {}): Promise< lexiconCheck(root, declared, estate), // An estate root is not itself servable — the hint has to name its members // (`behold serve a b c`, #31), which is what a stranger would otherwise - // discover by having the root serve nothing. - envCheck(envs, estate ? shape.members!.map((m) => `${dir.replace(/\/$/, "")}/${m.dir}`).join(" ") : dir), + // discover by having the root serve nothing. The `.` member (#387) is the + // directory, so the hint stays the directory. + envCheck(envs, estate ? shape.members!.map((m) => (m.dir === "." ? dir.replace(/\/$/, "") : `${dir.replace(/\/$/, "")}/${m.dir}`)).join(" ") : dir), kube, substrateCheck(substrates), opsCheck(root, discoverEstateOps(targets), resolveChant(primary).source, estate), @@ -423,6 +476,10 @@ export async function diagnose(dir: string, probes: DoctorProbes = {}): Promise< if (choudoufuMembers.length) { checks.push(await choudoufuCheck(root, choudoufuMembers, probes.choudoufu ?? { version: () => choudoufuVersion(), liveCheck: (d) => readLiveCheck(d) })); } + const terraformMembers = members.filter((m) => m.kind === "terraform"); + if (terraformMembers.length) { + checks.push(terraformCheck(terraformMembers, probes.terraform ?? { reader: () => terraformReaderState(), roots: (d) => discoverTerraformRoots(d) })); + } return { behold, dir: root, kind: shape.kind, ok: !checks.some((c) => c.status === "fail"), checks }; } diff --git a/src/estate.test.ts b/src/estate.test.ts index f468795..08f9825 100644 --- a/src/estate.test.ts +++ b/src/estate.test.ts @@ -26,6 +26,7 @@ import { } from "./estate.ts"; import { resetMemberIrCache } from "./member-ir.ts"; import { registerMemberKind } from "./member-kind.ts"; +import { hasTerraformRoots } from "./terraform-member.ts"; import { attachRuntimeContainment } from "./overlay.ts"; import type { GraphIR as ChantGraphIR } from "@intentius/chant"; @@ -622,6 +623,32 @@ describe("estate reads dispatch on member kind (#368)", () => { expect(subnet.attrs._status).toBe("neutral"); }); + // #384: the terraform kind is registered by its real probe, so this asserts + // the dispatch a bare Terraform directory actually takes — a directory of + // `.tf` files, no chant config, no `live` block — with the read stubbed + // where the lexicon would be. + it("reads a bare Terraform directory through the terraform kind, beside a chant member", async () => { + const root = mkdtempSync(join(tmpdir(), "behold-kinds-tf-")); + made.push(root); + mkdirSync(join(root, "app"), { recursive: true }); + writeFileSync(join(root, "app", "chant.config.ts"), "export default {};\n"); + mkdirSync(join(root, "access", "envs", "prod"), { recursive: true }); + writeFileSync(join(root, "access", "envs", "prod", "versions.tf"), "terraform {\n required_providers {\n aws = {}\n }\n}\n"); + writeFileSync(join(root, "access", "envs", "prod", "main.tf"), 'resource "aws_s3_bucket" "artifacts" {}\n'); + const tfRead = vi.fn(async (_dir: string) => stack("aws_s3_bucket.artifacts", "terraform")); + registerMemberKind({ kind: "terraform", probe: hasTerraformRoots, expects: "a Terraform root", via: { tool: () => "lexicon\0v1", read: tfRead as never } }); + vi.mocked(graphIr).mockImplementation((async () => stack("vpc")) as never); + + const chantDir = join(root, "app"); + const tfDir = join(root, "access"); + const ir = await composeEstate([chantDir, tfDir]); + + expect(estateMembers([chantDir, tfDir]).map((m) => m.kind)).toEqual(["chant", "terraform"]); + expect(tfRead.mock.calls.map(([dir]) => dir)).toEqual([tfDir]); + expect(vi.mocked(graphIr).mock.calls.map(([dir]) => dir)).toEqual([chantDir]); + expect(ir.nodes.map((n) => n.id).sort()).toEqual(["access/aws_s3_bucket.artifacts", "app/vpc"]); + }); + it("a directory no kind claims still reads as chant — the failure it always produced, from the same place", async () => { const root = mkdtempSync(join(tmpdir(), "behold-kinds-")); made.push(root); diff --git a/src/member-kind.ts b/src/member-kind.ts index d083cb9..1c5524d 100644 --- a/src/member-kind.ts +++ b/src/member-kind.ts @@ -36,10 +36,11 @@ import { choudoufuSpec } from "./choudoufu-live.ts"; import type { MemberVia } from "./member-ir.ts"; import { chantConfigPath } from "./project.ts"; +import { terraformSpec } from "./terraform-member.ts"; /** The vocabulary `.behold.json` may use. Closed on purpose: a kind is a * contract behold knows how to read, not a label a project invents. */ -export const MEMBER_KINDS = ["chant", "choudoufu"] as const; +export const MEMBER_KINDS = ["chant", "choudoufu", "terraform"] as const; export type MemberKind = (typeof MEMBER_KINDS)[number]; export function isMemberKind(s: unknown): s is MemberKind { @@ -87,6 +88,27 @@ export function memberKindOf(dir: string): MemberKind | undefined { return undefined; } +/** + * Does serving these directories go through the estate compose path (#389)? + * + * More than one always did. ONE does too when the directory is a member of a + * kind that is not chant: the single-project read is `chant graph `, and a + * lone choudoufu estate has no chant.config.ts and no lexicon, so that read + * answers "No lexicon detected in infrastructure files" and behold serves the + * no-project card over a perfectly good estate. src/project.ts already says + * `behold serve ` is a thing behold accepts; this is what + * makes it true, and the workbench's generated single-estate entries (#389 — + * every estate-gen cohort, every terralith) are what found it missing. + * + * A one-member estate composes exactly as a four-member one does, ids + * namespaced under the member's short name, so the graph, the pane and the + * morph agree on what a node is called. + */ +export function servesAsEstate(dirs: readonly string[]): boolean { + if (dirs.length > 1) return true; + return dirs.length === 1 && (memberKindOf(dirs[0]) ?? "chant") !== "chant"; +} + /** The chant member: what every member was until #368. No `via`: chant is * the reader the estate falls back to, for this kind and for a directory no * kind claims. */ @@ -100,3 +122,10 @@ registerMemberKind({ * two AWS tags, read through `live-check -json`. After chant, so a directory * that is both is a chant member. */ registerMemberKind(choudoufuSpec); + +/** The terraform member (#384): a directory of `.tf` files, read through a + * `chant.config.ts` behold generates in a scratch directory of its own. After + * choudoufu, so an estate whose roots carry a `live { … }` block is read as + * the choudoufu estate it is — the tag-owned reading is the richer one, and + * both would otherwise probe true. */ +registerMemberKind(terraformSpec); diff --git a/src/project.test.ts b/src/project.test.ts index f09f494..c5b617c 100644 --- a/src/project.test.ts +++ b/src/project.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect, beforeAll, afterAll } from "vitest"; import { detectProject, detectProjectShape, loadBeholdConfig, readExecutor, executorDesignation, resetExecutorCache } from "./project.ts"; +import { servesAsEstate } from "./member-kind.ts"; import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join, dirname } from "node:path"; @@ -231,9 +232,9 @@ describe("loadBeholdConfig", () => { }); it("keeps a member whose kind it cannot read, with the reason — never silently as chant", () => { - const cfg = loadBeholdConfig(make(JSON.stringify({ members: [{ dir: "x", kind: "terraform" }, { dir: "y", kind: "choudoufu" }, { kind: "chant" }, { dir: "" }] }))); + const cfg = loadBeholdConfig(make(JSON.stringify({ members: [{ dir: "x", kind: "pulumi" }, { dir: "y", kind: "choudoufu" }, { kind: "chant" }, { dir: "" }] }))); expect(cfg.members).toHaveLength(2); // an entry with no dir is not a declaration - expect(cfg.members![0]).toEqual({ dir: "x", invalid: expect.stringContaining('unknown member kind "terraform"') }); + expect(cfg.members![0]).toEqual({ dir: "x", invalid: expect.stringContaining('unknown member kind "pulumi"') }); expect(cfg.members![0]).toEqual({ dir: "x", invalid: expect.stringContaining("kinds are: chant, choudoufu") }); // #369: choudoufu is a registered kind, so its declaration is kept as one. expect(cfg.members![1]).toEqual({ dir: "y", kind: "choudoufu" }); @@ -295,7 +296,7 @@ describe("detectProjectShape", () => { // member used to disappear into. it("reports a declared member that fails its kind's probe, and one whose kind it does not know", () => { const dir = make({ - ".behold.json": JSON.stringify({ members: ["a", { dir: "b", kind: "chant" }, { dir: "c", kind: "terraform" }] }), + ".behold.json": JSON.stringify({ members: ["a", { dir: "b", kind: "chant" }, { dir: "c", kind: "pulumi" }] }), "a/chant.config.ts": "export default {};", "b/README.md": "not a chant project", }); @@ -305,12 +306,13 @@ describe("detectProjectShape", () => { membersFrom: "behold-config", invalidMembers: [ { dir: "b", invalid: "declared as chant, but b has no a chant.config.* file" }, - { dir: "c", invalid: expect.stringContaining('unknown member kind "terraform"') }, + { dir: "c", invalid: expect.stringContaining('unknown member kind "pulumi"') }, ], }); }); - // #369: the choudoufu kind's probe is a `live { … }` block in a root *.tf. + // #369: the choudoufu kind's probe is a `live { … }` block in a root *.tf, + // or (#387) the `estate.chdf.hcl` sidecar. it("keeps a declared choudoufu member whose root *.tf carries a live block, and reports one whose does not", () => { const dir = make({ ".behold.json": JSON.stringify({ members: ["app", { dir: "net", kind: "choudoufu" }, { dir: "plain", kind: "choudoufu" }] }), @@ -325,7 +327,7 @@ describe("detectProjectShape", () => { { dir: "net", kind: "choudoufu" }, ], membersFrom: "behold-config", - invalidMembers: [{ dir: "plain", invalid: "declared as choudoufu, but plain has no a `live { estate = … }` block in a root *.tf file" }], + invalidMembers: [{ dir: "plain", invalid: "declared as choudoufu, but plain has no an `estate.chdf.hcl` sidecar or a `live { estate = … }` block in a root *.tf file" }], }); // Workspaces: a choudoufu directory is claimed by its kind too; a chant // project is chant whatever else it holds. @@ -341,6 +343,36 @@ describe("detectProjectShape", () => { ]); }); + // #387: the sidecar is what `tools/estate-gen` writes, so a cohort has no + // `live` block anywhere and used to fail its own declared kind's probe. + it("keeps a declared choudoufu member declared by the estate.chdf.hcl sidecar", () => { + const dir = make({ + ".behold.json": JSON.stringify({ members: [{ dir: "s3", kind: "choudoufu" }, { dir: "net", kind: "choudoufu" }] }), + "s3/estate.chdf.hcl": 'estate = "s3-cohort"\n\nrecord_store "local" {\n path = ".tofu-records"\n}\n', + "s3/s3.tf": 'resource "aws_s3_bucket" "one" {\n bucket = "one"\n}\n', + "net/main.tf": 'terraform {\n live {\n estate = "net"\n }\n}\n', + }); + expect(detectProjectShape(dir).members).toEqual([ + { dir: "s3", kind: "choudoufu" }, + { dir: "net", kind: "choudoufu" }, + ]); + }); + + // #387: a rendered cohort is a bare directory nobody wrote a member list + // for, and `behold serve ` has taken one since #369 — so the shape says + // "a member" rather than the dead end doctor used to print. + it("reads a directory that is itself a member of a non-chant kind as an estate of one", () => { + const dir = make({ + "estate.chdf.hcl": 'estate = "s3-cohort"\n', + "s3.tf": 'resource "aws_s3_bucket" "one" {\n bucket = "one"\n}\n', + }); + expect(detectProjectShape(dir)).toEqual({ kind: "estate", members: [{ dir: ".", kind: "choudoufu" }], membersFrom: "itself" }); + // A chant project is still a project, not an estate of itself. + expect(detectProjectShape(make({ "chant.config.ts": "export default {};" })).kind).toBe("project"); + // And a directory no kind claims is still the dead end. + expect(detectProjectShape(make({ "README.md": "" })).kind).toBe("none"); + }); + it("is none, with the reasons, when every declared member is invalid", () => { const dir = make({ ".behold.json": JSON.stringify({ members: [{ dir: "b", kind: "chant" }] }), "b/README.md": "" }); expect(detectProjectShape(dir)).toEqual({ @@ -349,6 +381,26 @@ describe("detectProjectShape", () => { }); }); + // #384: the lane #378 exists for — "every estate this exists to draw is a + // directory of `.tf` files and nothing else". Nothing declares it a member; + // its own shape does. + it("is an estate of one when the directory itself is a member of a kind that is not chant", () => { + const tf = make({ + "envs/prod/versions.tf": "terraform {\n required_providers {\n aws = {}\n }\n}\n", + "envs/prod/main.tf": 'resource "aws_s3_bucket" "b" {}\n', + "modules/persona/versions.tf": "terraform {\n required_providers {\n aws = {}\n }\n}\n", + "modules/persona/main.tf": 'resource "aws_iam_role" "this" {}\n', + }); + expect(detectProjectShape(tf)).toEqual({ kind: "estate", members: [{ dir: ".", kind: "terraform" }], membersFrom: "itself" }); + + // A chant project that happens to hold Terraform is still a chant project, + // and a choudoufu estate is still choudoufu — the table's order decides. + const chant = make({ "chant.config.ts": "export default {};", "main.tf": 'terraform {}\nresource "aws_vpc" "x" {}\n' }); + expect(detectProjectShape(chant).kind).toBe("project"); + const tofu = make({ "main.tf": 'terraform {\n live {\n estate = "net"\n }\n}\n', "vpc.tf": 'resource "aws_vpc" "x" {}\n' }); + expect(detectProjectShape(tofu).members).toEqual([{ dir: ".", kind: "choudoufu" }]); + }); + it("is none for a directory that is neither — #193's dead end", () => { expect(detectProjectShape(make({ "notes.txt": "" }))).toEqual({ kind: "none" }); // A workspaces root whose members are not chant projects is not an estate. @@ -411,3 +463,43 @@ describe("executorDesignation — per member, not per primary (#165)", () => { rmSync(primary, { recursive: true, force: true }); }); }); + +// #389: the workbench's generated entries (an estate-gen cohort, a terralith) +// are ONE choudoufu estate each, and `behold serve ` used to answer +// the no-project card over them — the single-project read is `chant graph`, and +// a directory with no chant.config.ts and no lexicon has nothing for it to +// read. `servesAsEstate` is the predicate that routes such a directory through +// the estate compose path instead, where the member's own kind reads it. +describe("servesAsEstate", () => { + const dirs: string[] = []; + afterAll(() => dirs.forEach((d) => rmSync(d, { recursive: true, force: true }))); + const make = (files: Record): string => { + const dir = mkdtempSync(join(tmpdir(), "behold-serves-")); + dirs.push(dir); + for (const [rel, content] of Object.entries(files)) { + mkdirSync(dirname(join(dir, rel)), { recursive: true }); + writeFileSync(join(dir, rel), content); + } + return dir; + }; + + it("composes one directory that is a non-chant member — the sidecar form and the live-block form", () => { + expect(servesAsEstate([make({ "estate.chdf.hcl": 'estate = "s3-cohort"\n' })])).toBe(true); + expect(servesAsEstate([make({ "main.tf": 'terraform {\n live {\n estate = "tl"\n }\n}\n' })])).toBe(true); + }); + + it("leaves one chant project on the single-project path, as it always was", () => { + expect(servesAsEstate([make({ "chant.config.ts": "export default {};" })])).toBe(false); + }); + + it("leaves a directory no kind claims on the single-project path, so #193's no-project card still explains it", () => { + expect(servesAsEstate([make({ "README.md": "" })])).toBe(false); + }); + + it("composes more than one directory whatever the kinds, and composes nothing for no directory", () => { + const a = make({ "chant.config.ts": "export default {};" }); + const b = make({ "chant.config.ts": "export default {};" }); + expect(servesAsEstate([a, b])).toBe(true); + expect(servesAsEstate([])).toBe(false); + }); +}); diff --git a/src/project.ts b/src/project.ts index 164eb78..5fab11e 100644 --- a/src/project.ts +++ b/src/project.ts @@ -360,7 +360,12 @@ export function loadBeholdConfig(projectDir: string): BeholdConfig { * - `estate` — not itself a project, but it names member projects that are: * `.behold.json`'s `members`, else npm `workspaces`. This is the * `behold serve a b c` shape (#31) with the member list written - * down — `behold demo flux-estate` is one. + * down — `behold demo flux-estate` is one. Also a directory + * that IS a member of some non-chant kind (#387): a choudoufu + * estate is a bare directory with a sidecar in it, `behold + * serve ` already accepts one, and reporting it as + * an estate of one is what lets doctor run the kind's own + * line over it instead of "no chant.config.ts here". * - `none` — neither, which is #193's structured dead end. */ export type ProjectKind = "project" | "estate" | "none"; @@ -371,8 +376,10 @@ export interface ProjectShape { /** Members relative to `dir`, for `kind: "estate"` — only those whose * kind's probe accepts them, in declared order, each with its kind (#368). */ members?: { dir: string; kind: MemberKind }[]; - /** Where the member list came from, so a report never implies behold chose it. */ - membersFrom?: "behold-config" | "workspaces"; + /** Where the member list came from, so a report never implies behold chose + * it. `itself` is the one-member case above: nothing named it, the + * directory's own kind probe claimed it (#387). */ + membersFrom?: "behold-config" | "workspaces" | "itself"; /** Declared members behold cannot serve, with the reason (#368): a kind it * does not know, or a directory that fails its declared kind's probe. Set * only for `.behold.json` declarations — a workspaces entry that is not a @@ -436,6 +443,16 @@ export function detectProjectShape(projectDir: string): ProjectShape { } const invalidMembers = invalid.length ? { invalidMembers: invalid } : {}; if (members.length) return { kind: "estate", members, membersFrom: from, ...invalidMembers }; + // #387: nothing named a member, but the directory may BE one — a choudoufu + // estate is a bare directory with `estate.chdf.hcl` (or a `live` block) in + // it, and `behold serve ` has accepted one since #369 + // (`memberKindOf(dirs[0])`, src/server.ts). Only when no declaration failed: + // a `.behold.json` that named members and got them all wrong is a report + // about that, not a directory to re-probe. + if (!invalid.length) { + const own = memberKindOf(projectDir); + if (own && own !== "chant") return { kind: "estate", members: [{ dir: ".", kind: own }], membersFrom: "itself" }; + } return { kind: "none", ...invalidMembers }; } diff --git a/src/server.ts b/src/server.ts index 5caf547..d53e8d2 100644 --- a/src/server.ts +++ b/src/server.ts @@ -142,7 +142,8 @@ import { } from "./terraform-lens.ts"; import { choudoufuDiffNodes, readChoudoufuLive, type Runner as ChoudoufuRunner } from "./choudoufu-live.ts"; import { discoverCarvePlans, moveMembers, moveReceipt, movesPayload, readCarvePlan, type MoveMorphMoveInput } from "./choudoufu-moves.ts"; -import { memberKindOf } from "./member-kind.ts"; +import { memberKindOf, memberKindSpec, servesAsEstate } from "./member-kind.ts"; +import { TerraformReadError, discoverTerraformRoots, terraformRootsNote } from "./terraform-member.ts"; import { invalidateMember, memberIr } from "./member-ir.ts"; import { carveStatesFor, carveStatesUnder } from "./carve-discovery.ts"; import { foreignNote, type GraphIRWithForeign } from "./foreign.ts"; @@ -308,7 +309,7 @@ async function knownComponents(projectDir: string, opts: GraphOptions): Promise< * classified from chant's own stderr), plus "tier" — a non-default tier that * needed parameters this host doesn't have, generalized below from what used * to be a one-off `tierErrorNote`/`tierNote` bolted onto a plain error. */ -export type RouteErrorCode = ChantFailure["code"] | "tier" | "no-project" | "carve-report"; +export type RouteErrorCode = ChantFailure["code"] | "tier" | "no-project" | "carve-report" | "terraform-lexicon"; /** A read route's structured, typed error body (#72): a machine `code`, a * human `error` message, and a suggested `remedy` — what web/app.js's @@ -382,6 +383,11 @@ function noProjectError(projectDir: string): RouteError { } function errorResponse(c: Context, opts: GraphOptions, err: unknown) { + // #384: the terraform reader's own refusal already carries the one install + // line that fixes it, so it rides through as the precondition card rather + // than being classified from a chant stderr that was never written — the + // read refused before any spawn. + if (err instanceof TerraformReadError) return c.json(err.refusal, 500); const message = err instanceof Error ? err.message : String(err); const failure = err instanceof ChantCliError ? err.failure : classifyChantFailure(message); const routeError: RouteError = @@ -1437,7 +1443,8 @@ export function createApp( const switchServedProject = (dirs: string[], env?: string, spawnEnv?: Record): void => { addRecent(cfg.projectDir); cfg.projectDir = dirs[0]; - cfg.projectDirs = dirs.length > 1 ? dirs : undefined; + // #389: one directory composes too when chant cannot read it — see servesAsEstate. + cfg.projectDirs = servesAsEstate(dirs) ? dirs : undefined; // #372: a demo's scratch emulator reaches its choudoufu spawns through // this seam; a switch to anything else drops it, so no later project // inherits an endpoint meant for a demo. @@ -1493,6 +1500,10 @@ export function createApp( description: e.description, requires: e.requires, source: e.source, + // #388: which catalog listed it — `demos` ships in the package, + // `workbench` is this checkout's own `workbench.json`. The panel groups + // on it, the same two blocks `behold demo --list` prints. + catalog: e.catalog ?? "demos", // #268's consent half: a git entry is cloned from a public repo, so the // button says so (and names the repo) BEFORE anything is fetched. fetches: fetchesFromNetwork(e), @@ -1702,7 +1713,7 @@ export function createApp( { method: "GET", path: "/api/project", desc: "project info: dir, recents, environments, tiers, targets, stacks, preview lock" }, { method: "POST", path: "/api/project/open", desc: "switch the served project: JSON body {dir} (validated; preview-locked)" }, { method: "POST", path: "/api/project/reveal", desc: "open the OS file manager at a served/recent project dir: JSON body {dir?}" }, - { method: "GET", path: "/api/demos", desc: "the bundled demo catalog: [{name, description, requires, satisfiable, reason?, fetches, repo?, target, loaded}]" }, + { method: "GET", path: "/api/demos", desc: "the demo catalog, bundled and workbench: [{name, description, requires, catalog, satisfiable, reason?, fetches, repo?, target, loaded}]" }, { method: "POST", path: "/api/demos/open", @@ -1874,8 +1885,16 @@ export function createApp( // Entity graph only (not the component DAG, not multi-estate compose). const logical = url.searchParams.get("logical") === "1"; // Multi-estate (#31): graph each project and compose into one IR (namespaced - // ids, per-project boundary boxes, cross-stack edges). Single project → as-is. - const multi = cfg.projectDirs && cfg.projectDirs.length > 1; + // ids, per-project boundary boxes, cross-stack edges). A single chant + // project → as-is; a single directory of a non-chant kind composes too + // (#389, `servesAsEstate` — there is no chant to read it as-is with). + const multi = !!cfg.projectDirs; + // #384: how a single served directory is read when nothing composed it + // (a caller that set `projectDir` alone). `undefined` for a chant project + // and for a directory no kind claims — both keep the chant path below, + // byte for byte; a served non-chant directory normally arrives composed + // (#389) and never reaches this. + const ownKindVia = multi ? undefined : memberKindSpec(memberKindOf(cfg.projectDir) ?? "chant")?.via; // #382: what the Terraform zoom filter elided, when the estate branch ran it. let estateTfElision: TerraformElision = { dropped: {}, total: 0 }; let ir: GraphIR; @@ -2046,6 +2065,16 @@ export function createApp( // The SPA ignores it and paints the svg as before. const logicalNote = notesFor("logical", projected, undefined, base.nodes.length); return c.json({ ir: projected, svg, byContainer, meta: { projectDir: cfg.projectDir, env: metaEnv, tier: opts.tier ?? null, target: opts.target ?? null, mode: "logical", ...(logicalNote ? { note: logicalNote } : {}) } }); + } else if (ownKindVia) { + // #384: the served directory is itself a member of a kind that is not + // chant — a bare Terraform directory, a choudoufu estate — so it is + // read by that kind's own `via`, the same one `composeEstate` uses for + // it inside an estate (#368). Cached like every other source read + // (src/member-ir.ts), and none of the chant-shaped joins below run: + // there is no `cluster/` build root, no carve manifest and no + // kubeconfig behind a directory of `.tf` files, and #381 measured and + // refused the one edge pass that would have had an opinion about it. + ir = await memberIr(cfg.projectDir, opts, ownKindVia); } else { // The `cluster/` build root merges in (see the logical branch above). ir = mergeClusterRoot(await graphIr(cfg.projectDir, opts), await clusterRootGraphIr(cfg.projectDir, opts)); @@ -2103,7 +2132,7 @@ export function createApp( // components/logical emptiness has its own client handling (#182), a // lens can legitimately filter to nothing, and a chant.config.ts // project that declares no entities yet renders empty honestly. - if (!multi && !components && !logical && !opts.lens && ir.nodes.length === 0 && !existsSync(join(cfg.projectDir, "chant.config.ts"))) { + if (!multi && !components && !logical && !opts.lens && !ownKindVia && ir.nodes.length === 0 && !existsSync(join(cfg.projectDir, "chant.config.ts"))) { return c.json(noProjectError(cfg.projectDir), 404); } // Multi-estate (#31/M4): box each composed project's nodes via `groups. @@ -2139,8 +2168,17 @@ export function createApp( // (see /api/overlay's single-project branch, which already passed this) // — without it, example-k8s's `/api/graph` asserted "nothing in this // estate references anything else" at detail 2 while detail 3 has 2. + // #384: a served Terraform directory says which roots it found and which + // directories of `.tf` it skipped, so a root missing from the picture is + // visible here rather than by counting boxes. Ahead of the elision note, + // which is about the same estate's zoom: what is drawn, then what isn't. + // A served directory normally arrives composed (#389), so the terraform + // members are read off the composed list; the lone `projectDir` case is + // a caller that set it alone. + const tfDirs = (multi ? cfg.projectDirs! : [cfg.projectDir]).filter((d) => memberKindOf(d) === "terraform"); + const rootsNote = tfDirs.length ? tfDirs.map((d) => terraformRootsNote(discoverTerraformRoots(d))).join("; ") : undefined; const srcNote = - terraformElisionNote(tfElision, opts.detail) ?? + [rootsNote, terraformElisionNote(tfElision, opts.detail)].filter(Boolean).join("; ") || (multi ? estateLensNote : notesFor(srcZoom, ir, srcCompositeEdgesAttached, undefined, opts.detail ?? 2)); return c.json({ ir, @@ -2424,7 +2462,7 @@ export function createApp( // extras; #224 brought the runtime tier and the logical lens across (the // helm artifact join and the cluster-root merge are still primary-only — // both are per-project reads, not passes over the composed IR). - if (cfg.projectDirs && cfg.projectDirs.length > 1) { + if (cfg.projectDirs) { // The logical lens needs the rich attrs, exactly as on the // single-project path below (`logical ? { detail: 3 }`). The runtime // tier needs them too (#261): its whole subject is the Flux @@ -2769,7 +2807,7 @@ export function createApp( // plan, sliced per address and keyed by the composed id the pane looks up. // A member that cannot be read contributes nothing; the overlay's cover // note already says why. - const multi = !!cfg.projectDirs && cfg.projectDirs.length > 1; + const multi = !!cfg.projectDirs; for (const m of estateMembers(cfg.projectDirs ?? [cfg.projectDir])) { if (m.kind !== "choudoufu") continue; try { @@ -2903,7 +2941,7 @@ export function createApp( if (!env) return c.json({ error: "diff needs an environment — pick one, or start with --env" }, 400); // #370: a choudoufu member's node — the member whose composed prefix the // id carries (or the primary, on a single-member serve). - const multi = !!cfg.projectDirs && cfg.projectDirs.length > 1; + const multi = !!cfg.projectDirs; const owner = estateMembers(cfg.projectDirs ?? [cfg.projectDir]).find((m) => m.kind === "choudoufu" && (multi ? node.startsWith(`${m.name}/`) : true)); if (owner) { try { diff --git a/src/terraform-member.test.ts b/src/terraform-member.test.ts new file mode 100644 index 0000000..54cae51 --- /dev/null +++ b/src/terraform-member.test.ts @@ -0,0 +1,206 @@ +import { describe, it, expect, afterAll } from "vitest"; +import { existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readlinkSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, sep } from "node:path"; +import { fileURLToPath } from "node:url"; +import { memberSourceStamp } from "./member-ir.ts"; +import { + ESTATE_LINK, + HCL_PARSER_PKG, + TERRAFORM_LEXICON_PKG, + discoverTerraformRoots, + hasTerraformRoots, + readTerraformMember, + terraformReaderState, + terraformRootsNote, + terraformScratchConfig, + terraformScratchDir, + writeTerraformScratchProject, +} from "./terraform-member.ts"; + +// Fixture provenance (#384). `src/__fixtures__/terraform-estate/` is water +// park's `access/` (INTENTIUS/waterpark) reduced to the five shapes discovery +// has to tell apart, with the `.tf` bodies trimmed to the blocks that decide: +// +// envs/prod/ a root — `terraform { required_providers }`, a provider +// block, a resource, and a `module` call into modules/ +// baseline/ a root with NO provider block, called as a module by +// prod exactly as water park's is +// modules/persona/ the shared module. Its versions.tf is byte-for-byte +// baseline's, which is why #384's proposed probe ("what a +// root has and a called module does not") is not enough +// backends/ the backend fragment copied into a root — a `terraform` +// block and nothing to draw +// envs/dev/ a README and no `.tf`, so it is neither drawn nor +// reported +const HERE = dirname(fileURLToPath(import.meta.url)); +const ESTATE = join(HERE, "__fixtures__", "terraform-estate"); + +const made: string[] = []; +afterAll(() => made.forEach((d) => rmSync(d, { recursive: true, force: true }))); + +/** A temp directory that is nobody's estate — used as a `from` with no + * node_modules above it, which is what "the lexicon is not installed" looks + * like from a resolution's point of view. */ +function tmp(prefix: string): string { + const dir = mkdtempSync(join(tmpdir(), prefix)); + made.push(dir); + return dir; +} + +describe("root discovery (#384)", () => { + it("finds the roots and says why each other directory of .tf is not one", () => { + const scan = discoverTerraformRoots(ESTATE); + + expect(scan.roots.map((r) => r.dir)).toEqual(["baseline", "envs/prod"]); + expect(scan.roots.map((r) => r.name)).toEqual(["baseline", "prod"]); + expect(scan.skipped).toEqual([ + { dir: "backends", why: "no resource, data or module block — nothing to draw" }, + { dir: "modules/persona", why: "called as a module, never applied on its own" }, + ]); + }); + + it("keeps a called module out even though its terraform block is the root's, byte for byte", () => { + const versions = (rel: string): string => readFileSync(join(ESTATE, rel, "versions.tf"), "utf8"); + expect(versions("modules/persona")).toBe(versions("baseline")); + expect(discoverTerraformRoots(ESTATE).roots.some((r) => r.dir.startsWith("modules/"))).toBe(false); + }); + + it("does not report a directory that holds no .tf at all", () => { + const scan = discoverTerraformRoots(ESTATE); + expect([...scan.roots.map((r) => r.dir), ...scan.skipped.map((s) => s.dir)]).not.toContain("envs/dev"); + }); + + it("reads one root pointed at directly — the `behold serve access/envs/prod` case", () => { + const scan = discoverTerraformRoots(join(ESTATE, "envs", "prod")); + expect(scan.roots).toEqual([{ name: "prod", dir: "." }]); + expect(hasTerraformRoots(join(ESTATE, "envs", "prod"))).toBe(true); + }); + + it("probes a module pointed at directly as a root — it is only ever a module relative to a caller", () => { + expect(hasTerraformRoots(join(ESTATE, "modules", "persona"))).toBe(true); + }); + + it("claims nothing that is not Terraform", () => { + expect(hasTerraformRoots(tmp("behold-tf-empty-"))).toBe(false); + expect(hasTerraformRoots(join(ESTATE, "envs", "dev"))).toBe(false); + expect(hasTerraformRoots(join(ESTATE, "backends"))).toBe(false); + }); + + it("gives two roots with the same basename their full paths, so neither quietly takes the name", () => { + const root = tmp("behold-tf-collide-"); + for (const rel of ["a/prod", "b/prod"]) { + mkdirSync(join(root, ...rel.split("/")), { recursive: true }); + writeFileSync(join(root, ...rel.split("/"), "main.tf"), 'terraform {}\nresource "aws_s3_bucket" "b" {}\n'); + } + expect(discoverTerraformRoots(root).roots).toEqual([ + { name: "a-prod", dir: "a/prod" }, + { name: "b-prod", dir: "b/prod" }, + ]); + }); + + it("never walks into a Terraform working directory, which holds a copy of every module it fetched", () => { + const root = tmp("behold-tf-dotdir-"); + mkdirSync(join(root, ".terraform", "modules", "x"), { recursive: true }); + writeFileSync(join(root, ".terraform", "modules", "x", "main.tf"), 'terraform {}\nresource "aws_s3_bucket" "b" {}\n'); + expect(discoverTerraformRoots(root)).toEqual({ roots: [], skipped: [] }); + }); +}); + +describe("the note (#384)", () => { + it("names the roots and the skips with their reasons", () => { + expect(terraformRootsNote(discoverTerraformRoots(ESTATE))).toBe( + "2 roots — baseline, prod; skipped backends (no resource, data or module block — nothing to draw), modules/persona (called as a module, never applied on its own)", + ); + }); + + it("counts the skips it does not name, because a note is a line and not a report", () => { + const scan = { roots: [{ name: "prod", dir: "envs/prod" }], skipped: [1, 2, 3, 4, 5].map((n) => ({ dir: `m${n}`, why: "why" })) }; + expect(terraformRootsNote(scan)).toBe("1 root — prod; skipped m1 (why), m2 (why), m3 (why), +2 more"); + }); + + it("says nothing when there is nothing to say", () => { + expect(terraformRootsNote({ roots: [], skipped: [{ dir: "x", why: "why" }] })).toBeUndefined(); + expect(terraformRootsNote({ roots: [{ name: "prod", dir: "." }], skipped: [] })).toBe("1 root — prod"); + }); +}); + +describe("the scratch project (#384)", () => { + it("writes nothing under the estate — the whole write boundary this feature has to hold", () => { + const before = memberSourceStamp(ESTATE); + const project = writeTerraformScratchProject(ESTATE, discoverTerraformRoots(ESTATE)); + made.push(project); + + expect(memberSourceStamp(ESTATE)).toBe(before); + expect(project.startsWith(ESTATE + sep)).toBe(false); + expect(existsSync(join(ESTATE, "chant.config.ts"))).toBe(false); + expect(existsSync(join(ESTATE, ".behold"))).toBe(false); + }); + + it("is behold's own, named for scratch discipline, and the same directory every run", () => { + const first = terraformScratchDir(ESTATE); + expect(first).toBe(terraformScratchDir(ESTATE)); + expect(first.split(sep).at(-1)!.startsWith("behold-tf-")).toBe(true); + expect(terraformScratchDir(join(ESTATE, "baseline"))).not.toBe(first); + }); + + it("refuses outright when the temp directory it was handed lives inside the estate", () => { + expect(() => terraformScratchDir(ESTATE, join(ESTATE, "tmp"))).toThrow(/inside the estate/); + }); + + it("holds the config and the two symlinks the read needs, and rewrites the config only when it changed", () => { + const project = writeTerraformScratchProject(ESTATE, discoverTerraformRoots(ESTATE)); + made.push(project); + const config = join(project, "chant.config.ts"); + expect(lstatSync(join(project, ESTATE_LINK)).isSymbolicLink()).toBe(true); + expect(readlinkSync(join(project, ESTATE_LINK))).toBe(ESTATE); + + const stamp = memberSourceStamp(project); + writeTerraformScratchProject(ESTATE, discoverTerraformRoots(ESTATE)); + expect(memberSourceStamp(project)).toBe(stamp); + expect(readFileSync(config, "utf8")).toContain(`"prod": { dir: "${ESTATE_LINK}/envs/prod" }`); + }); + + it("names every root through the estate symlink, sorted, so the same estate generates the same bytes", () => { + const config = terraformScratchConfig(discoverTerraformRoots(ESTATE), ESTATE_LINK); + expect(config).toContain(`import "${TERRAFORM_LEXICON_PKG}";`); + expect(config).toContain('lexicons: ["terraform"]'); + expect(config.indexOf('"baseline"')).toBeLessThan(config.indexOf('"prod"')); + // A root that IS the served directory is the symlink itself, not `/.`. + expect(terraformScratchConfig({ roots: [{ name: "prod", dir: "." }], skipped: [] }, ESTATE_LINK)).toContain(`"prod": { dir: "${ESTATE_LINK}" }`); + }); +}); + +describe("the reader behold does not install (#384)", () => { + it("refuses politely, naming both packages, where it looked, and the one install line", () => { + const state = terraformReaderState(tmp("behold-tf-nolexicon-")); + + expect(state.refusal!.code).toBe("terraform-lexicon"); + expect(state.refusal!.error).toContain(TERRAFORM_LEXICON_PKG); + expect(state.refusal!.error).toContain(HCL_PARSER_PKG); + expect(state.refusal!.error).toContain("behold does not install"); + expect(state.refusal!.remedy).toMatch(/^Install .* beside behold, then reload\.$/); + // The declared ranges, read from behold's own manifest — so the line and + // the package.json cannot drift apart. + expect(state.refusal!.remedy).toContain(`${TERRAFORM_LEXICON_PKG}@${state.lexicon.range}`); + expect(state.lexicon.range).toMatch(/^\^?\d/); + }); + + it("is what a read throws, with the refusal on the error the routes render", async () => { + const state = terraformReaderState(tmp("behold-tf-nolexicon-")); + const stamped = memberSourceStamp(ESTATE); + await expect(readTerraformMember(ESTATE, {}, state)).rejects.toMatchObject({ + refusal: { code: "terraform-lexicon", remedy: state.refusal!.remedy }, + }); + // And it wrote nothing on the way out: the refusal comes before the scan + // and before the scratch project. + expect(memberSourceStamp(ESTATE)).toBe(stamped); + }); + + it("refuses a directory with no root the same way, pointing at the roots instead of an install", async () => { + const empty = tmp("behold-tf-noroots-"); + await expect(readTerraformMember(empty, {}, { lexicon: { pkg: TERRAFORM_LEXICON_PKG, version: "0.61.0" }, parser: { pkg: HCL_PARSER_PKG, version: "0.21.0" }, from: empty })).rejects.toThrow( + /holds no Terraform root/, + ); + }); +}); diff --git a/src/terraform-member.ts b/src/terraform-member.ts new file mode 100644 index 0000000..3e0c735 --- /dev/null +++ b/src/terraform-member.ts @@ -0,0 +1,527 @@ +/** + * Serving a bare Terraform directory (#384, the last behold-side piece of + * #378's lane). + * + * #379/#380/#382 made a Terraform estate legible, but only for a project that + * already declares the lexicon. Point behold at the thing this exists to draw — + * a directory of `.tf` files and nothing else — and it refused, correctly: + * there is no `chant.config.ts` there, and #378 chose not to ask an estate for + * one (INTENTIUS/waterpark#88 was withdrawn — the estate is more useful + * untouched). So behold generates the reader config itself, in a scratch + * directory of its own, and points chant at that. Nothing is written under the + * served directory; the Invariant's one in-project write stays + * `.behold/layout.json`. + * + * Three decisions shape this, and each was a real trade (#384's own three open + * questions). + * + * **The lexicon is opt-in, not part of behold's install.** chant resolves + * `@intentius/chant-lexicon-terraform` from the config file's own location, so + * the generated project has to see it. Making it a dependency would put an HCL + * parser — `@cdktf/hcl2json`, a ~1.8 MB wasm blob — in every user's install, + * most of whom serve chant projects, and would break `src/carve-lens.ts`'s + * posture that behold ships no Terraform tooling. So both are declared as + * OPTIONAL PEERS in behold's package.json (the versions this reader was built + * against, and the only place they are named), probed here, and refused with + * the one install line when absent — the way `behold demo` gates on binaries it + * does not ship and the choudoufu member refuses when its binary is missing. + * The refusal names where behold looked, because "install it" is only useful + * beside "here". + * + * **Roots are discovered, not declared.** `chant carve advise --from ` + * over the parent of six roots returns zero resources: the parse is one root at + * a time and the lexicon's config names roots explicitly. #384 proposed the + * cheap probe — a directory with a `.tf` declaring a `terraform {}` or + * `provider` block, "which is what a root has and a called module does not" — + * and the real estate says otherwise: water park's `access/modules/persona` is + * a shared module called by three roots and its `versions.tf` opens with + * `terraform {` and `required_providers`, byte-for-byte the shape + * `access/baseline` has. So the probe stands and two exclusions stand beside + * it, both measured on that estate and both reported rather than silent: + * a directory under a `modules/` segment is a called module (Terraform's own + * standard module structure, and what waterpark's README says of it), and a + * directory with no `resource`, `data` or `module` block has nothing to draw + * (`access/backends` is two backend fragments, one of which is copied into a + * root). behold parses no HCL: this is the same regex-depth probe the + * choudoufu member uses for its `live {` block, per candidate directory. + * + * **A member kind, not a serve-time special case.** #378 said there is no + * `terraform` member kind and meant it about READING: behold parses no HCL and + * the render goes through chant. A kind whose `read` shells `chant graph` + * against a generated config does not contradict that — it is a scaffold, not + * a second reader — and it inherits the probe, the cache stamp, the doctor line + * and estate composition (#368) for free. A Terraform root can therefore sit in + * a composed estate beside a chant project and a choudoufu estate. + * + * --------------------------------------------------------------------------- + * THE SCRATCH PROJECT. `/behold-tf-`: named + * `behold-*` and cleared through `assertScratch` like everything else behold + * creates (src/scratch.ts), deterministic per estate path so a reload reuses it + * instead of littering, and asserted to be outside the estate before a byte is + * written. It holds three things: + * + * - `chant.config.ts` — `lexicons: ["terraform"]` and one `terraform.roots` + * entry per discovered root. + * - `node_modules` — a symlink to behold's OWN `node_modules`. Node + * resolution from the config's location then finds the lexicon, chant and + * zod exactly as it would inside behold's install. The alternative (place + * the scratch project under behold's tree so resolution walks up into it) + * was refused: behold's install is often a global npm prefix or an npx + * cache, both of which may be read-only, and a scratch directory that + * lives beside the operator's other scratch state is easier to find and to + * delete. One symlink buys the same resolution with none of that. + * - `estate` — a symlink to the served directory, which is what each root's + * `dir` goes through. Not decoration: the lexicon's `hcl/descend.ts` sets a + * root's module boundary to the root's OWN directory when the root's `dir` + * resolves outside the project root, so absolute `dir`s cost every + * `../modules/x` call the estate makes. Measured on water park: 72 nodes + * and 6 resources with absolute dirs, 247 nodes and 43 resources through + * the symlink — the same 247 #378 measured with a config committed inside + * the estate. Nothing about the answer mentions the scratch path: a + * terraform entity carries `attrs.file` relative to its root and no + * `sourceLoc`. + * --------------------------------------------------------------------------- + */ +import { createHash } from "node:crypto"; +import { createRequire } from "node:module"; +import { existsSync, lstatSync, mkdirSync, readFileSync, readdirSync, readlinkSync, realpathSync, symlinkSync, unlinkSync, writeFileSync } from "node:fs"; +import { basename, dirname, join, relative, resolve, sep } from "node:path"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; +import type { GraphIR } from "@intentius/chant"; +import type { GraphOptions } from "./chant.ts"; +import type { MemberVia } from "./member-ir.ts"; +import type { MemberKindSpec } from "./member-kind.ts"; +import { SCRATCH_PREFIX, assertScratch } from "./scratch.ts"; + +/** The reader chant loads for a Terraform root, and the HCL parser it lazily + * loads underneath. Both are optional peers of behold (see the header); the + * ranges live in behold's package.json and are read from it, so the install + * line and the manifest cannot drift apart. */ +export const TERRAFORM_LEXICON_PKG = "@intentius/chant-lexicon-terraform"; +export const HCL_PARSER_PKG = "@cdktf/hcl2json"; + +/** How far below the served directory a root may sit. water park's roots are + * two levels down (`envs/prod`, `satellites/waterpark-runner`); three is one + * more than the estate this was measured against needs, and it bounds the walk + * on a repository that is mostly not Terraform. */ +export const ROOT_SCAN_DEPTH = 3; + +/** Directories the walk never enters: build output, installed packages, and + * Terraform's own working directory (`.terraform` holds a copy of every module + * it fetched, which would read as a tree of roots). */ +const SKIP_DIRS = new Set([".terraform", "node_modules", ".git", "dist", ".chant", "cdk.out"]); + +/** What a root states about itself: the providers it requires, its backend, or + * a provider configuration. Line-anchored, like the choudoufu probe's `live {` + * — a regex over the file, never a parse. */ +const ROOT_BLOCK = /^(terraform\s*\{|provider\s+")/m; + +/** Something to draw: #382's estate tier, in HCL. A directory of backend + * fragments has none. */ +const ESTATE_BLOCK = /^(resource|data|module)\s+"/m; + +/** Terraform's standard module structure puts called modules here. */ +const MODULES_SEGMENT = "modules"; + +// --------------------------------------------------------------------------- +// Root discovery. +// --------------------------------------------------------------------------- + +/** One discovered root: the name it is keyed and boxed under, and its path + * relative to the served directory (`.` when the served directory is itself a + * root). */ +export interface TerraformRoot { + name: string; + /** Relative to the estate, POSIX-separated. `.` for the estate itself. */ + dir: string; +} + +/** A directory holding `.tf` files that is not a root, and why — so a missing + * root is visible rather than silently absent (#384). */ +export interface TerraformSkip { + dir: string; + why: string; +} + +export interface TerraformRootScan { + roots: TerraformRoot[]; + skipped: TerraformSkip[]; +} + +/** Why this directory of `.tf` files is not a root, or undefined when it is. */ +function notARoot(rel: string, sources: string[]): string | undefined { + if (rel.split("/").slice(0, -1).includes(MODULES_SEGMENT)) return "called as a module, never applied on its own"; + if (!sources.some((s) => ROOT_BLOCK.test(s))) return "no `terraform` or `provider` block — a root states the providers it needs"; + if (!sources.some((s) => ESTATE_BLOCK.test(s))) return "no resource, data or module block — nothing to draw"; + return undefined; +} + +/** A directory's own `*.tf` files, read whole. Unreadable files are not + * evidence either way, exactly as `hasLiveBlock` treats them. */ +function tfSources(dir: string): string[] { + let names: string[]; + try { + names = readdirSync(dir).filter((f) => f.endsWith(".tf")).sort(); + } catch { + return []; + } + const out: string[] = []; + for (const name of names) { + try { + out.push(readFileSync(join(dir, name), "utf8")); + } catch { + // unreadable: skip it + } + } + return out; +} + +/** + * The roots under `estate`, and every other directory of `.tf` files with the + * reason it is not one. Sync, read-only, no HCL parsed and no code run — the + * contract a member kind's probe is held to (AGENTS.md, "Adding a member + * kind"). + * + * Names are the root directory's own basename (`envs/prod` → `prod`), which is + * what an operator calls it and what the box is titled. Two roots whose + * basenames collide both take their full relative path with `/` → `-`, so a + * name is never quietly given to one of them. + */ +export function discoverTerraformRoots(estate: string): TerraformRootScan { + const root = resolve(estate); + const found: { rel: string; skip?: string }[] = []; + const walk = (dir: string, depth: number): void => { + const rel = relative(root, dir).split(sep).join("/") || "."; + const sources = tfSources(dir); + if (sources.length) found.push({ rel, skip: notARoot(rel, sources) }); + if (depth >= ROOT_SCAN_DEPTH) return; + let entries; + try { + entries = readdirSync(dir, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : 1)); + } catch { + return; + } + for (const e of entries) { + if (!e.isDirectory() || SKIP_DIRS.has(e.name) || e.name.startsWith(".")) continue; + walk(join(dir, e.name), depth + 1); + } + }; + walk(root, 0); + + const rootRels = found.filter((f) => !f.skip).map((f) => f.rel); + const shortOf = (rel: string): string => (rel === "." ? basename(root) : rel.slice(rel.lastIndexOf("/") + 1)); + const counts = new Map(); + for (const rel of rootRels) counts.set(shortOf(rel), (counts.get(shortOf(rel)) ?? 0) + 1); + return { + roots: rootRels.map((rel) => ({ name: counts.get(shortOf(rel))! > 1 ? rel.split("/").join("-") : shortOf(rel), dir: rel })), + skipped: found.filter((f) => !!f.skip).map((f) => ({ dir: f.rel, why: f.skip! })), + }; +} + +/** Does `dir` hold at least one Terraform root? The kind's probe. */ +export function hasTerraformRoots(dir: string): boolean { + return discoverTerraformRoots(dir).roots.length > 0; +} + +/** + * What discovery found, as one clause for the graph's note — the roots drawn + * and the directories that hold `.tf` and are not roots, the way + * `terraformElisionNote` says what a zoom left out. At most three skips are + * named; the rest are counted, because a note is a line and not a report. + */ +export function terraformRootsNote(scan: TerraformRootScan): string | undefined { + if (!scan.roots.length) return undefined; + const roots = `${scan.roots.length} root${scan.roots.length === 1 ? "" : "s"} — ${scan.roots.map((r) => r.name).join(", ")}`; + if (!scan.skipped.length) return roots; + const named = scan.skipped.slice(0, 3).map((s) => `${s.dir} (${s.why})`); + const more = scan.skipped.length > 3 ? `, +${scan.skipped.length - 3} more` : ""; + return `${roots}; skipped ${named.join(", ")}${more}`; +} + +// --------------------------------------------------------------------------- +// The reader: is the lexicon here at all? +// --------------------------------------------------------------------------- + +/** One optional peer: the range behold declares, and the version installed + * beside it (undefined when it is not there at all). */ +export interface OptionalPeer { + pkg: string; + range?: string; + version?: string; +} + +/** behold's own directory — where its `node_modules` is resolved from, and the + * directory whose package.json declares the peers. `dist/` in a build, `src/` + * under tsx; both sit one level under the package root. */ +const beholdDir = (): string => dirname(fileURLToPath(import.meta.url)); + +/** The optional-peer ranges behold declares, read from its own manifest so the + * install line quotes what the package actually asks for (the same reading + * `chantFloor` does for the chant floor). */ +export function declaredPeerRange(pkg: string, manifestDir = join(beholdDir(), "..")): string | undefined { + try { + const manifest = JSON.parse(readFileSync(join(manifestDir, "package.json"), "utf8")) as { peerDependencies?: Record }; + return manifest.peerDependencies?.[pkg]; + } catch { + return undefined; + } +} + +/** The version of `pkg` as resolved from `from`, or undefined when it does not + * resolve. Walks up from the resolved entry to the manifest, because chant's + * packages ship raw TypeScript with an `exports` map that rewrites + * `./package.json` to a file that does not exist (src/chant.ts says the same + * of its own resolution). */ +export function resolvedVersion(pkg: string, from = beholdDir()): string | undefined { + const req = createRequire(join(resolve(from), "noop.js")); + let entry: string; + try { + entry = req.resolve(pkg); + } catch { + return undefined; + } + let dir = dirname(entry); + for (;;) { + try { + const manifest = JSON.parse(readFileSync(join(dir, "package.json"), "utf8")) as { name?: string; version?: string }; + if (manifest.name === pkg) return manifest.version ?? ""; + } catch { + // keep walking up + } + const parent = dirname(dir); + if (parent === dir) return undefined; + dir = parent; + } +} + +/** #193's structured refusal, in this reader's own words: what is missing and + * the one command that fixes it. */ +export interface TerraformRefusal { + error: string; + code: "terraform-lexicon"; + remedy: string; +} + +/** What behold can see of the reader: both peers, and the refusal when either + * is absent. */ +export interface TerraformReaderState { + lexicon: OptionalPeer; + parser: OptionalPeer; + /** Where behold looked — named in the refusal, since "install it" is only + * useful beside "here". */ + from: string; + refusal?: TerraformRefusal; +} + +const spec = (peer: OptionalPeer): string => `${peer.pkg}@${peer.range ?? "latest"}`; + +/** + * Probe the reader. Read-only and cheap (two module resolutions), so the graph + * route, the doctor line and the member's `read` all ask the same question and + * get the same answer. + */ +export function terraformReaderState(from = beholdDir()): TerraformReaderState { + const peer = (pkg: string): OptionalPeer => ({ pkg, ...(declaredPeerRange(pkg) ? { range: declaredPeerRange(pkg) } : {}), ...(resolvedVersion(pkg, from) !== undefined ? { version: resolvedVersion(pkg, from) } : {}) }); + const lexicon = peer(TERRAFORM_LEXICON_PKG); + const parser = peer(HCL_PARSER_PKG); + const missing = [lexicon, parser].filter((p) => p.version === undefined); + if (!missing.length) return { lexicon, parser, from }; + return { + lexicon, + parser, + from, + refusal: { + error: + `Reading a Terraform estate needs chant's terraform lexicon, which behold does not install: ` + + `${missing.map((p) => p.pkg).join(" and ")} ${missing.length === 1 ? "is" : "are"} not resolvable from ${from}.`, + code: "terraform-lexicon", + remedy: `Install ${missing.map(spec).join(" ")} beside behold, then reload.`, + }, + }; +} + +/** A refusal, thrown where a read cannot answer. `firstLine` in src/estate.ts + * reads the part after "exited N: ", so the message is shaped that way — + * the same shape `ChoudoufuReadError` takes. */ +export class TerraformReadError extends Error { + constructor( + readonly refusal: TerraformRefusal, + dir: string, + ) { + super(`terraform reader ${dir} exited 1: ${refusal.error}`); + } +} + +// --------------------------------------------------------------------------- +// The scratch project. +// --------------------------------------------------------------------------- + +/** behold's own `node_modules` — the directory the scratch project's symlink + * points at. Derived from the chant install behold itself resolves, so it is + * the same tree the rest of behold reads from. */ +export function beholdNodeModules(from = beholdDir()): string | undefined { + const req = createRequire(join(resolve(from), "noop.js")); + let entry: string; + try { + entry = req.resolve("@intentius/chant"); + } catch { + return undefined; + } + for (let dir = dirname(entry); ; ) { + if (basename(dir) === "node_modules") return dir; + const parent = dirname(dir); + if (parent === dir) return undefined; + dir = parent; + } +} + +/** The scratch project for one estate: `behold-tf-` under the OS temp + * directory. Deterministic (one directory per estate path, reused across + * runs), `behold-*` (src/scratch.ts, asserted here rather than at the write), + * and never inside the estate — which is checked rather than assumed, because + * that check is the whole write boundary this feature has to hold. */ +export function terraformScratchDir(estate: string, tmp = tmpdir()): string { + const target = resolve(estate); + const name = `${SCRATCH_PREFIX}tf-${createHash("sha1").update(target).digest("hex").slice(0, 12)}`; + assertScratch(name); + // realpath first: on macOS `tmpdir()` is a symlink (/var → /private/var), and + // an unresolved path makes the containment check below compare two different + // spellings of the same directory. + let root = resolve(tmp); + try { + root = realpathSync(root); + } catch { + // no such temp dir: the join below fails loudly at write time instead + } + const dir = join(root, name); + if (dir === target || dir.startsWith(target + sep)) { + throw new Error(`scratch discipline: the generated reader config for ${target} would land inside the estate (${dir})`); + } + return dir; +} + +/** The `chant.config.ts` a scan generates. Pure and deterministic: the same + * estate and the same roots produce the same bytes, so a reload rewrites + * nothing. Exported for testing. */ +export function terraformScratchConfig(scan: TerraformRootScan, estateLink: string): string { + const roots = [...scan.roots].sort((a, b) => (a.name < b.name ? -1 : 1)); + const entries = roots.map((r) => ` ${JSON.stringify(r.name)}: { dir: ${JSON.stringify(r.dir === "." ? estateLink : `${estateLink}/${r.dir}`)} },`); + return [ + "// Generated by behold (#384) — the reader config for a Terraform estate that", + "// has none of its own. behold writes this OUTSIDE the estate and rewrites it", + "// on each read; nothing here is yours to edit, and the estate itself is", + `// untouched. \`${estateLink}\` is a symlink to the served directory.`, + 'import type { ChantConfig } from "@intentius/chant/config";', + `import ${JSON.stringify(TERRAFORM_LEXICON_PKG)};`, + "", + "export default {", + ' lexicons: ["terraform"],', + " terraform: {", + " roots: {", + ...entries, + " },", + " },", + "} satisfies ChantConfig;", + "", + ].join("\n"); +} + +/** The symlink name the estate is reached through inside the scratch project. */ +export const ESTATE_LINK = "estate"; + +/** Point `path` at `target`, replacing whatever is there when it is not + * already that link. Nothing else in the scratch directory is ever removed. */ +function relink(path: string, target: string): void { + try { + if (lstatSync(path).isSymbolicLink() && readlinkSync(path) === target) return; + unlinkSync(path); + } catch { + // absent, or not a link we can read: fall through to creating it + } + symlinkSync(target, path); +} + +/** + * Write (or refresh) the scratch project for `estate` and return its directory. + * The only writes this feature performs, all of them under the OS temp + * directory: the config, and the two symlinks the header explains. + */ +export function writeTerraformScratchProject(estate: string, scan: TerraformRootScan, tmp = tmpdir()): string { + const target = resolve(estate); + const dir = terraformScratchDir(target, tmp); + mkdirSync(dir, { recursive: true }); + relink(join(dir, ESTATE_LINK), target); + const modules = beholdNodeModules(); + if (modules) relink(join(dir, "node_modules"), modules); + const config = terraformScratchConfig(scan, ESTATE_LINK); + const path = join(dir, "chant.config.ts"); + // Rewritten only when it changed: the file's mtime is what chant's own + // caching and a watcher would key on, and a reload that changed nothing + // should look like nothing changed. + if (!existsSync(path) || readFileSync(path, "utf8") !== config) writeFileSync(path, config); + return dir; +} + +// --------------------------------------------------------------------------- +// The kind. +// --------------------------------------------------------------------------- + +/** How a terraform member is read: behold's own chant over the generated + * config, which is the version half of the cache key, plus the lexicon that + * will actually answer — an upgraded reader is a different key, not a stale + * hit. */ +export const terraformVia: MemberVia = { + tool: () => { + // The reader, not the estate: behold's own chant is fixed for the life of + // the process, and the lexicon is what turns the same HCL into a different + // answer. An absent one stamps as absent rather than throwing — `read` is + // where a missing reader is refused, with the install line. + const state = terraformReaderState(); + return `${TERRAFORM_LEXICON_PKG}\0${state.lexicon.version ?? "absent"}\0${state.parser.version ?? "absent"}`; + }, + read: (dir: string, opts: GraphOptions): Promise => readTerraformMember(dir, opts), +}; + +/** + * The uncached read: discover the roots, write the scratch project, and let + * chant graph it. The reader state is an argument with a default, the way + * `readLiveCheck`'s spawn is, so a test can ask what an install without the + * lexicon answers without uninstalling anything. + */ +export async function readTerraformMember(dir: string, opts: GraphOptions, state: TerraformReaderState = terraformReaderState()): Promise { + if (state.refusal) throw new TerraformReadError(state.refusal, dir); + const scan = discoverTerraformRoots(dir); + if (!scan.roots.length) { + throw new TerraformReadError( + { + error: `${dir} holds no Terraform root — no directory under it declares a \`terraform\` or \`provider\` block beside a resource, data or module block.`, + code: "terraform-lexicon", + remedy: "Point behold at the directory that holds the roots (or at one root), then reload.", + }, + dir, + ); + } + const project = writeTerraformScratchProject(dir, scan); + // Source only. A Terraform estate has no live half here: what the cloud + // holds is choudoufu's question (#366) or an Op's, and `--live` against a + // reader lexicon would ask chant to observe entities it never applied. + const { live: _live, overlay: _overlay, env: _env, ...source } = opts; + // Imported here, not at module load: src/member-kind.ts registers this spec, + // src/chant.ts imports src/project.ts which imports that table, and a + // load-time edge back into src/chant.ts from here is the cycle that module's + // header refuses. A dynamic import inside the read is not one, and a test's + // mock of ./chant.ts still answers it. + const { graphIr } = await import("./chant.ts"); + return graphIr(project, source); +} + +/** The terraform member (#384): a directory of `.tf` files, read through a + * generated chant config. Registered after chant and choudoufu, so a directory + * that is also a chant project or a choudoufu estate is read as one of those. */ +export const terraformSpec: MemberKindSpec = { + kind: "terraform", + probe: hasTerraformRoots, + expects: "Terraform root under it — a `terraform` or `provider` block beside a resource, data or module block", + via: terraformVia, +}; diff --git a/src/terraform-route.test.ts b/src/terraform-route.test.ts index 6f374fb..daf11f8 100644 --- a/src/terraform-route.test.ts +++ b/src/terraform-route.test.ts @@ -1,12 +1,15 @@ -import { describe, it, expect, vi, beforeEach } from "vitest"; -import { readFileSync } from "node:fs"; +import { describe, it, expect, vi, afterAll, beforeEach } from "vitest"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; import type { GraphIR } from "@intentius/chant"; -// The routes read a Terraform estate the way they read any chant project — one -// `chant graph` shell-out — so the seam to mock is the same one estate.test.ts -// mocks. Everything else stays real: the three passes, the lens, the note. +// The routes read a Terraform estate that DECLARES the lexicon the way they +// read any chant project — one `chant graph` shell-out — so the seam to mock is +// the same one estate.test.ts mocks. Everything else stays real: the three +// passes, the lens, the note. The bare directory #384 serves takes the member +// kind's own read instead, and the block at the bottom of this file covers it. vi.mock("./chant.ts", async (importOriginal) => ({ ...(await importOriginal()), graphIr: vi.fn(), @@ -17,6 +20,8 @@ import { Broadcaster } from "./events.ts"; import { FrameBuffer } from "./frames.ts"; import { OpRunner } from "./op-runner.ts"; import { resetMemberIrCache } from "./member-ir.ts"; +import { registerMemberKind } from "./member-kind.ts"; +import { TerraformReadError, hasTerraformRoots } from "./terraform-member.ts"; // Provenance: `chant graph --format ir --detail 3` over water park's // `access/baseline` and its `waterpark-runner` satellite as two roots, through @@ -82,3 +87,83 @@ describe("GET /api/graph over a Terraform estate (#379/#380/#382)", () => { expect(meta.note ?? "").not.toContain("showing the estate"); }); }); + +// --------------------------------------------------------------------------- +// #384 — the same routes over a directory of `.tf` files and nothing else. +// The kind is registered with its REAL probe, so this is the dispatch a bare +// Terraform directory actually takes; only the read (where the lexicon and the +// generated config would be) is stubbed, and the roots note comes from the real +// walk over the estate on disk. +// --------------------------------------------------------------------------- +describe("GET /api/graph over a bare Terraform directory (#384)", () => { + const made: string[] = []; + afterAll(() => made.forEach((d) => rmSync(d, { recursive: true, force: true }))); + + /** water park's shapes in miniature: two roots, a called module, a backend + * fragment. See src/terraform-member.test.ts for the provenance. */ + function estate(): string { + const root = mkdtempSync(join(tmpdir(), "behold-tf-route-")); + made.push(root); + const write = (rel: string, content: string): void => { + mkdirSync(dirname(join(root, rel)), { recursive: true }); + writeFileSync(join(root, rel), content); + }; + const versions = "terraform {\n required_providers {\n aws = {}\n }\n}\n"; + write("envs/prod/versions.tf", versions); + write("envs/prod/main.tf", 'resource "aws_s3_bucket" "artifacts" {}\n'); + write("baseline/versions.tf", versions); + write("baseline/main.tf", 'resource "aws_iam_policy" "boundary" {}\n'); + write("modules/persona/versions.tf", versions); + write("modules/persona/main.tf", 'resource "aws_iam_role" "this" {}\n'); + write("backends/backend.local.tf", 'terraform {\n backend "local" {}\n}\n'); + return root; + } + + const serve = (dir: string) => { + const broadcaster = new Broadcaster(); + return createApp({ projectDir: dir, port: 0 }, broadcaster, new FrameBuffer(), new OpRunner({ projectDir: dir, broadcaster, onDone: () => {} })); + }; + + it("serves it, and the note says which roots it found and what it skipped", async () => { + const dir = estate(); + const read = vi.fn(async (_dir: string) => fixture()); + registerMemberKind({ kind: "terraform", probe: hasTerraformRoots, expects: "a Terraform root", via: { tool: () => "lexicon\0v1", read: read as never } }); + + const { ir, meta } = (await (await serve(dir).request("/api/graph?detail=2")).json()) as { ir: GraphIR; meta: { note?: string } }; + + // Read by the kind, never by chant. + expect(read.mock.calls.map(([d]) => d)).toEqual([dir]); + expect(graphIr).not.toHaveBeenCalled(); + // The #379/#380/#382 passes still ran: boxed by root, titled by resource type. + expect(Object.keys(ir.groups.byStack ?? {}).sort()).toEqual(["baseline", "runner"]); + expect(meta.note).toBe( + "2 roots — baseline, prod; skipped backends (no resource, data or module block — nothing to draw), modules/persona (called as a module, never applied on its own); " + + "showing the estate — 11 variables, 9 outputs, 3 terraform blocks, 2 locals blocks, 1 provider not drawn (outputs and variables appear at detail 3 — ⌘K → attributes)", + ); + }); + + it("answers the reader's own refusal, with the install line, when the lexicon is not there", async () => { + const dir = estate(); + const refusal = { + error: "Reading a Terraform estate needs chant's terraform lexicon, which behold does not install: …", + code: "terraform-lexicon" as const, + remedy: "Install @intentius/chant-lexicon-terraform@^0.61.0 @cdktf/hcl2json@^0.21.0 beside behold, then reload.", + }; + registerMemberKind({ + kind: "terraform", + probe: hasTerraformRoots, + expects: "a Terraform root", + via: { + tool: () => "lexicon\0absent", + read: (async () => { + throw new TerraformReadError(refusal, dir); + }) as never, + }, + }); + + const res = await serve(dir).request("/api/graph?detail=2"); + + expect(res.status).toBe(500); + expect(await res.json()).toEqual(refusal); + }); +}); diff --git a/workbench.json b/workbench.json new file mode 100644 index 0000000..127bc5d --- /dev/null +++ b/workbench.json @@ -0,0 +1,199 @@ +{ + "demos": [ + { + "name": "chant-getting-started", + "description": "chant's own getting-started example, served in place from ../chant — the source graph, no substrate, no setup, 8 nodes. The smallest thing in the workbench, and the one that answers 'did I break plain chant reading?' in a second. Served exactly as it sits: nothing is copied and nothing is installed in your chant tree, so the example needs its own node_modules (npm install there once) to read.", + "source": "local", + "path": "../chant/examples/getting-started", + "inPlace": true, + "requires": [], + "serve": {} + }, + { + "name": "chant-local-cloud-trio", + "description": "chant's local-cloud-trio example, served in place from ../chant — one project declaring across aws, azure and gcp, as the source graph: 8 nodes, 2 edges, no substrate booted. Served as it sits, nothing written: like chant-getting-started it needs its own node_modules to read.", + "source": "local", + "path": "../chant/examples/local-cloud-trio", + "inPlace": true, + "requires": [], + "serve": {} + }, + { + "name": "fountain-ops", + "description": "The mature estate on your working checkout: ../fountain-ops served IN PLACE with --env local. Its setup is the checkout's own `just up` — a k3d cluster named fountain-local, images built and pulled, the whole app applied and verified — so it RUNS IN YOUR WORKING COPY and switches your kubectl context, unlike every other entry here. About five minutes cold. `just down` in ../fountain-ops removes the cluster; behold never does. The git `fountain` demo is this same estate from a fresh clone.", + "source": "local", + "path": "../fountain-ops", + "inPlace": true, + "requires": [ + "docker", + "k3d", + "kubectl", + "jq", + "just" + ], + "setup": "just up", + "serve": { + "env": "local" + } + }, + { + "name": "choudoufu-workbench", + "description": "The live-mv workbench's own four estates, copied out of ../choudoufu (not out of behold's bundled copy, which drifts): the monolith's 21 resources applied to a scratch floci and three team estates of 7 re-declaring them, served composed with --env live — 42 cards, the monolith green and every taggable team card reading `owned by tlmig-sample-monolith`. These four declare their estate the older way, a `live { … }` block in main.tf. The fixture carries no carve.json and no cross-estate data source — `behold demo choudoufu-estate` is where those live. About 50s cold. Docker + choudoufu; provider download on first init. scripts/down.sh in the target removes the container.", + "source": "local", + "path": "../choudoufu/examples/live-mv-workbench/tests/fixtures/sample-run/estates", + "requires": [ + "docker", + "choudoufu" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/choudoufu-workbench/up.sh\"", + "serve": { + "env": "live", + "dirs": [ + "tlmig-sample-monolith", + "tlmig-sample-team-a", + "tlmig-sample-team-b", + "tlmig-sample-team-c" + ], + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4651", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "choudoufu-cohort-s3", + "description": "choudoufu's `s3` cohort, rendered fresh by ../choudoufu's tools/estate-gen into the target — an estate declared ONLY by its `estate.chdf.hcl` sidecar, which is what #387 taught behold's probe to read. Scratch floci, `choudoufu init`, then `apply` best effort: floci answers S3 Control tag reads on a hostname that does not resolve, so the apply stops and all 6 cards serve honestly unbound. About 35s cold. Docker + choudoufu + go + terraform (estate-gen runs `terraform init` for the provider schemas). scripts/down.sh in the target removes the container.", + "source": "local", + "requires": [ + "docker", + "choudoufu", + "go", + "terraform" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/choudoufu-cohort-s3/up.sh\"", + "serve": { + "env": "live", + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4652", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "choudoufu-cohort-iam-ecr", + "description": "choudoufu's `iam-ecr` cohort, rendered fresh by ../choudoufu's tools/estate-gen into the target — a sidecar-declared estate again, and the one cohort floci implements end to end (live/cohort-acceptance.json), so this one applies clean: 6 resources, 6 green cards, about 35s cold. Docker + choudoufu + go + terraform. scripts/down.sh in the target removes the container.", + "source": "local", + "requires": [ + "docker", + "choudoufu", + "go", + "terraform" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/choudoufu-cohort-iam-ecr/up.sh\"", + "serve": { + "env": "live", + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4653", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "choudoufu-cohort-ec2-networking", + "description": "choudoufu's `ec2-networking` cohort, rendered fresh by ../choudoufu's tools/estate-gen into the target — 49 resources, the widest of the three cohorts, and what a big sidecar-declared roster looks like. floci does not implement it (live/cohort-acceptance.json records it failing at apply; here it stops on a transit gateway call floci refuses), so the apply is best effort and all 49 cards serve declared and unbound. About 30s cold. Docker + choudoufu + go + terraform. scripts/down.sh in the target removes the container.", + "source": "local", + "requires": [ + "docker", + "choudoufu", + "go", + "terraform" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/choudoufu-cohort-ec2-networking/up.sh\"", + "serve": { + "env": "live", + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4654", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "terralith-1", + "description": "../choudoufu's tools/terralith-gen at scale 1, rendered into the target and given the `estate.chdf.hcl` sidecar the generator deliberately omits: 79 resources across network, IAM, ECS, DNS and a module tier, applied by choudoufu against a scratch floci from nothing, so all 79 cards serve bound. About 50s cold. The small end of the scale knob terralith-4 turns up. Docker + choudoufu + go. scripts/down.sh in the target removes the container.", + "source": "local", + "requires": [ + "docker", + "choudoufu", + "go" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/terralith-1/up.sh\"", + "serve": { + "env": "live", + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4655", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "terralith-4", + "description": "The same terralith at scale 4 — 301 resources at this generator, the same proportions — applied by choudoufu against a scratch floci and served with --env live: 301 nodes, every one of them green. The estate behold is sized against; #386 wrote 205, which is what an older composition rendered. About 105s cold, most of it the apply. Docker + choudoufu + go. scripts/down.sh in the target removes the container.", + "source": "local", + "requires": [ + "docker", + "choudoufu", + "go" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/terralith-4/up.sh\"", + "serve": { + "env": "live", + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4656", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "terralith-4-adopt", + "description": "The terralith at scale 4 again, but applied by STOCK terraform first — the generator's versions.tf has no live block, which is the point — so nothing wears a marker tag and there is no record store. The sidecar and `choudoufu init` come after, over the state terraform wrote, and behold serves 301 cards none of which this estate owns: 85 UNOWNED, 84 bound by derived identity alone (the name the configuration states, not a marker), 132 neutral — waiting on discovery or an unavailable parent. One `choudoufu live-import -state=terraform.tfstate -estate=behold-terralith-4-adopt -approve`, run by hand in the target — the up script prints it with the endpoint — stamps 137 of them and records all 301 identities; reload and all 301 are green. behold never runs that write. About 115s cold plus 23s for the import. Docker + choudoufu + go + terraform. scripts/down.sh in the target removes the container.", + "source": "local", + "requires": [ + "docker", + "choudoufu", + "go", + "terraform" + ], + "setup": "bash \"$BEHOLD_WORKBENCH_DIR/workbench/terralith-4-adopt/up.sh\"", + "serve": { + "env": "live", + "spawnEnv": { + "AWS_ENDPOINT_URL": "http://127.0.0.1:4657", + "AWS_ACCESS_KEY_ID": "test", + "AWS_SECRET_ACCESS_KEY": "test", + "AWS_REGION": "us-east-1" + } + } + }, + { + "name": "waterpark", + "description": "water park's own AWS access estate, ../waterpark/access, served IN PLACE as a bare Terraform directory (#384): five roots as boxes — envs/prod, identity, github, baseline, satellites/waterpark-runner — 58 cards at detail 2, modules/persona left out as a called module, and no file written under the estate. Needs chant's terraform lexicon beside behold, which behold does not install: npm install --no-save @intentius/chant-lexicon-terraform @cdktf/hcl2json (the refusal names the exact line). No cloud, no Docker.", + "source": "local", + "path": "../waterpark/access", + "inPlace": true, + "requires": [], + "serve": {} + } + ] +} diff --git a/workbench/choudoufu-cohort-ec2-networking/up.sh b/workbench/choudoufu-cohort-ec2-networking/up.sh new file mode 100755 index 0000000..34c7fa6 --- /dev/null +++ b/workbench/choudoufu-cohort-ec2-networking/up.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# behold workbench entry `choudoufu-cohort-ec2-networking` (#389): choudoufu's +# `ec2-networking` cohort — 49 resources, the widest of the three cohorts here — +# rendered by tools/estate-gen into the demo target, `choudoufu init` + `apply` +# against a scratch floci, served with --env live. floci does not implement this +# cohort (live/cohort-acceptance.json records it failing at apply), so the apply +# is best effort and the estate serves declared: the roster, the sidecar and the +# rungs are the point, not a green overlay. workbench/lib/cohort.sh is the body; +# this file names the cohort and the port. Runs with cwd = the demo target. +set -euo pipefail +LIB="$(cd "$(dirname "${BASH_SOURCE[0]}")/../lib" && pwd)" +# shellcheck source=../lib/common.sh +. "$LIB/common.sh" +# shellcheck source=../lib/cohort.sh +. "$LIB/cohort.sh" + +wb_cohort_up ec2-networking 4654 diff --git a/workbench/choudoufu-cohort-iam-ecr/up.sh b/workbench/choudoufu-cohort-iam-ecr/up.sh new file mode 100755 index 0000000..7b6f16b --- /dev/null +++ b/workbench/choudoufu-cohort-iam-ecr/up.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# behold workbench entry `choudoufu-cohort-iam-ecr` (#389): choudoufu's +# `iam-ecr` cohort, rendered by tools/estate-gen into the demo target, +# `choudoufu init` + `apply` against a scratch floci, served with --env live. +# The one cohort behold is developed against that floci implements end to end +# (live/cohort-acceptance.json), so this is the green one. The estate is +# declared in `estate.chdf.hcl`, choudoufu's leading form and the reason #387 +# taught behold's probe to read it. workbench/lib/cohort.sh is the body; this +# file names the cohort and the port. Runs with cwd = the demo target. +set -euo pipefail +LIB="$(cd "$(dirname "${BASH_SOURCE[0]}")/../lib" && pwd)" +# shellcheck source=../lib/common.sh +. "$LIB/common.sh" +# shellcheck source=../lib/cohort.sh +. "$LIB/cohort.sh" + +wb_cohort_up iam-ecr 4653 diff --git a/workbench/choudoufu-cohort-s3/up.sh b/workbench/choudoufu-cohort-s3/up.sh new file mode 100755 index 0000000..e7c5a47 --- /dev/null +++ b/workbench/choudoufu-cohort-s3/up.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# behold workbench entry `choudoufu-cohort-s3` (#389): choudoufu's `s3` cohort, +# rendered by tools/estate-gen into the demo target, `choudoufu init` + `apply` +# against a scratch floci, served with --env live. The estate is declared in +# `estate.chdf.hcl`, choudoufu's leading form and the reason #387 taught +# behold's probe to read it. workbench/lib/cohort.sh is the body; this file +# names the cohort and the port. Runs with cwd = the demo target. +set -euo pipefail +LIB="$(cd "$(dirname "${BASH_SOURCE[0]}")/../lib" && pwd)" +# shellcheck source=../lib/common.sh +. "$LIB/common.sh" +# shellcheck source=../lib/cohort.sh +. "$LIB/cohort.sh" + +wb_cohort_up s3 4652 diff --git a/workbench/choudoufu-workbench/up.sh b/workbench/choudoufu-workbench/up.sh new file mode 100755 index 0000000..4975586 --- /dev/null +++ b/workbench/choudoufu-workbench/up.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# behold workbench entry `choudoufu-workbench` (#389): the live-mv workbench's +# own four estates, copied out of ../choudoufu on every fresh target rather than +# out of behold's `example-choudoufu-estate`, which is a copy that drifts. +# +# Same shape as the bundled demo's scripts/choudoufu-up.sh: a scratch floci, +# `choudoufu init` in each of the four copies (behold's own target, never a +# served checkout — #366's rule), and `apply` in the monolith, whose 21 +# resources the three team estates re-declare. The teams stay unapplied on +# purpose: their resources are the monolith's, so every team card serves +# `owned by tlmig-sample-monolith`. +# +# Two things the bundled demo has that the fixture does not, and this script +# does not invent: a `carve.json` move plan, and team-a's extra VPC with the +# cross-estate data source that reads it. The fixture is four `main.tf` files +# and nothing else; `behold demo choudoufu-estate` is where those two live. +# +# These estates declare their estate the OTHER way — a `live { estate = "…" }` +# block inside terraform{}, not the `estate.chdf.hcl` sidecar the cohorts carry. +# Both forms serve, which is the shape of #387's probe. +# +# Runs with cwd = the demo target (the copied estates). +set -euo pipefail +LIB="$(cd "$(dirname "${BASH_SOURCE[0]}")/../lib" && pwd)" +# shellcheck source=../lib/common.sh +. "$LIB/common.sh" + +PORT=4651 +ESTATES="tlmig-sample-monolith tlmig-sample-team-a tlmig-sample-team-b tlmig-sample-team-c" +CONTAINER="$(wb_container)" + +wb_require_choudoufu +for e in $ESTATES; do + [ -f "$e/main.tf" ] || wb_die "$PWD/$e/main.tf is missing — the copy of ../choudoufu's live-mv fixture is incomplete." +done + +wb_floci_up "$CONTAINER" "$PORT" +wb_write_down "$CONTAINER" +wb_export_aws "$PORT" + +for e in $ESTATES; do + wb_say "choudoufu init in ${e} (provider schemas — what makes the rungs real)" + (cd "$e" && "$(wb_choudoufu)" init -input=false -no-color >/dev/null) || wb_die "choudoufu init failed in $PWD/$e" +done + +wb_say "choudoufu apply in tlmig-sample-monolith (the terralith: three teams' IAM and log groups in one estate)" +(cd tlmig-sample-monolith && "$(wb_choudoufu)" apply -auto-approve -input=false -no-color | tail -3) || + wb_die "choudoufu apply failed in $PWD/tlmig-sample-monolith — \`docker logs ${CONTAINER}\` and \`bash scripts/down.sh\`." + +for e in $ESTATES; do + wb_say "${e}: $( (cd "$e" && wb_instance_count) ) instances in live-check's roster" +done + +cat < +# +# What it does, in the demo target (empty on a first run — these entries name +# no `path`, so their setup IS the source, #386's third decision): +# +# 1. renders the cohort with choudoufu's own tools/estate-gen, +# 2. boots the entry's scratch floci, +# 3. `choudoufu init`, so the rungs come from the provider's schemas, +# 4. `choudoufu apply`, best effort — see the note below, +# 5. leaves scripts/down.sh behind. +# +# The cohort arrives carrying `estate.chdf.hcl`, choudoufu's leading estate +# declaration and the form behold's probe learned to read in #387. That sidecar +# is the whole point of these three entries: a cohort has no `live` block in any +# *.tf, so before M1 every one of them fell through as "no chant.config.ts here". +# +# Why estate-gen is asked for `-all`. `-cohort ` alone resolves its types +# through live/mapping.json by CFN SERVICE, which answers for "s3" (16 types) +# but not for a registry cohort whose name is not a service — `-cohort iam-ecr` +# fails with "no admitted type maps to a \"iam-ecr\" CFN service". `-all` renders +# every cohort in internal/live/cohorts, which is the registry these entries +# mean, off ONE provider-schema acquisition: measured at 13s for all 31 into +# 1.4MB. We keep the one we came for and drop the rest, rather than pin a copy +# of the registry's type lists here that would drift. +# +# Why apply is best effort. choudoufu measures this itself, in +# live/cohort-acceptance.json: 5 of 31 cohorts apply clean against floci; the +# rest hit an operation the emulator does not implement. A cohort that cannot +# apply still serves — as a choudoufu member with its sidecar read, its roster +# drawn and its cards honestly unbound — so the script prints exactly what +# refused and carries on instead of failing the demo. + +wb_cohort_up() { + local cohort="$1" port="$2" + local container render + container="$(wb_container)" + + wb_require_bin go + wb_require_bin terraform # estate-gen runs `terraform init` for the schemas + wb_require_choudoufu + + if [ -f versions.tf ]; then + wb_say "reusing the ${cohort} cohort already rendered here — nothing re-rendered over live state" + else + wb_say "rendering the ${cohort} cohort with choudoufu's tools/estate-gen" + render="$PWD/.behold-cohort-render" + rm -rf "$render" + env -u PWD go -C "$(wb_choudoufu_checkout)" run ./tools/estate-gen -all -out "$render" >/dev/null || + wb_die "estate-gen -all failed" + [ -d "$render/$cohort" ] || wb_die "estate-gen rendered no \"$cohort\" cohort — internal/live/cohorts no longer lists that name?" + cp -R "$render/$cohort/." . + rm -rf "$render" + [ -f estate.chdf.hcl ] || wb_die "the ${cohort} cohort has no estate.chdf.hcl — the sidecar is what makes it a choudoufu member" + wb_say "estate $(sed -n 's/^estate *= *"\(.*\)"/\1/p' estate.chdf.hcl), declared in estate.chdf.hcl (no live block in any *.tf)" + fi + + wb_floci_up "$container" "$port" + wb_write_down "$container" + wb_export_aws "$port" + + wb_say "choudoufu init (provider schemas — what makes the rungs real)" + "$(wb_choudoufu)" init -input=false -no-color >/dev/null || wb_die "choudoufu init failed in $PWD" + + wb_say "choudoufu apply (best effort — floci implements 5 of 31 cohorts, live/cohort-acceptance.json)" + if "$(wb_choudoufu)" apply -auto-approve -input=false -no-color 2>&1 | tail -12; then + : + fi + + wb_say "$(wb_instance_count) instances in live-check's roster" + local refused + refused="$(wb_refused_types)" + if [ -n "$refused" ]; then + wb_say "live-check still refuses, by type (count, type):" + echo "$refused" | sed 's/^/ /' + fi + echo "behold workbench ${BEHOLD_DEMO_NAME}: up. behold serves it next with --env live." +} diff --git a/workbench/lib/common.sh b/workbench/lib/common.sh new file mode 100644 index 0000000..8ce8ba1 --- /dev/null +++ b/workbench/lib/common.sh @@ -0,0 +1,172 @@ +#!/usr/bin/env bash +# Shared helpers for the workbench up scripts (#389, M3 of #386). +# +# Sourced, never run. `behold demo ` runs an entry's up script with cwd +# set to the demo target and two variables in the environment: BEHOLD_DEMO_NAME +# (the catalog entry) and BEHOLD_WORKBENCH_DIR (the directory workbench.json was +# read from, which is this repo's root). The sibling checkouts hang off the +# second — `$BEHOLD_WORKBENCH_DIR/../choudoufu` — so a script means the same +# directory from whatever terminal behold was started in. +# +# Scratch discipline (src/scratch.ts) is asserted here, in bash, because these +# scripts are the boot site: the container is `behold-wb-`, its host port +# is the entry's own and never a shared emulator's, and only the down.sh this +# file writes into the target removes it. + +wb_die() { + echo "behold workbench: $*" >&2 + exit 1 +} + +wb_say() { echo "→ $*"; } + +# The repo root: what behold hands us, or this file's own ../.. when a person +# runs an up script by hand. +wb_root() { + if [ -n "${BEHOLD_WORKBENCH_DIR:-}" ]; then + echo "$BEHOLD_WORKBENCH_DIR" + else + (cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) + fi +} + +# The choudoufu checkout these entries read their generators and image pin from. +wb_choudoufu_checkout() { + local dir + dir="$(wb_root)/../choudoufu" + [ -d "$dir" ] || wb_die "../choudoufu is not checked out (looked in $dir)" + (cd "$dir" && pwd) +} + +# #388's fourth decision: the same binary behold itself spawns. The Homebrew +# release is below behold's 0.16.0 floor, so the build that works is one +# somebody built from main and named with CHOUDOUFU_BIN. +wb_choudoufu() { echo "${CHOUDOUFU_BIN:-choudoufu}"; } + +wb_require_bin() { + command -v "$1" >/dev/null 2>&1 || wb_die "$1 is not on PATH" +} + +wb_require_choudoufu() { + local bin + bin="$(wb_choudoufu)" + command -v "$bin" >/dev/null 2>&1 || + wb_die "$bin is not on PATH — https://github.com/INTENTIUS/choudoufu (0.16.0 or newer), or set CHOUDOUFU_BIN to a build from main." +} + +wb_require_docker() { + wb_require_bin docker + docker info >/dev/null 2>&1 || wb_die "Docker is not running — start Docker and re-run." +} + +# The emulator image the choudoufu checkout pins. Read at run time, never +# copied here: a pin behold carried separately would be a second pin. +wb_floci_image() { + local pin + pin="$(wb_choudoufu_checkout)/live/floci-image" + [ -f "$pin" ] || wb_die "no floci pin at $pin" + tr -d '[:space:]' <"$pin" +} + +# The scratch name for the entry behold is loading. +wb_container() { + [ -n "${BEHOLD_DEMO_NAME:-}" ] || wb_die "BEHOLD_DEMO_NAME is unset — this script runs as a behold demo setup." + echo "behold-wb-${BEHOLD_DEMO_NAME}" +} + +# src/scratch.ts in bash. A workbench emulator is named behold-wb-, +# which no protected name can match, and it never binds a shared emulator's +# port. Both are checked before the docker run that would create the thing. +wb_assert_scratch() { + local name="$1" port="$2" + case "$name" in + behold-wb-?*) ;; + *) wb_die "scratch discipline: a workbench emulator is named behold-wb-, got '$name'" ;; + esac + case " 4566 4577 4588 " in + *" $port "*) wb_die "scratch discipline: port $port belongs to a shared emulator — a scratch substrate gets its own" ;; + esac +} + +# Boot the entry's floci, or reuse the one already running under our own name. +# +# Running: reused, so a second `behold demo ` restarts the serve against +# the estate it already applied. Present but stopped: refused, because its +# emulated cloud is gone while the record store and state on disk still claim +# those resources — down.sh, and re-run. +wb_floci_up() { + local name="$1" port="$2" image + wb_assert_scratch "$name" "$port" + wb_require_docker + if docker ps --format '{{.Names}}' | grep -qx "$name"; then + wb_say "reusing the scratch floci already running: ${name} on 127.0.0.1:${port}" + return 0 + fi + if docker ps -a --format '{{.Names}}' | grep -qx "$name"; then + wb_die "container ${name} exists but is not running — its emulated cloud is gone while this target's records still name those resources. \`bash scripts/down.sh\` in $PWD, then re-run." + fi + image="$(wb_floci_image)" + wb_say "floci on 127.0.0.1:${port} (${name})" + wb_say "image ${image} — the pin in ../choudoufu/live/floci-image" + docker run -d --name "$name" -p "127.0.0.1:${port}:4566" "$image" >/dev/null + local i + for i in $(seq 1 90); do + curl -fs -o /dev/null "http://127.0.0.1:${port}/" && break + sleep 1 + done + curl -fs -o /dev/null "http://127.0.0.1:${port}/" || + wb_die "floci on 127.0.0.1:${port} never answered — \`docker logs ${name}\`, then \`bash scripts/down.sh\`." +} + +# The credentials floci accepts, and the endpoint this entry's estates talk to. +# The same four the entry's `serve.spawnEnv` gives behold's own choudoufu reads. +wb_export_aws() { + export AWS_ENDPOINT_URL="http://127.0.0.1:$1" + export AWS_ACCESS_KEY_ID=test + export AWS_SECRET_ACCESS_KEY=test + export AWS_REGION=us-east-1 +} + +# Write the matching down script into the target (#386's third decision: the up +# script leaves behind the one that tears its scratch down). It removes OUR +# container by name and nothing else; the rendered estate stays, because it is +# yours to edit. +wb_write_down() { + local name="$1" + mkdir -p scripts + cat >scripts/down.sh </dev/null 2>&1 || true +echo "behold workbench: ${name} removed." +EOF + chmod +x scripts/down.sh +} + +# The types `choudoufu live-check -json` refuses in this directory, one +# "" line each. node, not jq: behold runs on node, so every +# machine that can run a demo can run this. +wb_refused_types() { + "$(wb_choudoufu)" live-check -json . 2>/dev/null | node -e ' + let s = ""; + process.stdin.on("data", (d) => (s += d)).on("end", () => { + let doc; + try { doc = JSON.parse(s); } catch { return; } + const counts = {}; + for (const i of doc.instances ?? []) if (i.refused) counts[i.type ?? "?"] = (counts[i.type ?? "?"] ?? 0) + 1; + for (const [t, n] of Object.entries(counts).sort()) console.log(`${n}\t${t}`); + });' +} + +# How many instances `live-check` sees here — the roster behold's estate box +# draws a card per. +wb_instance_count() { + "$(wb_choudoufu)" live-check -json . 2>/dev/null | node -e ' + let s = ""; + process.stdin.on("data", (d) => (s += d)).on("end", () => { + try { console.log((JSON.parse(s).instances ?? []).length); } catch { console.log(0); } + });' +} diff --git a/workbench/lib/terralith.sh b/workbench/lib/terralith.sh new file mode 100644 index 0000000..dc91fbf --- /dev/null +++ b/workbench/lib/terralith.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# The body of the terralith entries (#389). Sourced by workbench/terralith-*/up.sh. +# +# wb_terralith_render — the generated estate + its sidecar +# wb_terralith_greenfield — choudoufu from nothing (terralith-1, terralith-4) +# wb_terralith_adopt — stock terraform first, then choudoufu (terralith-4-adopt) +# +# tools/terralith-gen renders one estate whose composition is a function of +# -scale: 79 resources at 1, 205 at 4, the same proportions at both. Its +# versions.tf deliberately carries NO live block — it is a stock Terraform root +# module, which is what makes the adopt entry's stock `terraform apply` the +# honest starting point it is. The `estate.chdf.hcl` sidecar is what these +# scripts add, and it is the only file they write into the estate. +# +# The route53 assertion. Under choudoufu's built-in identity table +# `aws_route53_record` has no computable identity, so `live-check` refuses every +# one of the terralith's ten record sites. After `init` the provider's own +# schemas are in hand and it must not refuse them any more — that is the whole +# reason `choudoufu init` runs here rather than being left to the person, and it +# is asserted below so a regression in the rung fails the demo instead of +# quietly serving ten grey cards. + +# Render the estate, once. A target that already carries main.tf is reused: a +# second `behold demo ` restarts the serve, it does not re-render over +# live state. +wb_terralith_render() { + local scale="$1" estate="$2" + wb_require_bin go + if [ -f main.tf ]; then + wb_say "reusing the terralith already rendered here (scale ${scale}) — nothing re-rendered over live state" + else + wb_say "rendering the terralith at scale ${scale} with choudoufu's tools/terralith-gen" + env -u PWD go -C "$(wb_choudoufu_checkout)" run ./tools/terralith-gen -scale "$scale" -out "$PWD" || + wb_die "terralith-gen -scale ${scale} failed" + fi + if [ -f estate.chdf.hcl ]; then + wb_say "estate.chdf.hcl is already here — left as it is" + else + wb_say "writing estate.chdf.hcl: estate = ${estate} (the generator writes no live block; the sidecar is the leading form)" + printf 'estate = "%s"\n\nrecord_store "local" {\n path = ".tofu-records"\n}\n' "$estate" >estate.chdf.hcl + fi +} + +# After init the provider's schemas are in hand, so the type choudoufu's +# built-in table cannot identify must no longer be refused. +wb_terralith_assert_rung() { + local refused + refused="$(wb_refused_types)" + if echo "$refused" | grep -q 'aws_route53_record'; then + echo "$refused" | sed 's/^/ /' >&2 + wb_die "live-check still refuses aws_route53_record after \`choudoufu init\` — the rung should come from the provider's schema, not the built-in table." + fi + wb_say "live-check refuses no aws_route53_record after init (the rung is the provider's, not the built-in table's)" + if [ -n "$refused" ]; then + wb_say "still refused, by type (count, type):" + echo "$refused" | sed 's/^/ /' + fi +} + +# The greenfield stage: choudoufu owns the estate from the first apply, so every +# resource is marked and every card paints bound. +wb_terralith_greenfield() { + local scale="$1" port="$2" container + container="$(wb_container)" + wb_require_choudoufu + wb_terralith_render "$scale" "behold-terralith-${scale}" + wb_floci_up "$container" "$port" + wb_write_down "$container" + wb_export_aws "$port" + + wb_say "choudoufu init (provider schemas — what makes the rungs real)" + "$(wb_choudoufu)" init -input=false -no-color >/dev/null || wb_die "choudoufu init failed in $PWD" + wb_terralith_assert_rung + + wb_say "choudoufu apply -auto-approve (the terralith at scale ${scale}, into the scratch floci)" + "$(wb_choudoufu)" apply -auto-approve -input=false -no-color | tail -3 || + wb_die "choudoufu apply failed in $PWD — \`docker logs ${container}\` and \`bash scripts/down.sh\`." + + wb_say "$(wb_instance_count) instances in live-check's roster" + echo "behold workbench ${BEHOLD_DEMO_NAME}: up. behold serves it next with --env live; every card should be bound." +} + +# The adopted stage: STOCK terraform applies the estate first — no live block +# anywhere, no marker tags, no record store, which is exactly the estate a team +# has before it adopts choudoufu. The sidecar and `choudoufu init` come after, +# and the state file stays. Served, every card reads unowned until the person +# runs the one `live-import` line below, by hand, in the target. behold never +# runs the write: that is the bundled choudoufu demo's pattern (#372). +wb_terralith_adopt() { + local scale="$1" port="$2" container + container="$(wb_container)" + wb_require_bin terraform + wb_require_choudoufu + wb_terralith_render "$scale" "behold-terralith-${scale}-adopt" + wb_floci_up "$container" "$port" + wb_write_down "$container" + wb_export_aws "$port" + + if [ -f terraform.tfstate ]; then + wb_say "terraform.tfstate is already here — the stock apply already ran, nothing re-applied" + else + wb_say "STOCK terraform init (no live block in versions.tf — that is the point of this entry)" + terraform init -input=false -no-color >/dev/null || wb_die "terraform init failed in $PWD" + wb_say "STOCK terraform apply -auto-approve (the terralith at scale ${scale}, unmarked, into the scratch floci)" + terraform apply -auto-approve -input=false -no-color | tail -3 || + wb_die "terraform apply failed in $PWD — \`docker logs ${container}\` and \`bash scripts/down.sh\`." + [ -f terraform.tfstate ] || wb_die "terraform wrote no terraform.tfstate — live-import has nothing to read." + fi + + wb_say "choudoufu init over the stock state (the sidecar names the estate; the state file stays where it is)" + "$(wb_choudoufu)" init -input=false -no-color >/dev/null || wb_die "choudoufu init failed in $PWD" + wb_terralith_assert_rung + wb_say "$(wb_instance_count) instances in live-check's roster, none of them owned yet" + + cat <