You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This spec defines the visual-verification gate: the measuring instrument that turns "does the custom boss body render correctly in-game?" into an objective, reproducible PASS / FAIL / INCONCLUSIVE verdict over a FULL-FRAME screenshot, plus an offline rasterizer preview that predicts the verdict before a game launch is spent. It is built first in epic #65 so the later approach-spike specs (AssetBundle / runtime-glTF / procedural) can be judged against the same gate instead of by eyeballing hand-picked zoomed crops, which is the failure mode that produced repeated false "it's fixed" claims. The driving case is the Hollow Mire boss "Mudwretch Foreman" (ftkmf_mudwretch_foreman), a reskin of the vanilla enTroll01 troll. The gate is offline tooling plus the existing agent harness; it adds no public Content.* surface, no Core/ change, and no networked or serialized state.
Problem Statement
A custom boss body can be loaded and skinned at runtime, but the prior sessions could not reliably tell whether the result was correct. The body has variously rendered as a shattered triangle cloud, off-center, mis-scaled, or hunched, while a cropped-and-zoomed screenshot was read as a pass. The root causes were:
Subjective judgement. Pass/fail came from a human looking at a hand-picked crop. A full-frame dim shot hid the shatter; a flattering crop hid the placement error.
No reference for "correct." There was no objective anchor for "centered" or "correctly scaled" in the actual combat scene.
No pre-flight. Broken assets were discovered only after a full game launch, burning time.
The framework already ships the pieces to fix this but has not assembled them into a gate:
harness/ftk_mcp_server.py exposes ftk_observe / ftk_act(action,args) / ftk_wait_for(predicate) / ftk_screenshot(save_path). ftk_screenshot writes a FULL-FRAME PNG only (no crop, no analysis).
harness/README.md documents the deterministic boss scene and the lit-diorama / dismiss_message constraints.
tools/ai-model-pipeline/03_render_preview.py renders a .glb offline (Blender Workbench) but produces no metrics or verdict.
The in-game bridge (FTKModFramework/Agent/*) is env-gated by FTK_AGENT_BRIDGE=1, loopback-bound, and per-action single-player-checked, so a test-only capture path is already co-op-safe.
The gate's job is to assemble these into one objective, reproducible verdict.
Goals and Success Metrics
Goal
Metric
Target
Replace subjective crop judgement with an objective verdict
A machine-readable verdict.json is produced from the FULL frame for every capture
100% of captures produce a verdict; no PASS is derivable from a crop alone
Detect the shatter automatically
A known shard-cloud frame is classified FAIL on the connected criterion
Synthetic/known shatter -> connected: fail every run
Anchor placement and scale to reality
"Centered" and "correctly scaled" thresholds are computed relative to a captured vanilla-troll baseline
Thresholds reference baseline.json, not guessed pixel constants
Predict in-game result before launch
The offline preview verdict and the in-game verdict agree on the four mechanical criteria for the same asset
Agreement on a known-good and a known-broken asset; offline FAIL blocks the launch
Never report a false confident verdict
Unlit / black / boss-not-in-frame captures are classified INCONCLUSIVE
Those cases never return PASS or FAIL
Stay visual-only and deterministic
No new networked/serialized state; one env-gated C# branch that is byte-identical when off; build stays green with SELF-TEST PASS
Verified
Functional Requirements
FR-1: Deterministic boss-scene capture driver
Description: A harness driver runs the documented Hollow Mire boss sequence end to end and captures a full-frame PNG at the only moment the diorama is lit, with no manual steps. The sequence is start_run HollowMire -> enter_dungeon FloodedCrypt -> dungeon_regen -> cleared_room x10 (boss is level 1 room 4) -> dungeon_encounter -> dungeon_scroll_complete -> wait for combat.heroTurnReady==true -> dismiss_message -> ftk_screenshot. It must use dismiss_message, never advance (which ends the turn and blacks out the diorama).
Acceptance Criteria:
Running the driver against a launched, bridge-enabled game reaches phase==combat with combat.heroTurnReady==true and writes a full-frame PNG to a known path.
The driver clears the re-firing boss-intro dialog with dismiss_message and never calls advance before the screenshot.
On ANY step failure (503, no-session, timeout, or a mid-sequence failure such as an empty room list after dungeon_regen), the driver emits an INCONCLUSIVE result naming the failing step rather than a synthetic PASS/FAIL, and never throws into the game.
Two consecutive runs against the same build produce the boss in-frame and lit (no all-black frame).
Priority: P0 Layer: Harness
FR-2: Vanilla-troll baseline capture and env toggle
Description: The gate can render the boss with the STOCK troll chassis (custom body disabled) in the same scene, so the vanilla-troll silhouette anchors the placement and scale thresholds. This is driven by an environment variable FTK_BASELINE_STOCK_BODY=1 read in RealmBossAdventure.BuildBoss: when set, the custom glb mesh/texture assignment is skipped and the boss renders as the unmodified trollCaveA chassis (white tint, scale 1, no width boost, no swamp aura). When the variable is unset or not equal to 1, BuildBoss produces the identical visual configuration it does today and the env read has no effect on shipped/default behavior.
Acceptance Criteria:
With FTK_BASELINE_STOCK_BODY=1, the boss renders as the stock troll chassis (no custom glb body, no recolor, no scale change).
With the variable unset or not equal to 1, the custom body wiring is unchanged and the default runtime behavior is identical to the current shipped path (the env read has no effect when off).
Running FR-1's driver under FTK_BASELINE_STOCK_BODY=1 captures a baseline frame and produces baseline.json containing the stock-troll silhouette bbox, centroid, height, and aspect.
baseline.json is produced by the SAME segmentation front-end used for candidate analysis (FR-3), so baseline and candidate are measured identically.
FR-3: Screenshot analyzer with objective mechanical criteria
Description: An offline Python analyzer takes a full-frame PNG (plus baseline.json when present) and computes a single binary foreground mask via a luminance-plus-saturation threshold over a center region-of-interest band (excluding the bottom UI / health-bar strip and top vignette), then derives all mechanical metrics from that one mask. It emits per-criterion PASS/FAIL, the measured values, an annotated full-frame image (bbox, centroid, ROI overlaid), and a machine-readable verdict.json.
Acceptance Criteria:
The analyzer segments the lit-diorama boss silhouette without any ML model, using only analysis libraries already present in .venv-3dgen (numpy, scipy, Pillow). (The "no ML / numpy+scipy+Pillow only" constraint applies to this analysis stage; the offline RENDER stage in FR-5 reuses Blender.)
The segmentation and the four criteria are implemented as a SINGLE shared function, analyze(frame_png, baseline) -> verdict, with explicit inputs and no hidden state, imported by both the in-game analyzer (this FR) and the offline preview (FR-5), so "same criteria" is enforced structurally (one function, two callers) rather than by convention.
It computes and records: connected-component count, largest-component fill fraction, silhouette bbox, normalized centroid, bbox height ratio (vs baseline), and an upright proxy.
It writes a verdict.json conforming to the schema in Technical Architecture, plus an annotated full-frame PNG.
Given a known-good capture all four mechanical criteria report PASS; given a known shard-cloud capture the connected criterion reports FAIL.
Priority: P0 Layer: Tooling
FR-4: Objective pass criteria with baseline-anchored thresholds
Description: The four mechanical criteria are defined with explicit, baseline-anchored thresholds. connected is baseline-free; upright, centered, and scaled are expressed relative to baseline.json. Critical anchoring fact: the baseline is captured with FTK_BASELINE_STOCK_BODY=1, which forces the stock chassis at scale 1, while the custom boss renders at BossBodyScale = 1.4. Therefore the expected candidate-to-baseline HEIGHT ratio is approximately 1.4 (not 1.0), and the scale band is centered on ~1.4, not on 1.0. The exact center and tolerance are finalized against the captured baseline in Phase 3.
Acceptance Criteria:
connected: PASS when the largest-component fill fraction is at or above a documented floor and the count of components larger than a documented noise floor is at or below a documented small integer (to allow the body plus a baked lantern). The exact numbers are recorded in the spec's threshold table and in docs/AI-MODEL-PIPELINE.md.
upright: PASS when the largest component's major-axis angle is within a documented tolerance of vertical AND the bbox aspect proxy is at or above a documented fraction of the baseline aspect.
centered: PASS when the normalized centroid is within a documented fraction of the baseline centroid in x and y.
scaled: PASS when the height ratio vs baseline falls inside a documented band centered on ~1.4 (the BossBodyScale multiplier vs the scale-1 stock chassis), and the area ratio falls inside a documented band centered on ~1.4 squared. Whether the stock chassis itself receives any additional spawn-time scale is [UNVERIFIED -- needs game-decompile-analyst]; if it does, the expected ratio is recomputed accordingly in Phase 3.
Every threshold is a concrete number recorded in the verdict's threshold strings and in the docs; none is described with ambiguous language.
Priority: P0 Layer: Tooling
FR-5: Offline rasterizer preview that gates the launch
Description: The offline RENDER stage reuses the existing Blender Workbench rasterizer (extend 03_render_preview.py or fork to 03b_preview_metrics.py) to render the candidate .glb plus texture to a PNG. The ANALYSIS stage then runs the SAME shared analyze(...) function from FR-3 on that rendered PNG, emitting a preview_verdict.json. The offline mechanical result GATES the in-game launch: if the asset fails the offline mechanical checks (not a single coherent body, not upright, or an out-of-range bbox), the in-game capture is not run. This catches asset-side breakage offline; the in-game capture then catches runtime-reconstruction breakage (the shatter is a runtime-only bug that the offline render cannot reproduce). The preview camera and framing are an approximation of the in-game crypt camera, so preview-vs-in-game agreement is judged on the four NORMALIZED mechanical verdicts only, never on raw pixels or absolute resolution.
Acceptance Criteria:
The preview renders the candidate .glb plus texture (Blender) and produces preview_verdict.json using the same shared analyze(...) function as FR-3.
A well-formed coherent rig predicts mechanical PASS; a malformed/exploded .glb predicts mechanical FAIL.
When the preview mechanical verdict is FAIL, the single-command flow (FR-8) does not launch the in-game capture and reports the offline FAIL with its reasons.
The preview and in-game verdicts agree on the four NORMALIZED mechanical criteria for the same asset on at least one known-good and one known-broken example (agreement is on the normalized criteria, not on pixel dimensions).
Priority: P0 Layer: Tooling
FR-6: Full-frame verdict and recognizability crop
Description: The verdict is computed from the FULL frame. The analyzer additionally emits a defined zoomed crop of the boss region and records the frame dimensions and the foreground bbox within them. Recognizability ("reads as the intended mud/bog creature") is AGENT-JUDGED from that crop shown beside the offline reference render; it is recorded as a field but is never computed by an ML model and never blocks the four mechanical criteria.
Acceptance Criteria:
verdict.json records image_w, image_h, and the largest-component bbox within the full frame.
The analyzer emits a crop image at a defined region derived from the detected bbox, plus the path to the offline reference render for side-by-side judgement.
The recognizable field defaults to pending and is filled by the agent (pass/fail plus a one-line reason); the four mechanical criteria are decided without it.
No PASS verdict can be produced from a crop alone; the mechanical verdict always derives from the full-frame mask.
Priority: P0 Layer: Tooling
FR-7: INCONCLUSIVE handling
Description: Before any PASS/FAIL is decided, the analyzer detects degenerate captures and returns INCONCLUSIVE with a reason, so an unlit or empty frame never reads as a confident verdict in either direction.
Acceptance Criteria:
A black/unlit frame (mean luminance below a documented floor OR foreground fraction below a documented floor) returns verdict=inconclusive with reason unlit-or-empty-frame.
A boss-not-in-frame capture (largest component below a documented size floor, or centroid in a corner ROI) returns verdict=inconclusive with reason boss-not-in-frame.
A capture-time error (screenshot 503 / no session / timeout) returns verdict=inconclusive with a reason that names the cause.
INCONCLUSIVE is mutually exclusive with PASS and FAIL in the verdict.
Priority: P0 Layer: Tooling
FR-8: Single-command reproducible verdict
Description: One entry point runs the whole gate: offline preview (launch gate) -> in-game capture (if the preview passed) -> analyze -> print PASS/FAIL/INCONCLUSIVE with per-criterion reasons and the paths to the annotated image, crop, reference render, and verdict.json.
Acceptance Criteria:
A single invocation produces a verdict and the artifact paths, or reports the offline-gate FAIL without launching the game.
Re-running on the same build and the same captured frames yields the same mechanical verdict (deterministic given identical inputs).
The command prints, for each mechanical criterion, the measured value and the threshold applied.
Priority: P1 Layer: Tooling
Non-Functional Requirements
NFR-1: Reproducibility
Category: Reliability Requirement: Given the same build and the same captured frames, the analyzer produces an identical mechanical verdict. All placement/scale thresholds use normalized coordinates and areas; the verdict records image_w/image_h. The raw resolution-equality check applies ONLY between baseline.json and the in-game CANDIDATE capture (both produced by the same ftk_screenshot path at the same Screen size); preview-vs-in-game agreement is judged on normalized criteria only (per NFR-5), never on raw pixels or absolute resolution. Rationale: A gate that is not reproducible cannot adjudicate the later approach spikes; conflating the offline-render resolution with the in-game resolution would otherwise force a false INCONCLUSIVE on every comparison.
NFR-2: Co-op safety and visual-only
Category: Compatibility Requirement: The gate introduces no new networked or serialized state, no FTK_*DB row changes, and no IdAllocator calls. The only C# change is an env-gated branch that selects between two already-supported visual configurations. The capture path relies on the existing bridge, which is env-gated by FTK_AGENT_BRIDGE=1, loopback-bound, and per-action single-player-checked. Rationale: Epic #65 requires the whole pipeline to stay co-op-deterministic and save-safe.
NFR-3: net35 / shipped-behavior-identical
Category: Maintainability Requirement: The RealmBossAdventure.BuildBoss env branch targets net35 / Mono and, when FTK_BASELINE_STOCK_BODY is unset or not 1, leaves shipped behavior byte-identical. The build stays green and logs SELF-TEST PASS. Rationale: The framework targets net35 and must not regress shipped behavior for a test lever.
NFR-4: Dependency minimalism
Category: Maintainability Requirement: The tooling reuses the existing .venv-3dgen (numpy, scipy, Pillow, trimesh) and the local Blender; it adds no machine-learning dependency and no heavy new packages. Rationale: Avoid permanent review-debt and a model dependency for a single-creature proof case.
NFR-5: Resolution independence
Category: Reliability Requirement: All thresholds are expressed as fractions of frame dimensions or as ratios to the baseline, never as raw pixel constants tied to one resolution. This is what allows the Blender-rendered preview (its own resolution) and the in-game capture (Screen.width x Screen.height) to be compared on the same criteria. Rationale: Capture resolution varies by machine and window size, and the offline render resolution differs from the in-game resolution by design.
NFR-6: Divergence recorded, never auto-reconciled
Category: Reliability Requirement: The single-command output (FR-8) always prints BOTH the offline preview verdict and the in-game verdict side by side and never silently substitutes one for the other. A disagreement (e.g., preview PASS but in-game FAIL, the expected signature of a runtime-only shatter) is surfaced, not hidden. Rationale: This is the anti-eyeballing guarantee at the heart of the epic; a single reconciled number would re-introduce the ambiguity the gate exists to remove.
NFR-7: Baseline provenance
Category: Reliability Requirement: baseline.json records its capture resolution, the chassis name (trollCaveA), the framework version or commit, and that it was produced under FTK_BASELINE_STOCK_BODY=1, so a stale or mismatched baseline is detectable by the analyzer. Rationale: The placement/scale thresholds are entirely baseline-anchored; an unlabeled or stale baseline would silently invalidate every placement/scale verdict.
connected-component count; largest-component fill fraction = largest area / total foreground area
PASS if fill fraction at or above a documented floor (initial 0.85) AND count of components above a noise floor at or below a small integer (initial 2, for body plus lantern)
No
upright
PCA major-axis angle of largest component vs vertical; bbox aspect proxy = height/width
PASS if angle within a documented tolerance of vertical (initial 15 degrees) AND aspect proxy at or above a documented fraction of baseline aspect (initial 0.9)
Yes (aspect)
centered
normalized centroid (cx, cy) of largest component
PASS if abs(cx - baseline_cx) and abs(cy - baseline_cy) within documented fractions of frame size (initial 0.06 x, 0.08 y)
Yes
scaled
normalized bbox height and area
PASS if height ratio vs baseline within a documented band centered on ~1.4 (initial 1.15 to 1.70) AND area ratio within a documented band centered on ~1.4 squared (initial 1.30 to 2.90)
Yes
The scale band is centered on ~1.4 because the baseline is the scale-1 stock chassis and the boss renders at BossBodyScale = 1.4. Initial numbers are starting points; FR-4 requires them finalized and recorded against the captured baseline.json in implementation phase P3, and recomputed if the stock chassis is found to receive any spawn-time scale ([UNVERIFIED -- needs game-decompile-analyst]).
Key Types (verdict.json)
Type / Field
C# / JSON type
Description
Constraints
schema
string
Verdict schema version
e.g. ftk-visual-gate/1
verdict
string
Overall result
one of pass / fail / inconclusive
inconclusive_reason
string or null
Why no PASS/FAIL was possible
null unless verdict==inconclusive
source_image
string
Path to the analyzed full-frame PNG
non-empty
baseline_path
string or null
Path to the baseline.json used
null only for the baseline run
image_w, image_h
int
Frame dimensions
greater than 0
criteria.<name>.pass
bool
Per-criterion PASS
name in {connected, upright, centered, scaled}
criteria.<name>.value
number
Measured value
finite
criteria.<name>.threshold
string
Human-readable threshold applied
non-empty
metrics.fill_fraction
float
Largest-component fill fraction
0 to 1
metrics.component_count
int
Components above the noise floor
at or above 0
metrics.bbox
[int,int,int,int]
[x,y,w,h] of largest component
within frame
metrics.centroid_norm
[float,float]
normalized [cx,cy]
0 to 1
metrics.height_ratio
float or null
body height / baseline height
null for baseline run
annotated_image
string
Path to full-frame PNG with overlays
non-empty
crop_image
string
Path to the defined zoomed crop
non-empty
recognizable
string
Agent-judged recognizability
pending until the agent sets pass/fail; never machine-set
FTK_*DB Rows Touched
DB row type
Read / Write / New
Purpose
(none)
-
The gate touches no FTK_*DB rows. The baseline toggle reads an environment variable and skips the custom glb assignment; it adds no rows and writes no game data.
Content.* API Surface
Method / Property
Parameters
Return Type
Behaviour
(none)
-
-
No public Content.* surface is added. The gate is tooling/harness; modders never call it. The only C# change is an internal env-gated branch in RealmBossAdventure.BuildBoss.
Simplicity Constraints
The gate adds NO public Content.* API and NO Core/ change. The single C# edit is an internal, env-gated branch in Content/RealmBossAdventure.cs.
One segmentation front-end is implemented once and reused by both the offline preview (FR-5) and the in-game analyzer (FR-3), so "preview predicts in-game" is true by construction, not by two parallel implementations.
Forbidden: any ML / trained classifier; recognizability stays agent-judged from a crop.
Forbidden: any image processing inside the C# bridge or on the Unity main thread; all measurement is offline Python over the PNG the bridge already produces.
Forbidden: any new networked or serialized state, any IdAllocator call, or any FTK_*DB row change.
The capture driver hard-codes the one documented Hollow Mire / FloodedCrypt / room-4 sequence; no configurable scene-graph driver in this spec.
Edge Cases and Error Handling
Scenario
Expected Behaviour
advance ended the turn -> black diorama
Driver never calls advance; if the frame is black, analyzer returns INCONCLUSIVE (unlit-or-empty-frame)
Screenshot 503 / no active session
Driver returns INCONCLUSIVE (screenshot-503-no-session); never a synthetic PASS/FAIL
Boss not in frame / wrong room
Analyzer returns INCONCLUSIVE (boss-not-in-frame)
baseline.json missing
Analyzer can still report connected (baseline-free); placement/scale criteria report inconclusive for those criteria with a warning, and the overall verdict is INCONCLUSIVE
Baked lantern reads as a second component
connected allows up to a documented small component count and keys on largest-component fill fraction, which a real shatter fails regardless
In-game capture resolution differs between baseline and candidate
Driver asserts equal window resolution between baseline.json and the in-game candidate; mismatch is reported and the run is INCONCLUSIVE (this does NOT apply to the offline preview, which is compared on normalized criteria only)
Recognizability is subjective
Recorded as an agent-judged field; never blocks the mechanical verdict
Dependencies and Risks
Item
Impact
Likelihood
Mitigation
Lighting / segmentation variance (diorama lit only on first turn)
False INCONCLUSIVE or mis-segmentation
High
Anchor capture to dismiss_message after heroTurnReady==true; INCONCLUSIVE on unlit frames; tune the luminance+saturation thresholds against the captured baseline frame
Screenshot 503 / no session
No frame to analyze
Medium
Driver gates on phase==combat and heroTurnReady before screenshot; INCONCLUSIVE on failure
Resolution dependence
Thresholds wrong on another machine
Medium
Normalized metrics; record image_w/h; require equal baseline/candidate resolution
Baseline drift (boss visual re-tuned later)
Stale thresholds
Medium
baseline.json tracks the vanilla trollCaveA chassis (not the boss tuning); re-capture only if the chassis changes; baseline run is one env-gated command
Recognizability subjectivity
Inconsistent calls
Medium
Bound it: defined crop beside the offline reference render; agent records a one-line reason; never blocks mechanical criteria
Offline render cannot reproduce the runtime shatter
Preview passes, in-game shatters
Expected by design
The preview gates ASSET breakage; the in-game capture is the authority on RUNTIME breakage; both verdicts are recorded
Implementation Phases
Phase 1: Offline metrics and preview launch-gate
Goal: Offline mechanical FAIL blocks a wasted launch. No game, no C#. Tasks:
Extend tools/ai-model-pipeline/03_render_preview.py (or fork to 03b_preview_metrics.py) with the segmentation front-end and a preview_verdict.json over the rendered .glb.
Verify a coherent rig predicts mechanical PASS and a malformed .glb predicts mechanical FAIL.
Phase 2: In-game capture driver and baseline
Goal: Produce full-frame candidate and baseline PNGs. Tasks:
Add harness/capture_boss_scene.py driving the documented sequence via the MCP tools.
Add the FTK_BASELINE_STOCK_BODY env branch in RealmBossAdventure.BuildBoss (content-author), keeping shipped behavior byte-identical when off.
Capture and commit baseline.json from the stock-troll run.
Phase 3: Analyzer, verdict, and thresholds
Goal: A machine-readable verdict over the full frame. Tasks:
Whether to extend 03_render_preview.py in place or fork it to 03b_preview_metrics.py (decide during Phase 1; either reuses the single shared analyze(...) segmentation/criteria module).
[UNVERIFIED -- needs game-decompile-analyst] Whether the stock trollCaveA chassis receives any spawn-time scale (it determines whether the candidate-to-baseline height ratio is ~1.4 or some other value); confirm before finalizing the scaled band in Phase 3.
RESOLVED: the baseline artifact is committed at tools/ai-model-pipeline/baseline.json (co-located with the analyzer that consumes it and is the threshold authority); the harness capture driver writes there. Resolve before Phase 2 closes if a different tracked path is preferred.
[VERIFIED via FTKModFramework/Content/RealmBossAdventure.cs:130, per ftk-architect] BossBodyScale = 1.4; the baseline is the scale-1 stock chassis, so the candidate height ratio is expected near 1.4.
[VERIFIED via FTKModFramework/Agent/BridgeHost.cs:64, per ftk-architect] the in-game capture is full-frame at Screen.width x Screen.height, and the capture path lives in the Agent/ test bridge (not Core/).
[VERIFIED via harness/README.md] boss is level 1 room 4 (fid 1112402686); the diorama is lit only on the player's first turn; use dismiss_message, not advance.
Parent Epic
#65 -- Epic: Robust custom enemy/boss 3D-model pipeline (editor-free, in-game verified)
Parent epic: #65
Executive Summary
This spec defines the visual-verification gate: the measuring instrument that turns "does the custom boss body render correctly in-game?" into an objective, reproducible PASS / FAIL / INCONCLUSIVE verdict over a FULL-FRAME screenshot, plus an offline rasterizer preview that predicts the verdict before a game launch is spent. It is built first in epic #65 so the later approach-spike specs (AssetBundle / runtime-glTF / procedural) can be judged against the same gate instead of by eyeballing hand-picked zoomed crops, which is the failure mode that produced repeated false "it's fixed" claims. The driving case is the Hollow Mire boss "Mudwretch Foreman" (
ftkmf_mudwretch_foreman), a reskin of the vanillaenTroll01troll. The gate is offline tooling plus the existing agent harness; it adds no publicContent.*surface, noCore/change, and no networked or serialized state.Problem Statement
A custom boss body can be loaded and skinned at runtime, but the prior sessions could not reliably tell whether the result was correct. The body has variously rendered as a shattered triangle cloud, off-center, mis-scaled, or hunched, while a cropped-and-zoomed screenshot was read as a pass. The root causes were:
The framework already ships the pieces to fix this but has not assembled them into a gate:
harness/ftk_mcp_server.pyexposesftk_observe/ftk_act(action,args)/ftk_wait_for(predicate)/ftk_screenshot(save_path).ftk_screenshotwrites a FULL-FRAME PNG only (no crop, no analysis).harness/README.mddocuments the deterministic boss scene and the lit-diorama /dismiss_messageconstraints.tools/ai-model-pipeline/03_render_preview.pyrenders a.glboffline (Blender Workbench) but produces no metrics or verdict.FTKModFramework/Agent/*) is env-gated byFTK_AGENT_BRIDGE=1, loopback-bound, and per-action single-player-checked, so a test-only capture path is already co-op-safe.The gate's job is to assemble these into one objective, reproducible verdict.
Goals and Success Metrics
verdict.jsonis produced from the FULL frame for every captureconnected: failevery runbaseline.json, not guessed pixel constantsSELF-TEST PASSFunctional Requirements
FR-1: Deterministic boss-scene capture driver
Description: A harness driver runs the documented Hollow Mire boss sequence end to end and captures a full-frame PNG at the only moment the diorama is lit, with no manual steps. The sequence is
start_run HollowMire->enter_dungeon FloodedCrypt->dungeon_regen->cleared_roomx10 (boss is level 1 room 4) ->dungeon_encounter->dungeon_scroll_complete-> wait forcombat.heroTurnReady==true->dismiss_message->ftk_screenshot. It must usedismiss_message, neveradvance(which ends the turn and blacks out the diorama).Acceptance Criteria:
phase==combatwithcombat.heroTurnReady==trueand writes a full-frame PNG to a known path.dismiss_messageand never callsadvancebefore the screenshot.dungeon_regen), the driver emits an INCONCLUSIVE result naming the failing step rather than a synthetic PASS/FAIL, and never throws into the game.Priority: P0
Layer: Harness
FR-2: Vanilla-troll baseline capture and env toggle
Description: The gate can render the boss with the STOCK troll chassis (custom body disabled) in the same scene, so the vanilla-troll silhouette anchors the placement and scale thresholds. This is driven by an environment variable
FTK_BASELINE_STOCK_BODY=1read inRealmBossAdventure.BuildBoss: when set, the custom glb mesh/texture assignment is skipped and the boss renders as the unmodifiedtrollCaveAchassis (white tint, scale 1, no width boost, no swamp aura). When the variable is unset or not equal to1,BuildBossproduces the identical visual configuration it does today and the env read has no effect on shipped/default behavior.Acceptance Criteria:
FTK_BASELINE_STOCK_BODY=1, the boss renders as the stock troll chassis (no custom glb body, no recolor, no scale change).1, the custom body wiring is unchanged and the default runtime behavior is identical to the current shipped path (the env read has no effect when off).FTK_BASELINE_STOCK_BODY=1captures a baseline frame and producesbaseline.jsoncontaining the stock-troll silhouette bbox, centroid, height, and aspect.baseline.jsonis produced by the SAME segmentation front-end used for candidate analysis (FR-3), so baseline and candidate are measured identically.Priority: P0
Layer: Content (one env-gated branch) + Harness (capture)
FR-3: Screenshot analyzer with objective mechanical criteria
Description: An offline Python analyzer takes a full-frame PNG (plus
baseline.jsonwhen present) and computes a single binary foreground mask via a luminance-plus-saturation threshold over a center region-of-interest band (excluding the bottom UI / health-bar strip and top vignette), then derives all mechanical metrics from that one mask. It emits per-criterion PASS/FAIL, the measured values, an annotated full-frame image (bbox, centroid, ROI overlaid), and a machine-readableverdict.json.Acceptance Criteria:
.venv-3dgen(numpy, scipy, Pillow). (The "no ML / numpy+scipy+Pillow only" constraint applies to this analysis stage; the offline RENDER stage in FR-5 reuses Blender.)analyze(frame_png, baseline) -> verdict, with explicit inputs and no hidden state, imported by both the in-game analyzer (this FR) and the offline preview (FR-5), so "same criteria" is enforced structurally (one function, two callers) rather than by convention.verdict.jsonconforming to the schema in Technical Architecture, plus an annotated full-frame PNG.connectedcriterion reports FAIL.Priority: P0
Layer: Tooling
FR-4: Objective pass criteria with baseline-anchored thresholds
Description: The four mechanical criteria are defined with explicit, baseline-anchored thresholds.
connectedis baseline-free;upright,centered, andscaledare expressed relative tobaseline.json. Critical anchoring fact: the baseline is captured withFTK_BASELINE_STOCK_BODY=1, which forces the stock chassis atscale 1, while the custom boss renders atBossBodyScale = 1.4. Therefore the expected candidate-to-baseline HEIGHT ratio is approximately 1.4 (not 1.0), and the scale band is centered on ~1.4, not on 1.0. The exact center and tolerance are finalized against the captured baseline in Phase 3.Acceptance Criteria:
connected: PASS when the largest-component fill fraction is at or above a documented floor and the count of components larger than a documented noise floor is at or below a documented small integer (to allow the body plus a baked lantern). The exact numbers are recorded in the spec's threshold table and indocs/AI-MODEL-PIPELINE.md.upright: PASS when the largest component's major-axis angle is within a documented tolerance of vertical AND the bbox aspect proxy is at or above a documented fraction of the baseline aspect.centered: PASS when the normalized centroid is within a documented fraction of the baseline centroid in x and y.scaled: PASS when the height ratio vs baseline falls inside a documented band centered on ~1.4 (theBossBodyScalemultiplier vs the scale-1 stock chassis), and the area ratio falls inside a documented band centered on ~1.4 squared. Whether the stock chassis itself receives any additional spawn-time scale is[UNVERIFIED -- needs game-decompile-analyst]; if it does, the expected ratio is recomputed accordingly in Phase 3.thresholdstrings and in the docs; none is described with ambiguous language.Priority: P0
Layer: Tooling
FR-5: Offline rasterizer preview that gates the launch
Description: The offline RENDER stage reuses the existing Blender Workbench rasterizer (extend
03_render_preview.pyor fork to03b_preview_metrics.py) to render the candidate.glbplus texture to a PNG. The ANALYSIS stage then runs the SAME sharedanalyze(...)function from FR-3 on that rendered PNG, emitting apreview_verdict.json. The offline mechanical result GATES the in-game launch: if the asset fails the offline mechanical checks (not a single coherent body, not upright, or an out-of-range bbox), the in-game capture is not run. This catches asset-side breakage offline; the in-game capture then catches runtime-reconstruction breakage (the shatter is a runtime-only bug that the offline render cannot reproduce). The preview camera and framing are an approximation of the in-game crypt camera, so preview-vs-in-game agreement is judged on the four NORMALIZED mechanical verdicts only, never on raw pixels or absolute resolution.Acceptance Criteria:
.glbplus texture (Blender) and producespreview_verdict.jsonusing the same sharedanalyze(...)function as FR-3..glbpredicts mechanical FAIL.Priority: P0
Layer: Tooling
FR-6: Full-frame verdict and recognizability crop
Description: The verdict is computed from the FULL frame. The analyzer additionally emits a defined zoomed crop of the boss region and records the frame dimensions and the foreground bbox within them. Recognizability ("reads as the intended mud/bog creature") is AGENT-JUDGED from that crop shown beside the offline reference render; it is recorded as a field but is never computed by an ML model and never blocks the four mechanical criteria.
Acceptance Criteria:
verdict.jsonrecordsimage_w,image_h, and the largest-component bbox within the full frame.recognizablefield defaults topendingand is filled by the agent (pass/failplus a one-line reason); the four mechanical criteria are decided without it.Priority: P0
Layer: Tooling
FR-7: INCONCLUSIVE handling
Description: Before any PASS/FAIL is decided, the analyzer detects degenerate captures and returns INCONCLUSIVE with a reason, so an unlit or empty frame never reads as a confident verdict in either direction.
Acceptance Criteria:
verdict=inconclusivewith reasonunlit-or-empty-frame.verdict=inconclusivewith reasonboss-not-in-frame.verdict=inconclusivewith a reason that names the cause.Priority: P0
Layer: Tooling
FR-8: Single-command reproducible verdict
Description: One entry point runs the whole gate: offline preview (launch gate) -> in-game capture (if the preview passed) -> analyze -> print PASS/FAIL/INCONCLUSIVE with per-criterion reasons and the paths to the annotated image, crop, reference render, and
verdict.json.Acceptance Criteria:
Priority: P1
Layer: Tooling
Non-Functional Requirements
NFR-1: Reproducibility
Category: Reliability
Requirement: Given the same build and the same captured frames, the analyzer produces an identical mechanical verdict. All placement/scale thresholds use normalized coordinates and areas; the verdict records
image_w/image_h. The raw resolution-equality check applies ONLY betweenbaseline.jsonand the in-game CANDIDATE capture (both produced by the sameftk_screenshotpath at the sameScreensize); preview-vs-in-game agreement is judged on normalized criteria only (per NFR-5), never on raw pixels or absolute resolution.Rationale: A gate that is not reproducible cannot adjudicate the later approach spikes; conflating the offline-render resolution with the in-game resolution would otherwise force a false INCONCLUSIVE on every comparison.
NFR-2: Co-op safety and visual-only
Category: Compatibility
Requirement: The gate introduces no new networked or serialized state, no
FTK_*DBrow changes, and noIdAllocatorcalls. The only C# change is an env-gated branch that selects between two already-supported visual configurations. The capture path relies on the existing bridge, which is env-gated byFTK_AGENT_BRIDGE=1, loopback-bound, and per-action single-player-checked.Rationale: Epic #65 requires the whole pipeline to stay co-op-deterministic and save-safe.
NFR-3: net35 / shipped-behavior-identical
Category: Maintainability
Requirement: The
RealmBossAdventure.BuildBossenv branch targets net35 / Mono and, whenFTK_BASELINE_STOCK_BODYis unset or not1, leaves shipped behavior byte-identical. The build stays green and logsSELF-TEST PASS.Rationale: The framework targets net35 and must not regress shipped behavior for a test lever.
NFR-4: Dependency minimalism
Category: Maintainability
Requirement: The tooling reuses the existing
.venv-3dgen(numpy, scipy, Pillow, trimesh) and the local Blender; it adds no machine-learning dependency and no heavy new packages.Rationale: Avoid permanent review-debt and a model dependency for a single-creature proof case.
NFR-5: Resolution independence
Category: Reliability
Requirement: All thresholds are expressed as fractions of frame dimensions or as ratios to the baseline, never as raw pixel constants tied to one resolution. This is what allows the Blender-rendered preview (its own resolution) and the in-game capture (
Screen.widthxScreen.height) to be compared on the same criteria.Rationale: Capture resolution varies by machine and window size, and the offline render resolution differs from the in-game resolution by design.
NFR-6: Divergence recorded, never auto-reconciled
Category: Reliability
Requirement: The single-command output (FR-8) always prints BOTH the offline preview verdict and the in-game verdict side by side and never silently substitutes one for the other. A disagreement (e.g., preview PASS but in-game FAIL, the expected signature of a runtime-only shatter) is surfaced, not hidden.
Rationale: This is the anti-eyeballing guarantee at the heart of the epic; a single reconciled number would re-introduce the ambiguity the gate exists to remove.
NFR-7: Baseline provenance
Category: Reliability
Requirement:
baseline.jsonrecords its capture resolution, the chassis name (trollCaveA), the framework version or commit, and that it was produced underFTK_BASELINE_STOCK_BODY=1, so a stale or mismatched baseline is detectable by the analyzer.Rationale: The placement/scale thresholds are entirely baseline-anchored; an unlabeled or stale baseline would silently invalidate every placement/scale verdict.
Technical Architecture
Component Diagram
graph TD subgraph Tooling["Tooling (.venv-3dgen Python, offline)"] A1["preview metrics<br/>(extend 03_render_preview.py)<br/>render .glb + mechanical metrics"] A2["visual_verdict.py (NEW)<br/>segment -> 4 criteria -> verdict.json<br/>+ annotated PNG + crop"] A3["baseline.json<br/>stock-troll bbox/centroid/height/aspect"] end subgraph Harness["Harness (existing MCP client)"] H1["capture_boss_scene.py (NEW)<br/>drives the documented sequence"] H2["ftk_mcp_server.py (UNCHANGED)<br/>ftk_act / ftk_wait_for / ftk_screenshot"] end subgraph Content["Content/ (one env-gated branch)"] C1["RealmBossAdventure.BuildBoss<br/>FTK_BASELINE_STOCK_BODY=1 -> stock troll"] end subgraph Agent["Agent/ (UNCHANGED test bridge)"] K1["BridgeHost.CapturePng<br/>full-frame ReadPixels PNG<br/>(Screen.width x Screen.height)"] end A1 -->|"offline mechanical PASS gates launch"| H1 C1 -.->|"baseline run only"| H1 H1 --> H2 --> K1 K1 -->|"full-frame PNG"| A2 A3 -->|"anchors centered/scaled/upright"| A2 H1 -->|"baseline run feeds"| A3Mechanical Criteria and Threshold Forms
The scale band is centered on ~1.4 because the baseline is the scale-1 stock chassis and the boss renders at BossBodyScale = 1.4. Initial numbers are starting points; FR-4 requires them finalized and recorded against the captured
baseline.jsonin implementation phase P3, and recomputed if the stock chassis is found to receive any spawn-time scale ([UNVERIFIED -- needs game-decompile-analyst]).Key Types (verdict.json)
schemaftk-visual-gate/1verdictpass/fail/inconclusiveinconclusive_reasonverdict==inconclusivesource_imagebaseline_pathbaseline.jsonusedimage_w,image_hcriteria.<name>.passcriteria.<name>.valuecriteria.<name>.thresholdmetrics.fill_fractionmetrics.component_countmetrics.bbox[x,y,w,h]of largest componentmetrics.centroid_norm[cx,cy]metrics.height_ratioannotated_imagecrop_imagerecognizablependinguntil the agent setspass/fail; never machine-setFTK_*DB Rows Touched
FTK_*DBrows. The baseline toggle reads an environment variable and skips the custom glb assignment; it adds no rows and writes no game data.Content.* API Surface
Content.*surface is added. The gate is tooling/harness; modders never call it. The only C# change is an internal env-gated branch inRealmBossAdventure.BuildBoss.Simplicity Constraints
Content.*API and NOCore/change. The single C# edit is an internal, env-gated branch inContent/RealmBossAdventure.cs.IdAllocatorcall, or anyFTK_*DBrow change.Edge Cases and Error Handling
advanceended the turn -> black dioramaadvance; if the frame is black, analyzer returns INCONCLUSIVE (unlit-or-empty-frame)screenshot-503-no-session); never a synthetic PASS/FAILboss-not-in-frame)baseline.jsonmissingconnected(baseline-free); placement/scale criteria reportinconclusivefor those criteria with a warning, and the overall verdict is INCONCLUSIVEconnectedallows up to a documented small component count and keys on largest-component fill fraction, which a real shatter fails regardlessbaseline.jsonand the in-game candidate; mismatch is reported and the run is INCONCLUSIVE (this does NOT apply to the offline preview, which is compared on normalized criteria only)Dependencies and Risks
dismiss_messageafterheroTurnReady==true; INCONCLUSIVE on unlit frames; tune the luminance+saturation thresholds against the captured baseline framephase==combatandheroTurnReadybefore screenshot; INCONCLUSIVE on failureimage_w/h; require equal baseline/candidate resolutionbaseline.jsontracks the vanillatrollCaveAchassis (not the boss tuning); re-capture only if the chassis changes; baseline run is one env-gated commandImplementation Phases
Phase 1: Offline metrics and preview launch-gate
Goal: Offline mechanical FAIL blocks a wasted launch. No game, no C#.
Tasks:
tools/ai-model-pipeline/03_render_preview.py(or fork to03b_preview_metrics.py) with the segmentation front-end and apreview_verdict.jsonover the rendered.glb..glbpredicts mechanical FAIL.Phase 2: In-game capture driver and baseline
Goal: Produce full-frame candidate and baseline PNGs.
Tasks:
harness/capture_boss_scene.pydriving the documented sequence via the MCP tools.FTK_BASELINE_STOCK_BODYenv branch inRealmBossAdventure.BuildBoss(content-author), keeping shipped behavior byte-identical when off.baseline.jsonfrom the stock-troll run.Phase 3: Analyzer, verdict, and thresholds
Goal: A machine-readable verdict over the full frame.
Tasks:
tools/ai-model-pipeline/visual_verdict.py(four criteria + INCONCLUSIVE + annotated PNG + crop), reusing the Phase 1 segmentation.baseline.json.Phase 4: Single-command verdict and docs
Goal: One recipe, documented.
Tasks:
docs/AI-MODEL-PIPELINE.md, with the Mudwretch Foreman as the worked example.Open Questions
baseline.jsonin Phase 3 (this resolves epic Epic: Robust custom enemy/boss 3D-model pipeline (editor-free, in-game verified) #65's open question about exact "centered" and "correctly scaled" tolerances).03_render_preview.pyin place or fork it to03b_preview_metrics.py(decide during Phase 1; either reuses the single sharedanalyze(...)segmentation/criteria module).[UNVERIFIED -- needs game-decompile-analyst]Whether the stocktrollCaveAchassis receives any spawn-time scale (it determines whether the candidate-to-baseline height ratio is ~1.4 or some other value); confirm before finalizing thescaledband in Phase 3.tools/ai-model-pipeline/baseline.json(co-located with the analyzer that consumes it and is the threshold authority); the harness capture driver writes there. Resolve before Phase 2 closes if a different tracked path is preferred.FTKModFramework/Content/RealmBossAdventure.cs:130, per ftk-architect]BossBodyScale = 1.4; the baseline is the scale-1 stock chassis, so the candidate height ratio is expected near 1.4.FTKModFramework/Agent/BridgeHost.cs:64, per ftk-architect] the in-game capture is full-frame atScreen.widthxScreen.height, and the capture path lives in theAgent/test bridge (notCore/).harness/README.md] boss is level 1 room 4 (fid1112402686); the diorama is lit only on the player's first turn; usedismiss_message, notadvance.