[Epic] Code organization, test infrastructure, and documentation

[fabianlim]

[Epic] Code organization, test infrastructure, and documentation

Umbrella for the non-feature work: restructuring the KTDP pass library, moving
structural testing onto lit/FileCheck, and putting the docs on a build-time
generator. None of it changes what the compiler lowers — it changes where the
code lives, how it is tested, and how it is documented.

Kernel/lowering features are tracked separately (#48 end-to-end scheduler
validation, #87 stick-tiled layouts). This epic is the substrate those depend on.

Why now

Three coupled problems, each blocking the next:

1. Pass library is flat. All KTDP passes sit in one KTDPTransforms
   library, mixing dialect conversions with plain rewrites. Adding a pass means
   adding to an undifferentiated pile, and the single CMake target links
   everything everything needs.
2. Structural coverage is split across two mechanisms. lit/FileCheck exists
   and works (Build spyre-triton-opt and add lit/FileCheck infrastructure #64 built spyre-triton-opt; test/Conversion/, test/Triton/
   carry real RUN: lines), but the older Python single-pass testers
   (test_distribute_work.py, test_lower_compute_ops.py,
   test_lower_desc_memory.py, test_lower_scalar_load.py) and the fixture
   extra_checks still assert structure through the full _make_ktir pipeline.
   Two places to look, and the pytest half still can't isolate a pass.
3. Docs are generated from the tests. scripts/gen_patterns_docs.py reads
   @pattern-decorated Python test methods. So (2) cannot be fixed without
   also moving (3) — changing the test mechanism changes the doc source.

The ordering below reflects that coupling. The tooling step is already done
(#64), so the critical path is: finish the test migration, then the test-derived
docs, then the cleanups the new structure makes safe.

Key steps

Step 0 — lit/FileCheck infrastructure ✅ done

Landed already, no work item:

• Build spyre-triton-opt and add lit/FileCheck infrastructure #64 — builds spyre-triton-opt (third_party/spyre/bin/), adds
  test/lit.cfg.py, lit.site.cfg.py.in, test/CMakeLists.txt. Registers
  both Triton and KTDP dialects/passes, so any pass can be run standalone on
  raw MLIR. Usage (incl. generate-test-checks.py) documented in CLAUDE.md.
• Add lit/FileCheck tests for inter-tile-reduce lowering #72 — first real lit tests (inter-tile-reduce lowering); test/Triton/
  and test/Conversion/ now carry RUN: lines.

Step 1 — Restructure the pass library

• [Cleanup] Split KTDPTransforms into Conversions and Transforms #46 — Split KTDPTransforms into Conversions/, Transforms/,
  Utils/. Conversions are the passes that use ConversionTarget +
  applyPartialConversion and cross dialect boundaries
  (LowerDescriptorMemory, LowerComputeOps, ConvertFunctions,
  LowerScalarLoad); Transforms/ holds LowerProgramId (renamed from
  DistributeWork); Utils/ holds cleanupDeadOps. Three CMake libraries
  replace one. No Analysis/ layer yet.
• Simplify ConvertFunctions to remove two latent bugs #79 — Simplify ConvertFunctions. Replace the three-mechanism
  sequence (hand-rolled clone + applyPartialConversion for tt.return
  alone + manual block-arg retyping) with ordinary MLIR APIs. Removes two
  latent bugs outright: multi-block tt.func bodies currently crash, and
  the clone path assumes a single public function.

Step 2 — Move structural testing to lit/FileCheck

• [Test] Adopt lit/FileCheck for Transform-level tests; migrate @pattern docs to .mlir headers #55 — Migrate the remaining Python structural testers to lit. The
  infrastructure question is settled by Build spyre-triton-opt and add lit/FileCheck infrastructure #64 (Step 0); what remains is
  migration. Port test_distribute_work.py, test_lower_compute_ops.py,
  test_lower_desc_memory.py, test_lower_scalar_load.py to .mlir cases
  run through spyre-triton-opt with a single --pass each, and retire
  SinglePassTester / utils_pattern.py. Also fixes the /// doc comment
  header format that replaces the @pattern decorator — Step 3 depends on
  that format. Supersedes the parts of this epic's test story that Build spyre-triton-opt and add lit/FileCheck infrastructure #64 did
  not already cover; scope [Test] Adopt lit/FileCheck for Transform-level tests; migrate @pattern docs to .mlir headers #55 to migration, not infrastructure.
• [Test] Remove KTIRStructuralTester and extra_checks from fixtures; structural coverage moves to lit #71 — Remove KTIRStructuralTester and extra_checks. Once lit
  covers the structural checks, delete the pipeline-invariant tests and the
  per-variant extra_checks lambdas from meta.py. Fixtures then assert
  only two things: end-to-end compilation succeeds, and numerics match the
  oracle.

Step 3 — Put docs on a build-time generator

• [Docs] Migrate gen_patterns_docs.py to MkDocs + mkdocs-gen-files #70 — Migrate gen_patterns_docs.py to MkDocs + mkdocs-gen-files.
  Replace the manual script and its --check CI step with a build-time hook
  that parses /// headers from .mlir files under test/Transforms/ and
  test/Conversion/, cross-referencing fixture VARIANTS tags. CI runs
  mkdocs build --strict. Blocked on [Test] Adopt lit/FileCheck for Transform-level tests; migrate @pattern docs to .mlir headers #55 landing the header format.
• [Feature, Docs] Surface Spyre-only Triton patterns in the generated pattern docs #2 — Surface Spyre-only patterns in the docs. Some lowerings depend on
  Spyre-only relaxations of upstream Triton semantics (first case: tl.cat /
  tl.join). A kernel author copying such a pattern to upstream Triton hits
  a failure with no warning. Needs a spyre_only flag threaded through the
  new generator — so it should be designed into [Docs] Migrate gen_patterns_docs.py to MkDocs + mkdocs-gen-files #70 rather than retrofitted.

Step 4 — Fix the install/build documentation

• [docs] UV_PROJECT_ENVIRONMENT does not affect uv pip install — install guidance silently targets the wrong venv #81 — UV_PROJECT_ENVIRONMENT does not affect uv pip install. Both
  install guides claim it does. Following the documented steps can build
  Triton into one venv while uv run pytest runs from another, silently.
  Fix CLAUDE.md and README.md.
• Consolidate setup.py-generated LLVM path artifacts into one source #34 — Consolidate the generated LLVM path artifacts. setup.py emits
  both install-ktdp-mlir-bindings.sh and test/_llvm_config.py, each
  encoding the same build-time MLIR_DIR. Two sources of truth for one
  value.
• Note: two MLIR runtimes in the spyre numerical tests #10 — [Tentative] Document/resolve the two MLIR runtimes in numerical tests.
  libtriton.so and mlir_ktdp each statically embed their own MLIR;
  co-loading segfaults. The subprocess-parse workaround is temporary and
  currently only recorded in the issue.

Step 5 — Structural cleanups unlocked by the above

• Replace ktir-mlir-frontend submodule with dataflow-scheduler #68 — [Tentative] Replace the ktir-mlir-frontend submodule with
  dataflow-scheduler. Consume the frontend through the scheduler, which
  already submodules it. Touches .gitmodules, third_party/spyre/CMakeLists.txt,
  and two setup.py paths. Note this moves the submodule that
  spyre-triton-opt links against, so re-verify the lit suite still builds
  after it lands.