Skip to content

Migrate to ktir-mlir-frontend 233d851: memory_space rename + tools_ktdp move - #208

Merged
fabianlim merged 5 commits into
torch-spyre:mainfrom
fabianlim:bump-ktir
Aug 18, 2026
Merged

Migrate to ktir-mlir-frontend 233d851: memory_space rename + tools_ktdp move#208
fabianlim merged 5 commits into
torch-spyre:mainfrom
fabianlim:bump-ktir

Conversation

@fabianlim

@fabianlim fabianlim commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Bumps the mlir-frontend extra's ktir-mlir-frontend pin 973d67e233d851 and adapts to two breaking changes in that range.

Needed by torch-spyre/triton#108, which bumps the same submodule. Until this lands, that PR's numerical tests fail — every test_numerical[*] dies in the MLIR parse worker.

1. spyre_memory_spacememory_space (a3d6bd1)

Not a pure rename — the enum changed meaning and the optional parameter was renamed:

old:  #ktdp.spyre_memory_space<HBM>            kind enum {unspecified, LX, HBM}
new:  #ktdp.memory_space<global>               kind enum {global, ct_local}
      #ktdp.memory_space<ct_local, ct_id = 0>  (was `core = 0`)

Memory-space parsing is now centralized in parser_utils.parse_memory_space, which accepts the new spelling and maps it onto the interpreter's existing internal names, so the execution model is unchanged:

global    → HBM
ct_local  → LX   (ct_id → lx_core_id)

Both frontends (regex and MLIR) route through that one helper, so they can't drift apart.

The old spelling is now explicitly rejected rather than silently defaulting to HBM — see test_construct_memory_view_rejects_pre_rename_spelling. Worth a look during review: silently accepting the pre-rename form would have been the more dangerous failure, since it would place data in the wrong memory space rather than erroring.

2. tools_ktdp moved into mlir_ktdp.tools

b4cb211 consolidated tools_ktdp into mlir_ktdp/tools/, and 822079f inlined ir_utils into tools/__init__.py, so from tools_ktdp.ir_utils import ... no longer resolves.

That import lives inside the _HAS_MLIR try/except ImportError, so the failure surfaced as the misleading "mlir_ktdp / tools_ktdp not installed" — with mlir_ktdp in fact installed and working. This cost real debugging time on the triton side, so the message is corrected to name only what it actually checks.

Scope

51 files, +422/−223 — mostly mechanical (163 attribute substitutions across examples/ and tests/). The substantive changes are confined to 5 files under ktir_cpu/, plus the CI workflow below:

File
parser_utils.py new parse_memory_space (+72)
mlir_frontend/parser.py import fix; route through the helper
dialects/ktdp_ops.py, ir_types.py, ops/memory_ops.py call sites

pyproject.toml / uv.lock carry the pin bump itself.

Also: CI needed cmake>=4.0

Installing the new frontend pin fails in CI with OSError: [Errno 36] File name too long, and the fix has to live here rather than upstream.

torch-spyre/ktir-mlir-frontend#59 addresses the root cause — MLIR's mlir_generate_type_stubs() joins DEPENDS_TARGETS into a 283-char target name, and CMake 3.x writes that whole name into its file-API reply filename, past NAME_MAX (255). #59 pins cmake>=4.0 in build-system.requires, which truncates the name and relies on the trailing hash.

But this workflow installs with --no-build-isolation, which bypasses build-system.requires entirely, so that pin never applies. The pre-install step supplied only scikit-build-core and nanobind, leaving the build to use the runner's CMake 3.x. Adding cmake>=4.0 / ninja>=1.11 there satisfies it directly.

This is why the pin is 233d851 (the #59 branch head) rather than ed069fb6 — and why it alone was not sufficient.

Testing

1322 passed, 10 skipped, 13 xfailed locally; CI green (1191 passed via the MLIR parser, 131 via the regex parser).

Cross-checked against torch-spyre/triton#108 with this branch installed: the 106 failures there drop to 0 (1146 passed, 47 skipped).

@fabianlim
fabianlim force-pushed the bump-ktir branch 3 times, most recently from dcfde9b to 1fe8ddc Compare August 15, 2026 00:58
fabianlim added a commit to fabianlim/triton that referenced this pull request Aug 15, 2026
The spyre-test extra pins fabianlim/ktir-cpu@bump-ktir, whose head moved to
1fe8ddc: the rank-0 regex changes were dropped (unrelated to this migration,
and untested here — see torch-spyre/ktir-cpu#208), and its ktir-mlir-frontend
pin was corrected ed069fb6 -> 233d851 so the bindings build picks up the
cmake>=4.0 stubgen fix.

Empty commit to re-run CI against that head.

Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
…dp move

Bumps the `mlir-frontend` extra's pin 973d67e -> 233d851 and adapts to two
breaking changes in that range.

1. `a3d6bd1` replaced `#ktdp.spyre_memory_space<HBM|LX>` with
   `#ktdp.memory_space<global|ct_local>`. This is not a pure rename — the enum
   changed meaning, and the optional `core =` parameter became `ct_id =`.
   Memory-space parsing is now centralized in `parser_utils.parse_memory_space`,
   which accepts the new spelling and maps it onto the interpreter's existing
   HBM/LX names so the execution model is unchanged:

       global    -> HBM
       ct_local  -> LX   (ct_id -> lx_core_id)

   Both frontends (regex and MLIR) go through it, so they cannot drift. The old
   spelling is now explicitly rejected rather than silently defaulting to HBM;
   see test_construct_memory_view_rejects_pre_rename_spelling.

2. `b4cb211` consolidated `tools_ktdp` into `mlir_ktdp/tools/` and `822079f`
   inlined `ir_utils` into `tools/__init__.py`, so
   `from tools_ktdp.ir_utils import ...` no longer resolves. That import sits
   inside the `_HAS_MLIR` try/except, so the failure surfaced as the misleading
   "mlir_ktdp is not installed" — the message is corrected too.

The pin is 233d851 rather than ed069fb6 (the tip of that PR range) because
ktir-mlir-frontend#59 sits on top and is required to build the bindings at all:
`05e7a0f` enabled MLIR_PYTHON_STUBGEN_ENABLED, which activated a latent
filename overflow (MLIR joins DEPENDS_TARGETS into a 283-char target name that
CMake 3.x writes into its file-API reply filename, past NAME_MAX). torch-spyre#59 pins
cmake>=4.0, which truncates the name. Repin to the squashed commit on main once
torch-spyre#59 lands.

Verified: 1322 passed, 10 skipped, 13 xfailed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
@fabianlim fabianlim changed the title Migrate to ktir-mlir-frontend ed069fb6: memory_space rename + tools_ktdp move Migrate to ktir-mlir-frontend 233d851: memory_space rename + tools_ktdp move Aug 15, 2026
Comment thread ktir_cpu/dialects/ktdp_ops.py Outdated
Comment on lines +513 to +517
# The dim portion is optional: a rank-0 tile is "index" alone, with no
# dims and hence no 'x' separator (emitted by Triton's LowerScalarLoad
# for a scalar read). "128" without an element type still fails, since
# group(2) is required.
type_match = re.match(r'^(?:(\d+(?:x\d+)*)x)?([a-zA-Z_]\w*)$', inner)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this rank-0 treatment become legal now. See offline discussion about dealing with rank-0.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry this was a runway commit. I was experimenting with rank-0 in a diff set of experiments and got contaminated in. thanks for finding it out. i have now removed it. This PR should be restricted to only renames of memory spaces and the removal of tools_ktdp.

fabianlim and others added 2 commits August 17, 2026 20:21
Addresses review feedback on torch-spyre#208.

Drop the unrelated rank-0 changes (lasch's review comment). Three hunks in
`ktdp_ops.py` — the wrapped-form `parse_tensor_or_memref_type` call, the
optional-dims `access_tile` regex, and the matching `access_shape`
construction — were pre-existing uncommitted work for rank-0 / scalar-tensor
support (Triton's LowerScalarLoad) that got swept into the previous commit.
They are unrelated to the memory_space rename and belong in their own PR, so
they are reverted to their pre-PR state here. `ktdp_ops.py` now differs from
main only by the memory_space change. Test count drops 1326 -> 1322 because
four rank-0 tests came along with those hunks; the two rank-0 tests that
remain in tests/test_dialects_parse.py predate this branch and are untouched.

State the RFC conflict explicitly. The previous wording said the new spelling
"supersedes" RFC 0682 in a parenthetical, which reads as settled rather than
as an open conflict needing a spec update. The rename is a semantic change,
not just a spelling change — the enum went from naming *devices* (HBM/LX) to
naming *visibility* (reachable by all compute tiles vs. private to one) — and
`unspecified` was dropped with no replacement. Now recorded as:

  - docs/gap_analysis.md: new row 4a under a new ⚠️ "spec conflict" legend
    entry, with the action item that RFC 0682 §MemorySpaceAttr be revised.
    Row 4's status is also corrected to ✅ (it used 🟢, absent from the
    legend).
  - CLAUDE.md and .claude/skills/ktir-dialect.md: flagged as a conflict, with
    the note that the dialect is the executable authority here because the RFC
    spelling no longer parses at all, and that reviews should not flag the new
    spelling as a deviation.

Repin 233d851 -> ecfb9ed now that ktir-mlir-frontend#59 has merged; ecfb9ed is
that merge commit and the current tip of main, so the temporary pin below the
PR tip is no longer needed. The pinned LLVM hash is unchanged. The explicit
`cmake>=4.0` install in ci.yml stays: CI installs with --no-build-isolation,
which bypasses the `build-system.requires` where torch-spyre#59 declares that pin, so it
must still be satisfied in the workflow. Its comment is reworded to say so
rather than implying torch-spyre#59 is unmerged.

Verified: 1322 passed, 10 skipped, 13 xfailed, with _HAS_MLIR True (the
frontend suite ran rather than silently skipping).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
…dp move

Bumps the `mlir-frontend` extra's pin 973d67e -> ecfb9ed and adapts to two
breaking changes in that range.

1. `a3d6bd1` replaced `#ktdp.spyre_memory_space<HBM|LX>` with
   `#ktdp.memory_space<global|ct_local>`. This is not a pure rename — the enum
   changed meaning, and the optional `core =` parameter became `ct_id =`.
   KTDP names memory spaces by visibility; the interpreter names the concrete
   Spyre memories they map onto, so the two vocabularies are bridged by one
   table in ir_types:

       KTDP_MEMORY_SPACE_KINDS = {"global": "HBM", "ct_local": "LX"}

   Both frontends (regex in dialects/ktdp_ops.py, MLIR in
   mlir_frontend/parser.py) and MemRef's own validator read that single table,
   so they cannot drift. The execution model is unchanged. The old spelling is
   now rejected rather than silently defaulting to HBM; see
   test_construct_memory_view_rejects_pre_rename_spelling.

2. `b4cb211` consolidated `tools_ktdp` into `mlir_ktdp/tools/` and `822079f`
   inlined `ir_utils` into `tools/__init__.py`, so
   `from tools_ktdp.ir_utils import ...` no longer resolves. That import sits
   inside the `_HAS_MLIR` try/except, so the failure surfaced as the misleading
   "mlir_ktdp is not installed" — the message is corrected too.

CI also needs `cmake>=4.0` at the pre-install step. ktir-mlir-frontend#59
(merged as ecfb9ed) declares that in build-system.requires, but this workflow
installs with --no-build-isolation, which bypasses build-system.requires
entirely — so the pin has to be satisfied here as well.

Verified: 1302 passed, 10 skipped, 13 xfailed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
@fabianlim
fabianlim enabled auto-merge (squash) August 18, 2026 00:35
@fabianlim

Copy link
Copy Markdown
Collaborator Author

@lasch ready for another look. I removed that stray rank-0 change, and also I refactored abit more of the code, as I felt the earlier version was introducing unncessary helpers that made it look over-complicated

@nwang-ibm

Copy link
Copy Markdown
Collaborator

LGTM.

nwang-ibm
nwang-ibm previously approved these changes Aug 18, 2026
fabianlim and others added 2 commits August 18, 2026 16:38
The RoPE kernel (torch-spyre#194) landed on main after this branch renamed
`#ktdp.spyre_memory_space<HBM>` to `#ktdp.memory_space<global>`, so the merge
of main brought in four occurrences of the old spelling. It no longer parses
under the bumped ktir-mlir-frontend pin, which is why the frontend job failed
on the merge commit while the regex-parser job passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>

@lasch lasch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtmac

@fabianlim
fabianlim merged commit 12577b6 into torch-spyre:main Aug 18, 2026
2 checks passed
@fabianlim
fabianlim deleted the bump-ktir branch August 18, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants