Skip to content

⚡ Optimize release wheels with SDK/Core PGO (Phase 2) - #2476

Draft
burgholzer wants to merge 12 commits into
mainfrom
codex/optimized-release-toolchain
Draft

burgholzer wants to merge 12 commits into
mainfrom
codex/optimized-release-toolchain

Conversation

@burgholzer

@burgholzer burgholzer commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Description

Add combined SDK/Core profile-guided optimization (PGO) to Linux and macOS release wheels. CMake owns Core ThinLTO in the instrumented and final builds; the SDK rebuild uses PGO with LTO disabled. Linux applies BOLT to the final binaries and validates the repaired wheel with training workloads and the shared installed CMake consumer under Clang and GCC.

This draft is stacked on #2715, including #2717's CMake package isolation and #2718's concrete DD operations. It retains portable CPU settings on every wheel platform, the assertion-free LLVM/MLIR 23.1.2 SDK, and the current Workflows #464 compiler-cache and #491 native mold 3 pins. Linux BOLT tooling depends on SDK #94.

Linux uses matched distribution Clang, LLD, compiler-rt, and llvm-profdata packages from manylinux_2_28. The Phase 1 static Clang 22 bundle lacks the profiling tools required here. macOS uses Apple Clang. scripts/prepare_release.py supplies the generated CMAKE_PROJECT_INCLUDE_BEFORE configuration for both platforms; Windows clears the profiling hooks and keeps its normal portable build. The rebuilt SDK retains its assertion, exception-handling, and RTTI settings.

Validation on the restacked branch:

  • Repository lint, Python compilation, and Bash syntax checks pass.
  • The Linux provisioning preflight accepts a complete SDK and rejects missing or non-executable BOLT tools before downloading LLVM sources; temporary command shims also verify macOS skips this check.
  • Cibuildwheel 4.3 accepts Linux, macOS, and Windows configurations. Effective settings retain the PGO include on Linux/macOS and disable profiling hooks on Windows.
  • Native CMake configuration succeeds with ThinLTO and build RPATHs selected.
  • The shared installed consumer builds and passes CTest with GCC 13 and Clang 23 against the qualified Phase 1 Clang 22 ThinLTO wheel.

Linux qualification is currently blocked: the selected released SDK lacks the BOLT tools, and SDK #94 remains open. Provisioning now requires executable llvm-bolt, merge-fdata, and llvm-strip before the LLVM source download or PGO build. The full SDK/Core PGO build, Linux BOLT processing, final optimized-wheel validation, and macOS qualification have not been run on this head. Baseline consumer checks do not establish this pipeline's correctness or performance. No C++ source changes are introduced relative to #2715. AI assistance covered the restack, configuration alignment, review, and focused validation.

Checklist

  • The pull request only contains commits that are focused and relevant to this change.
  • I have added appropriate tests that cover the new/changed functionality.
  • I have updated the documentation to reflect these changes.
  • The changes follow the project's style guidelines and introduce no new warnings.
  • The changes are fully tested and pass the CI checks.
  • I have reviewed my own code changes.

If PR contains AI-assisted content:

  • Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope, as required by our AI Usage Guidelines.
  • Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖 (titles are exempt).
  • I have disclosed AI assistance in the PR description.
  • I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

@codecov

codecov Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.98658% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
include/mqt-core/dd/CachedEdge.hpp 83.3% 1 Missing ⚠️
src/dd/CachedEdge.cpp 94.1% 1 Missing ⚠️
src/dd/Edge.cpp 99.0% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@burgholzer burgholzer changed the title ⚡ Optimize wheel links and release SDK selection ⚡ Optimize release wheels with matched LTO and BOLT Sep 8, 2026
@burgholzer burgholzer changed the title ⚡ Optimize release wheels with matched LTO and BOLT ⚡ Optimize release wheels with LTO and BOLT Sep 9, 2026
@mergify mergify Bot added the conflict label Sep 10, 2026
@burgholzer burgholzer added this to the v4.1.0 - QDMI 1.4 / MQSF milestone Sep 10, 2026
@burgholzer
burgholzer force-pushed the codex/optimized-release-toolchain branch from 706fd8f to f1eeefe Compare September 12, 2026 13:58
@burgholzer burgholzer changed the title ⚡ Optimize release wheels with LTO and BOLT ⚡ Train and optimize native SDK release wheels Sep 12, 2026
@mergify mergify Bot added conflict and removed conflict labels Sep 12, 2026
@burgholzer
burgholzer force-pushed the codex/optimized-release-toolchain branch from cc686e4 to 3443517 Compare September 14, 2026 18:18
@burgholzer burgholzer changed the title ⚡ Train and optimize native SDK release wheels ⚡️ Optimize release wheels with SDK/Core PGO Sep 14, 2026
@burgholzer
burgholzer changed the base branch from main to codex/fix-mlir-exception-boundaries September 14, 2026 18:18
@mergify mergify Bot added the conflict label Sep 19, 2026
@burgholzer
burgholzer force-pushed the codex/fix-mlir-exception-boundaries branch 5 times, most recently from bf8226b to 3b8d8db Compare September 25, 2026 18:40
@simon1hofmann
simon1hofmann added this pull request to stack #2677 October 5, 2026 11:53
@simon1hofmann simon1hofmann self-assigned this Oct 5, 2026
@burgholzer burgholzer mentioned this pull request Oct 5, 2026
10 tasks done
@burgholzer burgholzer changed the title ⚡️ Optimize release wheels with SDK/Core PGO ⚡ Optimize release wheels with SDK/Core PGO (Phase 2) Oct 5, 2026
Keep cache launchers and project defaults in the source build rather
than applying them during find_package. Load install directories only
when an explicit device helper needs them, and retain the exported
helpers and their targets.

Check consumer settings and a device-helper build against the installed
package.

Assisted-by: GPT-6 via Codex
Replace node-constrained edge members with concrete vector and matrix
function overloads so supported compilers agree on the exported symbols.
Keep edge layouts, numerical behavior, and Python methods unchanged.

Move traversal recursion into the implementation and migrate all C++
callers, including bindings, shared MLIR test helpers, and documentation.

Assisted-by: GPT-6 via Codex
Document the new edge parameters required by Doxygen and record the
completed DD API migration with its validation and remaining scope.

Assisted-by: GPT-6 via Codex
The DD bindings use complex numbers but no declarations from cmath.
Remove the two includes flagged by hosted clang-tidy 23.1.3.

Assisted-by: GPT-6 via Codex
Use the manylinux Clang distribution and LLD, enable Apple Clang IPO,
and retain speed optimization in nanobind bindings. Select portable CPU
settings explicitly for wheels while keeping native tuning for source
builds.

Exercise the installed package and concrete DD API in wheel tests,
including GCC consumers on Linux. Document compiler selection, measured
tradeoffs, and the limits of LTO across native SDK inputs.

Assisted-by: GPT-6 via Codex
@burgholzer
burgholzer force-pushed the codex/optimized-release-toolchain branch from 6793f09 to f1846a2 Compare October 8, 2026 21:22
@burgholzer
burgholzer changed the base branch from main to codex/release-wheel-lto October 8, 2026 21:22
Restore two-level symbol lookup after HandleLLVMOptions, following the upstream AddMLIRPython helper. Keep binding modules from resolving private symbols against another loaded package.

Assisted-by: GPT-6 via Codex
Preserve combined SDK/Core PGO and Linux BOLT while using current public
APIs for release training and installed consumers. Retain Phase 1's SDK
selection and Windows cleanup, and pin the pending wheel cache workflow.

Full optimized-wheel qualification remains pending. The baseline GCC
consumer passes; its Clang 23 counterpart exposes a constrained-template
ABI mismatch in the GCC 13-built baseline wheel.

Assisted-by: GPT-6 via Codex
Record the constrained-template aliases dropped by GCC 14 LTO and the
independent ABI checks needed by the Clang PGO/BOLT pipeline.

Assisted-by: GPT-6 via Codex
Let CMake own Core IPO in both PGO stages and limit Linux ThinLTO
workers through LLD. Keep SDK LTO disabled and preserve profile and BOLT
flags. Update release documentation and the installed consumer for the
concrete DD API prerequisite.

Assisted-by: GPT-6 via Codex
Check repaired wheels with the shared package-policy and DD fixture under
both Clang and GCC. Run its CTest checks and remove the duplicate release
consumer and its lint-only target.

Assisted-by: GPT-6 via Codex
Preserve profile configuration on Linux and macOS while retaining the
matched Linux profiling tools and portable Windows settings. Reuse the
shared installed consumer, retain the current workflow pins, and document
the concrete DD prerequisite and remaining PGO/BOLT qualification.

Assisted-by: GPT-6 via Codex
Reject Linux SDKs missing executable BOLT, profile merging, or strip tools
before downloading LLVM sources and starting the PGO build. Record the
pending SDK prerequisite for full Phase 2 qualification.

Assisted-by: GPT-6 via Codex
@burgholzer
burgholzer force-pushed the codex/optimized-release-toolchain branch from f1846a2 to 5e75e50 Compare October 8, 2026 21:28
@burgholzer
burgholzer force-pushed the codex/release-wheel-lto branch from 8e308fd to fcf206a Compare October 9, 2026 10:27
@mergify mergify Bot added the conflict label Oct 9, 2026
@burgholzer
burgholzer force-pushed the codex/release-wheel-lto branch from fcf206a to d6d02f3 Compare October 9, 2026 10:50
@simon1hofmann
simon1hofmann added this pull request to stack #2721 October 9, 2026 11:49
@burgholzer
burgholzer force-pushed the codex/release-wheel-lto branch 3 times, most recently from be7a8e2 to 23fcc22 Compare October 9, 2026 15:28
@burgholzer
burgholzer force-pushed the codex/release-wheel-lto branch 2 times, most recently from 83be162 to 5938109 Compare October 9, 2026 18:48
Base automatically changed from codex/release-wheel-lto to main October 9, 2026 22:52
@burgholzer
burgholzer removed this pull request from stack #2721 October 9, 2026 22:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflict continuous integration Anything related to the CI setup enhancement Improvement of existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants