From 3d3327dc7ce314d823cf5b37a4d9b328ba63b78f Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 10:59:28 -0400 Subject: [PATCH 01/22] address doc CI issues --- docs/source/user_guide/graph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index b1c239b080..b12fd4f6a8 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -4,7 +4,7 @@ Graphs reduce kernel launch overhead by capturing a sequence of GPU operations i ## Backend support -`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires CUDA SM 9.0+ / Hopper for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop that copies the condition value GPU → host each iteration (causing a pipeline stall). `qd.checkpoint` gating runs entirely on the device on every GPU backend; only the CPU backend uses host-side gating. +`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. | Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | | --- | --- | --- | --- | --- | --- | --- | @@ -44,7 +44,7 @@ my_kernel(x, y) # first call: builds and caches the graph my_kernel(x, y) # subsequent calls: replays the cached graph ``` -This works the same way on CUDA and AMDGPU. The cache is keyed per (compiled-kernel-specialization, launch-id), so different template instantiations (different field bindings, etc.) get their own cached graph. +This works the same way on CUDA and AMDGPU. ### Restrictions From dde771bc1626fae328897b7b8c65c44ee0b361e0 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 09:24:49 -0700 Subject: [PATCH 02/22] [CI] Fold reading-order (no-forward-reference) check into doc-quality Add RULE 3 to the doc-quality agent prompt: within a single doc, information must be ordered so a first-time reader can follow it top-to-bottom without jumping ahead (no forward references). Includes carve-outs for roadmaps, optional "see below" pointers, backward references, cross-doc references, and conventional preamble ordering, and hands pure undefined-term cases to Rule 1. Adds the [order] tag and reworks the violation cap so one rule cannot crowd out the others. --- .github/workflows/check_doc_quality.yml | 34 ++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/.github/workflows/check_doc_quality.yml b/.github/workflows/check_doc_quality.yml index 5e163d8b52..7bd82eae70 100644 --- a/.github/workflows/check_doc_quality.yml +++ b/.github/workflows/check_doc_quality.yml @@ -61,7 +61,7 @@ jobs: /tmp/changed_docs.txt contains a newline-separated list of Markdown (.md) doc files (paths relative to the repo root) that were added or modified in this PR. For EACH file in that list, read the FULL current contents of the file (not just the diff) and review it against - the two rules below. + the three rules below. Target audience: an *end user* of Quadrants. That is, someone who writes GPU programs in Python using the Quadrants library (`@qd.kernel`, `@qd.func`, ndarrays, fields, etc.). @@ -118,16 +118,42 @@ jobs: VIOLATION: internal / non-end-user material (subject to the carve-outs above) appears outside of such a clearly-marked advanced / under-the-hood section. + RULE 3 — Reading order (no forward references). + Within a single doc, information should be ordered so a first-time reader can follow it + top-to-bottom in one pass, without jumping ahead. A FORWARD REFERENCE is when a passage + depends on a concept, parameter, behavior, code element, or result that is only introduced + LATER in the SAME file, such that the reader cannot understand the current passage without + first reading the later one. + The following are NOT order violations: + - A brief overview / roadmap near the top that previews upcoming sections ("this guide + covers A, then B") — signposting the structure, not a dependency: the reader does not + need to understand A or B yet to keep reading. + - Optional "for more detail, see
below" pointers, where the current passage is + fully understandable WITHOUT following them. The test is: MUST read ahead to understand + (violation) vs more depth merely available later (fine). + - Backward references (to something earlier in the same file) are always fine. + - References to OTHER docs/files; this rule is about ordering WITHIN a single file only. + - Conventional preamble ordering (e.g. Prerequisites -> Installation -> Usage). + A plain term that is simply never defined belongs to Rule 1, not here; Rule 3 covers + ordering dependencies (concepts, examples, parameters, results) where the explanation DOES + exist in the file but appears too late. Do not double-flag the same issue under both rules. + VIOLATION: a passage cannot be understood by a first-time reader at the point they reach + it, because what it relies on is only introduced later in the same file. + Read each listed file in full before judging it. You may open any other file in the repo (e.g. linked docs) to verify whether a term is defined elsewhere. Do NOT modify any files. Be precise and conservative: only flag clear violations, not borderline cases. Judge each - file independently. Stop after finding 10 violations in total. + file independently against all three rules. Stop after reporting 10 violations in total, + but do not let one rule crowd out the others: if more than one rule has violations, make + sure each such rule is represented in your list before you stop. If there are NO violations, your final output must start with the word PASS. If there ARE violations, your final output must start with the word FAIL, followed by a list of violations (up to 10), one per line, in the format: - : [term|scope]: - where [term] is a Rule 1 violation and [scope] is a Rule 2 violation. + : [term|scope|order]: + where [term] is a Rule 1 violation, [scope] is a Rule 2 violation, and [order] is a Rule 3 + violation (for [order], cite both line numbers — what is used at line N is only introduced + at line M). PROMPT )" --model claude-4.6-opus-high-thinking --mode ask --output-format text --trust) From aafcc41c71b4703d0d0e9d8a1a267e1d887e4bd1 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 09:25:17 -0700 Subject: [PATCH 03/22] [Doc] Document reading-order rule in doc-quality check description The doc-quality check now enforces a third rule (reading order / no forward references); describe it and its carve-outs in contributing.md. --- docs/source/user_guide/contributing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/contributing.md b/docs/source/user_guide/contributing.md index bc7231fda0..f7826be26c 100644 --- a/docs/source/user_guide/contributing.md +++ b/docs/source/user_guide/contributing.md @@ -180,9 +180,9 @@ The agent reports up to 5 violations, each annotated with the host file's hotnes ### Doc quality check (`check_doc_quality.yml`) -Uses an AI agent to review documentation changes for an end-user audience (someone writing Quadrants kernels in Python, not a compiler engineer). For each `docs/**/*.md` file added or modified in the PR, it reads the entire current file (not just the diff) and checks two things: (1) **undefined terms** — a term a typical user is unlikely to know (specialized or internal jargon, project-specific abbreviations) must be defined at its first use in that file, either inline or via a link to a doc that defines it; (2) **end-user relevance** — internal / implementation / contributor-only material must be confined to a clearly-marked section whose heading contains "Advanced", "Under the hood", "Internals", or "Implementation". +Uses an AI agent to review documentation changes for an end-user audience (someone writing Quadrants kernels in Python, not a compiler engineer). For each `docs/**/*.md` file added or modified in the PR, it reads the entire current file (not just the diff) and checks three things: (1) **undefined terms** — a term a typical user is unlikely to know (specialized or internal jargon, project-specific abbreviations) must be defined at its first use in that file, either inline or via a link to a doc that defines it; (2) **end-user relevance** — internal / implementation / contributor-only material must be confined to a clearly-marked section whose heading contains "Advanced", "Under the hood", "Internals", or "Implementation"; (3) **reading order** — information must be ordered so a first-time reader can follow the file top-to-bottom in one pass, without forward references (a passage that can only be understood by reading something introduced later in the same file). -The following do not count as violations: references to public APIs that the author links to their docs and/or labels as public; brief reader-directed pointers suggesting the reader could contribute upstream or file an issue; the core public API vocabulary a user already knows (e.g. `@qd.kernel`, `@qd.func`, `qd.Template`, fields, ndarrays). The agent reports up to 10 violations. This check is delayed by 30 minutes, to avoid running repeatedly if multiple commits pushed with a short delay between each. +The following do not count as violations: references to public APIs that the author links to their docs and/or labels as public; brief reader-directed pointers suggesting the reader could contribute upstream or file an issue; the core public API vocabulary a user already knows (e.g. `@qd.kernel`, `@qd.func`, `qd.Template`, fields, ndarrays); and, for the reading-order check, an overview/roadmap near the top, optional "see below" pointers (where the current passage still reads fine on its own), and backward references. The agent reports up to 10 violations. This check is delayed by 30 minutes, to avoid running repeatedly if multiple commits pushed with a short delay between each. ### PR change report (`pr_change_report.yml`) From 899600ef3501bf4cbb9bc2090b1c67d597a8e2e6 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 09:27:39 -0700 Subject: [PATCH 04/22] [CI] Simplify doc-quality RULE 3: drop reworked cap and double-flag note Both added cognitive load to the agent for little gain. Revert to the original "stop after 10 violations" cap and remove the Rule 1/Rule 3 delineation note. --- .github/workflows/check_doc_quality.yml | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/.github/workflows/check_doc_quality.yml b/.github/workflows/check_doc_quality.yml index 7bd82eae70..4fbb9f8aed 100644 --- a/.github/workflows/check_doc_quality.yml +++ b/.github/workflows/check_doc_quality.yml @@ -134,18 +134,13 @@ jobs: - Backward references (to something earlier in the same file) are always fine. - References to OTHER docs/files; this rule is about ordering WITHIN a single file only. - Conventional preamble ordering (e.g. Prerequisites -> Installation -> Usage). - A plain term that is simply never defined belongs to Rule 1, not here; Rule 3 covers - ordering dependencies (concepts, examples, parameters, results) where the explanation DOES - exist in the file but appears too late. Do not double-flag the same issue under both rules. VIOLATION: a passage cannot be understood by a first-time reader at the point they reach it, because what it relies on is only introduced later in the same file. Read each listed file in full before judging it. You may open any other file in the repo (e.g. linked docs) to verify whether a term is defined elsewhere. Do NOT modify any files. Be precise and conservative: only flag clear violations, not borderline cases. Judge each - file independently against all three rules. Stop after reporting 10 violations in total, - but do not let one rule crowd out the others: if more than one rule has violations, make - sure each such rule is represented in your list before you stop. + file independently. Stop after finding 10 violations in total. If there are NO violations, your final output must start with the word PASS. If there ARE violations, your final output must start with the word FAIL, followed by a list From 57f07bd4011f6fa509e5efb754be74bbfeb72b0d Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 09:01:45 -0400 Subject: [PATCH 05/22] fix doc ci failures --- docs/source/user_guide/graph.md | 38 ++++++++++++++++----------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index b12fd4f6a8..2ad0cbd182 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -2,20 +2,6 @@ Graphs reduce kernel launch overhead by capturing a sequence of GPU operations into a graph, then replaying it in a single launch. -## Backend support - -`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. - -| Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | -| --- | --- | --- | --- | --- | --- | --- | -| `graph=True` | hardware accelerated | hardware accelerated | hardware accelerated | runs (no acceleration) | runs (no acceleration) | runs (no acceleration) | -| `qd.graph_do_while` | hardware accelerated | host fallback | host fallback | host fallback | host fallback | host fallback | -| `qd.checkpoint` | GPU-side | GPU-side | GPU-side | GPU-side | GPU-side | host-side | - -AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of ROCm 7.2). - -Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now — see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). - ## Basic usage Add `graph=True` to a `@qd.kernel` decorator: @@ -49,9 +35,9 @@ This works the same way on CUDA and AMDGPU. ### Restrictions - **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. -- **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint. Autodiff kernels use the normal launch path. +- **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint (backward). [Autodiff](autodiff.md) kernels use the normal launch path. - **Device-resident ndarrays.** Graph mode bakes device pointers into the cached graph, so all ndarray arguments must be on the GPU. Passing a host-resident ndarray raises an error. -- **`qd_stream` is incompatible** with `graph=True`. Choose one or the other. +- [streams](streams.md) are incompatible** with graph. ### Passing different arguments @@ -179,7 +165,7 @@ Therefore on unsupported platforms, you might consider creating a second impleme ## Checkpoints with `qd.checkpoint` *(experimental)* -> **Experimental.** `qd.checkpoint`, `qd.GraphStatus`, and `kernel.resume(from_checkpoint=...)` are experimental APIs. The shape of the public surface (the context-manager signature, the `@qd.kernel(checkpoints=True)` flag, the `GraphStatus` fields, the host-side resume loop, the error messages, and the cross-backend lowering details) may change in any future release without a deprecation cycle. +> **Experimental.** `qd.checkpoint`, `qd.GraphStatus`, and `kernel.resume(from_checkpoint=...)` are experimental APIs, and may change in the near future, or be removed, or replaced. `qd.checkpoint` lets a graph kernel break partway through, surface a reason to the host, let the host fix things up, and resume from the same location on the next launch. An example use-case is an algorithm implemented as a graph that may need to allocate additional memory partway through, where the operations in the graph are in-place, and therefore cannot be rerun without changing/corrupting the output, and therefore for which simply retrying the whole graph from the start is not an option. @@ -268,7 +254,21 @@ while status.yielded: arr[i] = arr[i] + 1 ``` -The restriction is by design: each top-level statement inside a checkpoint becomes its own GPU task / graph node, so silently wrapping bare statements would hide a sequence of N field writes ballooning into N kernel launches. Forcing the user to write the `for`-wrap themselves keeps the lowering visible and gives a single obvious place to fuse multiple writes into one task by sharing a single wrapper. +The restriction is by design: each top-level statement inside a checkpoint becomes its own GPU task / graph node, so silently wrapping bare statements would hide a sequence of N field writes ballooning into N kernel launches. + +## Backend support + +`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. + +| Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | +| --- | --- | --- | --- | --- | --- | --- | +| `graph=True` | hardware accelerated | hardware accelerated | hardware accelerated | runs (no acceleration) | runs (no acceleration) | runs (no acceleration) | +| `qd.graph_do_while` | hardware accelerated | host fallback | host fallback | host fallback | host fallback | host fallback | +| `qd.checkpoint` | GPU-side | GPU-side | GPU-side | GPU-side | GPU-side | host-side | + +AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of ROCm 7.2). + +Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now — see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). ## Performance @@ -397,7 +397,7 @@ If you do want fixed-size for loops to run optimally on unsupported hardware pla - on graph-do-while-supported hardware: handle adding the additional increment kernel - on graph-do-while-unsupported hardware: handle running the loop entirely on the host-side, to avoid adding a gpu pipeline stall -In practice, for our own kernels, i.e. in genesis-world, they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. +In practice, for our own kernels, i.e. in [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world), they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. ### A while loop, conditional on a device-side scalar tensor, that has been optimized into a fixed-size for loop From cc8ea8a57cd45810bbb7faf9edb5e2f6727c3ef5 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 14:13:39 -0400 Subject: [PATCH 06/22] doc tewaks for ci --- docs/source/user_guide/graph.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 2ad0cbd182..84ee6d594f 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -75,7 +75,7 @@ solve(x, counter) The argument to `qd.graph_do_while()` must be the name of a scalar `qd.i32` ndarray parameter. The loop body repeats while this value is non-zero. -- On CUDA SM 9.0+ (Hopper), this uses CUDA conditional while nodes — the entire iteration runs on the GPU with no host involvement. +- On [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus), this uses CUDA conditional while nodes — the entire iteration runs on the GPU with no host involvement. - On older CUDA GPUs, AMDGPU, and non-GPU backends, it falls back to a host-side do-while loop (see the [backend support table](#backend-support)). ### Patterns @@ -151,7 +151,7 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Caveats -On platforms without native device-side conditional graph nodes — currently CUDA pre-SM 9.0 and **AMDGPU** (HIP has no conditional / while node API as of ROCm 7.2) — the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: +On platforms without native device-side conditional graph nodes — currently CUDA pre-SM 9.0 and **AMDGPU** ([HIP](https://rocm.docs.amd.com/projects/HIP/en/latest/what_is_hip.html) has no conditional / while node API as of [ROCm](https://www.amd.com/en/products/software/rocm.html) 7.2) — the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: - wait for GPU async queue to finish processing - copy condition value to hostside - evaluate condition value on hostside @@ -393,11 +393,9 @@ k1(a, count) The recommendation is to use the graph do while here anyway, if you need it for any platform, in order to ensure the code is compact and maintainable. -If you do want fixed-size for loops to run optimally on unsupported hardware platforms, we could add a specializd `qd.graph_range_for` function. This would: -- on graph-do-while-supported hardware: handle adding the additional increment kernel -- on graph-do-while-unsupported hardware: handle running the loop entirely on the host-side, to avoid adding a gpu pipeline stall +If you do want fixed-size for loops to run optimally on unsupported hardware platforms, please raise an issue, and we can look into this. -In practice, for our own kernels, i.e. in [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world), they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. +In practice, for our own [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world) kernels, they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. ### A while loop, conditional on a device-side scalar tensor, that has been optimized into a fixed-size for loop From ee7754cf7146c43eff29ccc2a60cf3f8d4ea03ce Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 14:15:51 -0400 Subject: [PATCH 07/22] Apply suggestion from @graphite-app[bot] Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com> --- docs/source/user_guide/graph.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index a51b74e215..2e7c2ee95d 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -37,7 +37,9 @@ This works the same way on CUDA and AMDGPU. - **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. - **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint (backward). [Autodiff](autodiff.md) kernels use the normal launch path. - **Device-resident ndarrays.** Graph mode bakes device pointers into the cached graph, so all ndarray arguments must be on the GPU. Passing a host-resident ndarray raises an error. -- [streams](streams.md) are incompatible** with graph. + +- [streams](streams.md) are **incompatible** with graph. + ### Passing different arguments From 04f2f0e77eb12cd19bf07e5cdfcd84f8806a2066 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 16:33:04 -0400 Subject: [PATCH 08/22] address doc CI --- docs/source/user_guide/graph.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 2e7c2ee95d..e2e6dffa9c 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -149,7 +149,7 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Restrictions -- The counter ndarray may be swapped between calls: the cached graph reads each counter through an indirection slot that is refreshed on every launch, so passing a different ndarray (or alternating between several) replays the cached graph without rebuilding it. +- The counter ndarray may be swapped between calls: passing a different ndarray (or alternating between several) replays the cached graph without rebuilding it. ### Caveats @@ -242,7 +242,7 @@ while status.yielded: ### Restrictions -- Must be used inside `@qd.kernel(graph=True, checkpoints=True)`. Without the flag, `qd.checkpoint(...)` raises `QuadrantsSyntaxError` at compile time with a fix-it pointing at `checkpoints=True`. +- Must be used inside `@qd.kernel(graph=True, checkpoints=True)`. Without the flag, `qd.checkpoint(...)` raises `QuadrantsSyntaxError` at compile time. - `cp_id` must be an int literal or an `IntEnum` value, and must be unique across the kernel. - `yield_on=` must be a kernel parameter that is a 0-d `qd.types.ndarray(qd.i32, ndim=0)`; expressions are not supported. - Checkpoints cannot be nested inside other checkpoints. Checkpoints inside a `qd.graph_do_while` body are fine. @@ -465,4 +465,4 @@ In this case, our recommendation is: - use graph do while anyway, if you need it on any platform - this will ensure your code is compact and maintainable - if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform - - for example it could somehow run MAX_ITER iterations anyway, similar to the earlier hand-rolled version, but via the graph abstraction, hence allowing the code to be compact, cross-platform, and also optimally fast + - advanced: for example it could somehow run MAX_ITER iterations anyway, similar to the earlier hand-rolled version, but via the graph abstraction, hence allowing the code to be compact, cross-platform, and also optimally fast From a1d925b3f96e06869ca1c51ebfc26891a91464ef Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 16:54:49 -0400 Subject: [PATCH 09/22] address doc ci issues --- docs/source/user_guide/graph.md | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index e2e6dffa9c..9b3ca47fb5 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -289,9 +289,9 @@ def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): for i in range(c.shape[0]): fn_b(c, i) ``` -We have three top-level for loops, which we call 'offloaded tasks'. Each offloaded task is compiled into a separate GPU kernel. When we call `k1` from python, the c++ host-side code launches three gpu kernels. +We have three top-level for loops, which we call 'offloaded tasks'. Each offloaded task is compiled into a separate GPU kernel. When we call `k1` from python, three gpu kernels are launched, from the host side. -We can migrate it to graph by adding `graph=True`: +We can migrate this qd.kernel to graph by adding `graph=True`: ``` @qd.kernel(graph=True) def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): @@ -304,8 +304,8 @@ def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): ``` Results: -- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 kernels -- on other platforms, there is no change: we still launch 3 gpu kernels: no change: not better, not worse +- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 separate kernels +- on other platforms, there is no change: we still launch 3 separate kernels: no change: not better, not worse ### A while loop, conditional on a device-side scalar tensor @@ -328,11 +328,10 @@ So, we have: - the kernel contains device code, which will run on the gpu - each iteration, we copy the value of cond, from the gpu to the host, and check the value - this causes a gpu pipeline stall: - - first we wait for the entire default stream gpu work to complete/drain - - then we wait for the value of cond to copy from the gpu to the host - - then we run the python code to check the value of cond - - if we continue the loop, we now have to run through the python and c++ machinery to prepare the gpu kernel launch - - then launch the gpu kernels inside k1 + - first Quadrants wait for the entire default stream gpu work to complete/drain + - then Quadrants wait for the value of cond to copy from the gpu to the host + - then Quadrants run the python code to check the value of cond + - if we continue the loop, Quadrants now launches the gpu kernels inside k1 - together, these steps can cause a noticeable delay, reducing throughput speed After migrating to graph with graph do while we have: @@ -351,8 +350,6 @@ Now: - on supported hardware, the cond evaluation takes place on the gpu - and we avoid the gpu pipeline stall - on unsupported hardware, we still incur the pipeline stall, as before - - note that there will be some small acceleration, because the condition evaluation and kernel launch will take place entirely from c++, bypassing python - - no worse, incrementally better ### A fixed-size for loop @@ -371,8 +368,8 @@ for _ in range(num_its): In this case, we have `num_its` launches of the three gpu kernels in k1 - there is nothing on the host side that waits for anything to finish on the gpu-side - there is kernel launch latency associated with: - - running k1 from host-side python - - launching the gpu kernels for each of fn_1, fn_2, fn_3 from host-side c++ + - running k1 from host-side + - launching the gpu kernels for each of fn_1, fn_2, fn_3, also from host-side After migrating to graph we have something like: ``` @@ -464,5 +461,4 @@ The effect in reality is situation dependent: In this case, our recommendation is: - use graph do while anyway, if you need it on any platform - this will ensure your code is compact and maintainable -- if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform - - advanced: for example it could somehow run MAX_ITER iterations anyway, similar to the earlier hand-rolled version, but via the graph abstraction, hence allowing the code to be compact, cross-platform, and also optimally fast +- if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform, or raising an issue From 04f1f092c039bdf8cd7634ec07a6006d1bad5b93 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 09:47:47 -0400 Subject: [PATCH 10/22] address some comments --- docs/source/user_guide/graph.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 9b3ca47fb5..7276f37fac 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -32,6 +32,8 @@ my_kernel(x, y) # subsequent calls: replays the cached graph This works the same way on CUDA and AMDGPU. +Note that graph=True is a pre-requisite for all other options presented in this doc. + ### Restrictions - **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. @@ -149,11 +151,12 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Restrictions -- The counter ndarray may be swapped between calls: passing a different ndarray (or alternating between several) replays the cached graph without rebuilding it. +- The counter ndarray may be swapped between calls: passing a different ndarray (or alternating between several) does not trigger a recompile. ### Caveats -On platforms without native device-side conditional graph nodes — currently CUDA pre-SM 9.0 and **AMDGPU** ([HIP](https://rocm.docs.amd.com/projects/HIP/en/latest/what_is_hip.html) has no conditional / while node API as of [ROCm](https://www.amd.com/en/products/software/rocm.html) 7.2) — the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: +On GPU backends without native device-side conditional graph nodes (see [Backend Support](#backend-support) below): +— the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: - wait for GPU async queue to finish processing - copy condition value to hostside - evaluate condition value on hostside @@ -304,8 +307,8 @@ def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): ``` Results: -- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 separate kernels -- on other platforms, there is no change: we still launch 3 separate kernels: no change: not better, not worse +- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 separate device kernels +- on other platforms, there is no change: we still launch 3 separate device kernels: no change: not better, not worse ### A while loop, conditional on a device-side scalar tensor From 1e2e53bbab2fc6c5eef11dcfb1bb409570444829 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:05:37 -0700 Subject: [PATCH 11/22] Add Advanced section to graph doc explaining kernel launch latency Explain what a GPU kernel launch is, break down launch latency into python/c++/gpu-driver sources, and describe why graph_do_while helps even on unsupported hardware (host-side loop driven in c++, avoiding per-iteration python latency). --- docs/source/user_guide/graph.md | 59 +++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 7276f37fac..74ac8b807b 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -465,3 +465,62 @@ In this case, our recommendation is: - use graph do while anyway, if you need it on any platform - this will ensure your code is compact and maintainable - if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform, or raising an issue + +## Advanced + +The rest of this guide treats "kernel launch latency" as a cost to be reduced, without pinning down what it actually is. +This section unpacks it: what launching a GPU kernel involves, where the latency comes from, and why `graph_do_while` +can still speed things up even on hardware that has no native device-side loop support. + +### What a GPU kernel launch is + +When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead +Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver +enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has +finished — usually before it has even started. Host and GPU run asynchronously. + +Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So +the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch +carries a fixed overhead that is independent of how much data the kernel processes — that per-launch overhead is the +"kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; +when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total +runtime. + +### Where kernel launch latency comes from + +It helps to split the latency of a single launch into three parts: + +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime: the + call into the `@qd.kernel` wrapper, argument type checking, the cache lookup that maps this call to an + already-compiled kernel, packing the arguments into a launch context (argument buffer), and crossing the Python→C++ + boundary. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python + `while`/`for` loop that calls a kernel N times pays it N times. +- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary: selecting the compiled + kernel, copying the (usually tiny) argument buffer to the device, recording the launch, and invoking the driver API + for each offloaded task. Cheaper than the Python side, but non-zero, and paid per offloaded task. +- **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on + the GPU itself before the kernel begins running on the hardware. This is the irreducible floor: even a launch issued + entirely from C++ pays it. + +Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches +into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the +graph is built, and then only a single graph replay per call. + +### Why graph_do_while helps even without hardware support + +Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, +every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads +`cond[()]` and evaluates the `if` before deciding whether to loop again. + +On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — +the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while +loop (see [Backend support](#backend-support)) — it is still typically faster than the hand-written Python version, +because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the +runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide +whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. + +In terms of the three-way split above, the fallback still pays the C++ side and GPU/driver side costs each iteration +(including the GPU→host copy of the condition value and the pipeline stall described in [Caveats](#caveats)), but it +eliminates the Python side cost per iteration. So `graph_do_while` on unsupported hardware lands between the two +extremes: slower than the fully-on-device path, but faster than an equivalent pure-Python `while` loop — while letting +you write the loop once and have it run on every backend. From e323513901898baba9b6791e7642a866d4994a27 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:08:54 -0700 Subject: [PATCH 12/22] Trim python-side latency bullet in graph doc Advanced section Drop the detailed enumeration of per-call python work; too much detail. --- docs/source/user_guide/graph.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 74ac8b807b..7317d3b9bc 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -490,11 +490,9 @@ runtime. It helps to split the latency of a single launch into three parts: -- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime: the - call into the `@qd.kernel` wrapper, argument type checking, the cache lookup that maps this call to an - already-compiled kernel, packing the arguments into a launch context (argument buffer), and crossing the Python→C++ - boundary. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python - `while`/`for` loop that calls a kernel N times pays it N times. +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. + This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` + loop that calls a kernel N times pays it N times. - **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary: selecting the compiled kernel, copying the (usually tiny) argument buffer to the device, recording the launch, and invoking the driver API for each offloaded task. Cheaper than the Python side, but non-zero, and paid per offloaded task. From e78e40ee803bcb6d078e44c4f8d49bee54d43745 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:10:41 -0700 Subject: [PATCH 13/22] Trim c++-side latency bullet in graph doc Advanced section Drop the detailed enumeration of per-task c++ runtime work; too much info. --- docs/source/user_guide/graph.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 7317d3b9bc..d6c1094dd9 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -493,9 +493,8 @@ It helps to split the latency of a single launch into three parts: - **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` loop that calls a kernel N times pays it N times. -- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary: selecting the compiled - kernel, copying the (usually tiny) argument buffer to the device, recording the launch, and invoking the driver API - for each offloaded task. Cheaper than the Python side, but non-zero, and paid per offloaded task. +- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python + side, but non-zero, and paid per offloaded task. - **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on the GPU itself before the kernel begins running on the hardware. This is the irreducible floor: even a launch issued entirely from C++ pays it. From deb8969e3fe9b3aee8e08dc91dc193b5f14a47ed Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:11:13 -0700 Subject: [PATCH 14/22] Trim gpu/driver-side latency bullet in graph doc Advanced section Drop the "irreducible floor" sentence; doesn't add signal. --- docs/source/user_guide/graph.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index d6c1094dd9..3ebfa3049b 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -496,8 +496,7 @@ It helps to split the latency of a single launch into three parts: - **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python side, but non-zero, and paid per offloaded task. - **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on - the GPU itself before the kernel begins running on the hardware. This is the irreducible floor: even a launch issued - entirely from C++ pays it. + the GPU itself before the kernel begins running on the hardware. Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the From b6f97326223bdffa21f54de9abb5526198f8067a Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:11:51 -0700 Subject: [PATCH 15/22] Use 'launch' instead of undefined 'replay' in graph doc Advanced section --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 3ebfa3049b..48e9f8a999 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -500,7 +500,7 @@ It helps to split the latency of a single launch into three parts: Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the -graph is built, and then only a single graph replay per call. +graph is built, and then only a single graph launch per call. ### Why graph_do_while helps even without hardware support From 18016887461ed83f29d0d5fb537b594542733bb3 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:14:09 -0700 Subject: [PATCH 16/22] Add latency hiding section to graph doc Advanced section Explain that launches overlap with execution, so reducing launch latency only helps throughput when kernels are small enough that launch latency is exposed rather than hidden behind execution. --- docs/source/user_guide/graph.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 48e9f8a999..29af230d5e 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -502,6 +502,21 @@ Graphs attack all three. Capturing a sequence of launches into a graph and repla into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the graph is built, and then only a single graph launch per call. +### Latency hiding + +Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next +launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the +GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. + +This is why reducing launch latency does not always improve overall throughput. When each kernel runs long enough that +its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — +the launches were already hidden behind execution. + +It matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, +so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that +regime, cutting launch latency — for example by collapsing many launches into a single graph — directly increases +throughput. + ### Why graph_do_while helps even without hardware support Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, From b192ca8b715b3664a2d86697c2ce18742d2c4e20 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:15:15 -0700 Subject: [PATCH 17/22] Reword 'This is why' to 'So' in graph doc latency hiding section --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 29af230d5e..2d46f7111b 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -508,7 +508,7 @@ Because host and GPU run asynchronously, kernel launches overlap with kernel exe launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. -This is why reducing launch latency does not always improve overall throughput. When each kernel runs long enough that +So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — the launches were already hidden behind execution. From b58a08bda0583a44a4079fa7c3eff9b72ec4dee9 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:15:29 -0700 Subject: [PATCH 18/22] Clarify subject in graph doc latency hiding section ('It' -> 'Kernel launch latency') --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 2d46f7111b..9f626f6209 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -512,7 +512,7 @@ So reducing launch latency does not always improve overall throughput. When each its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — the launches were already hidden behind execution. -It matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, +Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency — for example by collapsing many launches into a single graph — directly increases throughput. From 80eb935a46e20c700357a5e4c189889cc9b714c5 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:15:59 -0700 Subject: [PATCH 19/22] Soften claim in graph doc latency hiding section ('directly increases' -> 'can directly increase') --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 9f626f6209..0ceacaeb86 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -514,7 +514,7 @@ the launches were already hidden behind execution. Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that -regime, cutting launch latency — for example by collapsing many launches into a single graph — directly increases +regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase throughput. ### Why graph_do_while helps even without hardware support From d634672d14baaac57a6fac454597fc38bfb7e86c Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:16:50 -0700 Subject: [PATCH 20/22] Soften unsupported-hardware claim in graph doc graph_do_while section Reframe from 'typically faster' to 'launch latency can be slightly reduced'. --- docs/source/user_guide/graph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 0ceacaeb86..be51d91f93 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -525,8 +525,8 @@ every iteration pays the full Python-side launch latency: Python re-enters the k On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while -loop (see [Backend support](#backend-support)) — it is still typically faster than the hand-written Python version, -because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the +loop (see [Backend support](#backend-support)) — the launch latency can be slightly reduced compared to the hand-written +Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. From c14a85e3268b8cd7b7218e5e72ecbf5c0fea90d2 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:17:25 -0700 Subject: [PATCH 21/22] Remove closing summary paragraph from graph doc graph_do_while section --- docs/source/user_guide/graph.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index be51d91f93..fe4813256c 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -529,9 +529,3 @@ loop (see [Backend support](#backend-support)) — the launch latency can be sli Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. - -In terms of the three-way split above, the fallback still pays the C++ side and GPU/driver side costs each iteration -(including the GPU→host copy of the condition value and the pipeline stall described in [Caveats](#caveats)), but it -eliminates the Python side cost per iteration. So `graph_do_while` on unsupported hardware lands between the two -extremes: slower than the fully-on-device path, but faster than an equivalent pure-Python `while` loop — while letting -you write the loop once and have it run on every backend. From cfde9ce8ab02f58805cc2f45c9d6fb18cb2fa6d8 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:17:52 -0700 Subject: [PATCH 22/22] Soften graph_do_while subsection heading in graph doc ('helps' -> 'can help slightly') --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index fe4813256c..52aea87424 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -517,7 +517,7 @@ so the GPU sits idle waiting for the next launch and the launch latency is *expo regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase throughput. -### Why graph_do_while helps even without hardware support +### Why graph_do_while can help slightly even without hardware support Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads