Flaky CI: TPC-H SF10 job intermittently fails with "No space left on device"

[andygrove]

Describe the bug

The TPC-H SF10 (all queries, AQE off + on) CI job (.github/workflows/tpch.yml) intermittently fails with No space left on device (os error 28) while writing/spilling shuffle files. It is non-deterministic — the same commit and even main pass on some runs and fail on others — so it is a flaky, infra-level failure rather than a code regression. It shows up as red X's on unrelated PRs and erodes trust in CI signal.

The failure surfaces two ways depending on where the disk fills:

Task 27 failed: DataFusion error: Arrow error: Io error:
No space left on device (os error 28)

called `Result::unwrap()` on an `Err` value:
External(Os { code: 28, kind: StorageFull, message: "No space left on device" })

To Reproduce

Not deterministically reproducible (that's the problem), but recent examples on 2026-07-09:

• main @ fd30f37 (PR chore(deps): bump taiki-e/install-action from 2.82.7 to 2.82.11 #1973 merge commit) — SF10 failed, StorageFull.
• PR feat: reuse repeated exchanges in the distributed planner (ReuseExchange analog) #1954 branch reuse-exchange @ ebfba6a — SF10 failed, No space left on device; the same branch passed SF10 on several earlier commits the same day.

Job configuration for context:

• runs-on: ubuntu-latest (standard GitHub-hosted runner, ~14 GB free disk) inside a container.
• Generates TPC-H scale factor 10 data into $RUNNER_TEMP/tpch-data, then runs a scheduler + executor cluster and executes all 22 queries with AQE off and on.
• Shuffle files are written (LZ4-compressed) to the executor work dir on the same volume, on top of the SF10 source data and build artifacts.

So the disk holds: build artifacts + SF10 source parquet + shuffle spill for all 22 queries × 2 AQE modes — and occasionally that exceeds the runner's free space.

Expected behavior

SF10 CI should be a reliable signal — either it always has enough headroom to complete, or a genuine failure is distinguishable from "the runner ran out of disk."

Additional context

Filing this so solutions can be discussed rather than proposing one. Some options that have come up / are worth weighing:

• Free up runner disk before the job (e.g. remove preinstalled toolchains/SDKs, docker image prune) — cheap, common GitHub-Actions mitigation, buys several GB.
• Reduce on-disk footprint — clean up per-query shuffle output between queries, and/or point shuffle work dir and TPC-H data at different volumes if available.
• Denser shuffle compression — the shuffle codec is currently hard-coded to LZ4; a ZSTD option would shrink the shuffle footprint. Tracked separately in Make shuffle-file compression codec configurable (add ZSTD option) #1980.
• Larger runner — a runner/volume with more disk (if the project has access to larger hosted or self-hosted runners).
• Lower the scale factor or split the matrix — smaller SF, or splitting AQE-off / AQE-on into separate jobs so peak concurrent disk use is lower.
• Surface disk usage — a df -h step before/after to quantify actual headroom and confirm which artifact dominates.

Related: #1980 (configurable/denser shuffle compression codec).

[andygrove]

Reopening context rather than the issue itself: #1982 fixed this for the static planner only. Under AQE the immediate reclaim was a silent no-op, because create_resolved_stage always leaves output_links empty so intermediate_stage_ids() returned nothing. That is #1996, with a fix in #2149.

The practical effect is that the AQE-on half of the SF10 CI job got no relief at all, and the flake described here is still reachable through that path. It showed up again on #2064, where query 21 failed with No space left on device (os error 28) while the AQE-off leg passed.