TPC-H q8 is 2.5x slower when run after q1-17

[andygrove]

Describe the bug
TPC-H q8 is 2.5x slower when run after q1-7 compared to running q18 in a fresh cluster.

Theories:

• Contention with async cleanup from previous queries
• Memory pool leaks leading to more spilling
• Shuffle-fetch resource leaks

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[andygrove]

Some data from a run on our SF1000 K8s cluster (AWS EKS, 32 executors × 8 vCPU × 64 GiB memory, --concurrent-tasks=8, --memory-pool-size=48GB, AQE on, Parquet on S3):

• Q8 standalone: 47.4 s
• Q8 running after Q1–Q7 (in a full-suite run): 228.4 s — 4.8× slower

Same commit (main @ 696ca29b, Ballista 54.0.0, DataFusion 54.1.0), same physical K8s nodes (5 of 6 nodes shared between the two runs with identical executor placement per node — ruling out node placement / co-tenant differences at the K8s level).

The slowdown is concentrated in one stage

Comparing per-stage EXPLAIN ANALYZE aggregates (all times summed across the 256 tasks in the stage):

stage (op)	metric	standalone	in-suite	Δ
1 (lineitem scan → hash shuffle write)	write_time	1694 s	8392 s	+396 %
2 (part⋈lineitem hash join)	fetch_time	1263 s	239 s	−81 %
6	fetch_time	7235 s	1281 s	−82 %
6	permit_wait_time	3874 s	1446 s	−63 %
8	fetch_time	509 s	123 s	−76 %
8	permit_wait_time	24 179 s	6 656 s	−72 %

Every stage except stage 1 is faster in-suite (warm shuffle-client connections after Q1–Q7). Stage 1's inflation swamps those wins.

Stage 1: same bytes, no spilling, 5× slower writes

stage 1 metric	standalone	in-suite
output_rows	6.00 B	6.00 B
output_bytes	313.2 GB	313.2 GB
bytes_scanned	85.81 GB	85.81 GB
output_batches	733.9 K	733.9 K
repart_time	61 s	64 s
spill_bytes	0	0
spill_count	0	0
write_time	1694 s	8392 s

Per-task effective throughput: standalone ≈ 7.1 GB/s, in-suite ≈ 1.4 GB/s. Same 313 GB of shuffle output, no extra spilling — just 5× slower per byte.

What's actually going on: kernel page cache vs bandwidth-limited disk

7.1 GB/s per task is far above any local-disk sustained write rate — meaning the standalone writes never actually reach the disk synchronously; they're absorbed by the kernel's page cache and write() returns immediately (buffered writes on the container overlay FS).

Our K8s worker nodes are on AWS Nitro instances with EBS-backed root filesystems (/dev/nvme0n1p1). Ballista's executor working dir (Executor working directory: /tmp/.tmpXXX from the executor log) is on /tmp, which is on the container overlay FS, which is on the root EBS volume. EBS has strict per-instance bandwidth caps (~600 MB/s on typical m5/r5 sizes, shared across all pods on the instance).

In-suite hypothesis: Q1–Q7 dirty hundreds of GB of pages that the kernel is still flushing to EBS when Q8's stage 1 begins. With page cache already loaded, Q8's stage-1 writes force synchronous flushes at EBS bandwidth (~1.4 GB/s per task = ~few hundred MB/s per pod = matches a per-instance EBS cap split across 4–7 pods per node). Standalone Q8 hits a clean cache and never pays that.

Ruling out other explanations

• Node placement: identical, 5/6 nodes shared with same executor distribution.
• Additional spilling: spill_bytes = 0 and spill_count = 0 on every operator in both runs.
• Memory pool exhaustion: per-task pool peaks (measured via a small TrackedMemoryPool wrapper I wrote in feat(executor): log per-task memory pool peak usage #2163) top out at ~1.7 GB / 5.6 GiB limit on Q8. Nowhere near ceiling. Zero residual reservations at pool drop.

Suggested next diagnostic

Run q8, q1, q2, ..., q7, q8 back-to-back on the same cluster. If the second q8 is much slower than the first, it confirms the within-run effect on the same executor processes on the same nodes — separating it from any external variability.

The broader implication is a benchmarking-setup lesson too: on EBS-backed nodes without dedicated instance-store or a provisioned-throughput data volume, sustained shuffle-heavy workloads will hit page-cache saturation and drop to EBS bandwidth. Might be worth documenting the recommended node-storage config for benchmarks/.

[andygrove]

shuffle files were going to wrong ebs volumes - fixed