Iceberg compaction doesn't parallelize well across partitions

[itamarwe]

I'm performing compaction for my Iceberg table and Trino doesn't parallelize well. I see short spikes of high parallelization followed by a single worker working most of the time.
My table is rather skewed. I have a few partitions with many small files that require compaction, while most of the partitions are well compacted.
I would have expected Trino to parallelize the compaction, but is it possible that it doesn't parallelize between partitions?
(I'm using mostly default configuration. And my table is sorted)

EDIT: I'm not sure it even parallelizes between partitions, as even though I have several partitions that require significant compaction I see no more than a single worker working on the compaction.

[ebyhr]

@itamarwe What is your Trino version? Also, could you join Trino Slack workspace so we can talk quickly? https://trino.io/slack.html

[itamarwe]

I'm on Trino 475. I'll join Slack so we can continue the discussion there.

[itamarwe]

To summarize the slack discussion, in cases where there are partitions with many files, since the files are read in sequential order and there is a single writer per partition it creates a downstream bottleneck in the writer stage, and all of the readers are blocked most of the time, leading to very poor resource utilization.

As a workaround, I'm currently running multiple optimization queries in parallel, which does not feel like a good practice. We would have wanted the optimize procedure to parallelize without the need for external orchestration.

I opened a PR for partition aware readers and would greatly appreciate your review.
#26632

[MikeWallis42]

I am experiencing this very same issue and have implemented a similar workaround! We do have some partitions that are actually so large that Trino cannot optimise them in a reasonable amount of time. I think I waited around 10 hours and Trino was reporting about 20% completion.

[itamarwe]

There are actually 2 separate issues here:

1. Trino doesn't parallelize well across partitions - That's the issue I was referring to here. I suggested a workaround (orchestrate and run multiple optimization queries in parallel, one for each partition), and suggested a fix here: Partition-Aware Fan-Out for Iceberg OPTIMIZE #26632
2. Trino doesn't parallelize within a partition - That's the issue that you are referring to. The iceberg connector was intentionally limited to one writer per partition (to avoid small files), but that also kills parallelism and limits the optimization throughput within each partition to a single writer. I opened another issue here: Trino doesn't parallelize OPTIMIZE within a partition #26649