Is your feature request related to a problem or challenge?
SessionConfig::new_with_ballista() overrides datafusion.execution.target_partitions to a hard-coded 16:
https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/extension.rs#L370-L377
DataFusion's own default is the number of CPU cores available to the process, which is not meaningful for Ballista because the work runs on executors rather than in the client process, so some override is needed. But a fixed 16 is arbitrary and, as noted in #2125 (comment), it may be too low for the clusters people actually run.
target_partitions is the main lever on scan parallelism in Ballista. File listing chunks a table's files into at most target_partitions file groups, and Ballista disables round robin repartitioning, so a scan will never produce more partitions than this setting no matter how much executor capacity is available. With the default, a 200 file table scanned on a cluster with 64 total vcores still produces only 16 partitions, and because ballista.scheduler.max_partitions_per_task defaults to 1, that stage runs as 16 tasks and leaves most of the cluster idle. Users have to know to raise the setting before their cluster is used.
Describe the solution you'd like
Revisit the default. A few directions worth considering:
- Raise the constant to something larger that better matches typical cluster sizes.
- Derive the default from cluster capacity (total executor vcores) at session creation time, rather than using a constant. This matches what the scheduler tests already do (
SessionConfig::new_with_ballista().with_target_partitions(total_vcores)), but the scheduler would need to decide how to handle sessions created before executors register and clusters that scale up or down mid-session.
- Keep a constant, but pick it deliberately and document the reasoning plus the interaction with
ballista.scheduler.max_partitions_per_task in the tuning guide.
Whichever direction is chosen, the change should come with TPC-H numbers, since raising target_partitions increases shuffle partition counts as well as scan parallelism and that is not uniformly a win.
Describe alternatives you've considered
Leaving the default at 16 and relying on documentation, which is the status quo after #2125.
Additional context
Raised by @milenkovicm in review of #2125.
Is your feature request related to a problem or challenge?
SessionConfig::new_with_ballista()overridesdatafusion.execution.target_partitionsto a hard-coded16:https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/extension.rs#L370-L377
DataFusion's own default is the number of CPU cores available to the process, which is not meaningful for Ballista because the work runs on executors rather than in the client process, so some override is needed. But a fixed
16is arbitrary and, as noted in #2125 (comment), it may be too low for the clusters people actually run.target_partitionsis the main lever on scan parallelism in Ballista. File listing chunks a table's files into at mosttarget_partitionsfile groups, and Ballista disables round robin repartitioning, so a scan will never produce more partitions than this setting no matter how much executor capacity is available. With the default, a 200 file table scanned on a cluster with 64 total vcores still produces only 16 partitions, and becauseballista.scheduler.max_partitions_per_taskdefaults to1, that stage runs as 16 tasks and leaves most of the cluster idle. Users have to know to raise the setting before their cluster is used.Describe the solution you'd like
Revisit the default. A few directions worth considering:
SessionConfig::new_with_ballista().with_target_partitions(total_vcores)), but the scheduler would need to decide how to handle sessions created before executors register and clusters that scale up or down mid-session.ballista.scheduler.max_partitions_per_taskin the tuning guide.Whichever direction is chosen, the change should come with TPC-H numbers, since raising
target_partitionsincreases shuffle partition counts as well as scan parallelism and that is not uniformly a win.Describe alternatives you've considered
Leaving the default at
16and relying on documentation, which is the status quo after #2125.Additional context
Raised by @milenkovicm in review of #2125.