[SPARK-54422][K8S] Increase spark.kubernetes.allocation.batch.size to 20

### What changes were proposed in this pull request?

This PR aims to increase `spark.kubernetes.allocation.batch.size` to 20 from 10 in Apache Spark 4.2.0.

### Why are the changes needed?

Since Apache Spark 4.0.0, Apache Spark uses `10` as the default value of executor allocation batch size. This PR aims to increase it further in 2025.
- #49681

### Does this PR introduce _any_ user-facing change?

Yes, the users will see faster Spark job resource allocation. The migration guide is updated correspondingly.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #53134 from dongjoon-hyun/SPARK-54422.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: dongjoon-hyun

docs/core-migration-guide.md

@@ -22,6 +22,10 @@ license: |
 * Table of contents
 {:toc}
 
+## Upgrading from Core 4.1 to 4.2
+
+- Since Spark 4.2, Spark will allocate executor pods with a batch size of `20`. To restore the legacy behavior, you can set `spark.kubernetes.allocation.batch.size` to `10`.
+
 ## Upgrading from Core 4.0 to 4.1
 
 - Since Spark 4.1, Spark Master deamon provides REST API by default. To restore the behavior before Spark 4.1, you can set `spark.master.rest.enabled` to `false`.

docs/running-on-kubernetes.md

@@ -685,7 +685,7 @@ See the [configuration page](configuration.html) for information on Spark config
 </tr>
 <tr>
   <td><code>spark.kubernetes.allocation.batch.size</code></td>
-  <td><code>10</code></td>
+  <td><code>20</code></td>
   <td>
     Number of pods to launch at once in each round of executor pod allocation.
   </td>

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

@@ -487,7 +487,7 @@ private[spark] object Config extends Logging {
       .version("2.3.0")
       .intConf
       .checkValue(value => value > 0, "Allocation batch size should be a positive integer")
-      .createWithDefault(10)
+      .createWithDefault(20)
 
   val KUBERNETES_ALLOCATION_BATCH_DELAY =
     ConfigBuilder("spark.kubernetes.allocation.batch.delay")