feat: bump default gRPC max message size 16 MiB -> 128 MiB

[andygrove]

Summary

At SF1000 several TPC-H plans (Q11, Q21, Q22) encode above the 16 MiB default gRPC message-size ceiling and fail with:

Status { code: OutOfRange, message: "Error, decoded message length too large:
        found ~22 MB, the limit is: 16777216 bytes" }

Raising the runtime flags on scheduler/executor pods (--grpc-server-max-*-message-size=...) only lifts the server-side receive limits. Several paths still use GrpcClientConfig::default() (16 MiB) unconditionally — most notably the scheduler-to-executor gRPC client in ExecutorManager::new:

https://github.com/apache/datafusion-ballista/blob/main/ballista/scheduler/src/state/executor_manager.rs#L88

So task-assignment RPCs with large plans reject even when the servers have been raised, unless the user knows to also configure a config-producer override. Same story in flight_proxy_service.rs and the executor's client pool Default lookups.

This PR bumps every default that fed a 16 MiB ceiling to 128 MiB.

Context: I hit this in #2168 while running SF1000 TPC-H and got stuck for a while before realizing the scheduler-side flags don't cover this path.

What changed

• ballista/core/src/config.rs — BALLISTA_CLIENT_GRPC_MAX_MESSAGE_SIZE default (the SessionConfig-driven client path used by distributed_query).
• ballista/core/src/utils.rs — GrpcClientConfig::default() max_message_size (used by scheduler-to-executor, flight proxy, executor client pool default lookups).
• ballista/scheduler/src/config.rs and ballista/executor/src/config.rs — both CLI flags (--grpc-server-max-{decoding,encoding}-message-size) and the matching struct defaults.
• ballista/executor/src/executor_process.rs — matching ExecutorProcessConfig defaults.
• Help strings updated.

The value is a ceiling, not a preallocation, so bumping it has no runtime cost for messages below the old limit.

New test

Adds a round-trip test that ballista.client.grpc_max_message_size, set on a SessionConfig via options_mut().set(...), reaches BallistaConfig::grpc_client_max_message_size() unchanged. That path (options.set → ExtensionOptions → get_usize_setting) had no existing coverage, and it's the one Python clients rely on via their cluster_config overrides.

Test plan

• cargo test --workspace --lib — all green (all pre-existing suites + 2 new/updated).
• cargo fmt --all.
• cargo clippy --workspace --all-targets -- -D warnings clean.
• Reviewer sanity-check: any other places I missed that hard-code 16 MiB in a default? (Non-default hard-coded values in examples/ were left alone.)

[milenkovicm]

There is also outstanding pr which helps reducing message size #1853

[phillipleblanc]

Looks good to me, one thing to check is whether we want to wire this into create_grpc_client_endpoint to pass along the max message size value.

i.e.

   SchedulerConfig.grpc_client_max_message_size
           v
   GrpcClientConfig.max_message_size
           v
   create_grpc_client_endpoint(config) // here
           v
   ExecutorGrpcClient::new(channel)

create_grpc_client_endpoint reads settings such as:

• connection timeout
• keepalive interval
• HTTP/2 window sizes

But it never reads config.max_message_size. Message-size limits are Tonic codec settings, not HTTP/2 Endpoint settings.

It would need something like:

  let client = ExecutorGrpcClient::new(connection)
      .max_encoding_message_size(grpc_client_config.max_message_size)
      .max_decoding_message_size(grpc_client_config.max_message_size);

[martin-g]

I wonder whether 128MiB is too much ?!
Having a lower value led to an optimisation like #1853.
With a big value like 128MiB it will be harder to notice that some message(s) became too heavy.