[SPARK-51667][SS][PYTHON] Disable Nagle's algorithm (via TCP_NODELAY = true) in TWS + PySpark for python <-> state server

[HeartSaVioR]

What changes were proposed in this pull request?

This PR proposes to disable Nagle's algorithm (TCP_NODELAY = true) for the connection between Python worker and state server, in TWS + PySpark.

Why are the changes needed?

We have observed the consistent latency increment, which is almost slightly more than 40ms, from specific state interactions. e.g. ListState.put() / ListState.get() / ListState.appendList().

The root cause is figured out as the bad combination of Nagle's algorithm and delayed ACK. The sequence is following:

1. Python worker sends the proto message to JVM, and flushes the socket.
2. Additionally, Python worker sends the follow-up data to JVM, and flushes the socket.
3. JVM reads the proto message, and realizes there is follow-up data.
4. JVM reads the follow-up data.
5. JVM processes the request, and sends the response back to Python worker.

Due to delayed ACK, even after 3, ACK is not sent back from JVM to Python worker. It is waiting for some data or multiple ACKs to be sent, but JVM is not going to send the data during that phase.

Due to Nagle's algorithm, the message from 2 is not sent to JVM since there is no ACK for the message from 1. (There is in-flight unacknowledged message.)

This deadlock situation is resolved after the timeout of delayed ACK, which is 40ms (minimum duration) in Linux. After the timeout, ACK is sent back from JVM to Python worker, hence Nagle's algorithm allows the message from 2 to be finally sent to JVM.

The direction can be flipped depending on the command - the same thing can happen on the opposite direction of communication, JVM to Python worker.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually tested (via adding debug log to measure the time spent from the state interaction).

Beyond that, this should pass the existing tests, which will be verified by CI.

Was this patch authored or co-authored using generative AI tooling?

No.

[neilramaswamy]

Is this not a problem for two back-to-back calls to ValueState.get because the second call to valueState.get() on the Python side will only proceed after the response (not ack) from the first one is received?

[HeartSaVioR]

I think we do not design any "asynchronous" or "concurrent" execution of state interaction. Every request with state server is "blocking". As an implementation detail, we limit the amount of data in a call for iterator on state interaction, and the other state interaction can be made while the consumption of iterator has not finished, but this doesn't mean we support "concurrent" state interations.

[HeartSaVioR]

cc. @HyukjinKwon @cloud-fan PTAL, thanks!

[HeartSaVioR]

cc. @bogao007 since he authored the code.

[HeartSaVioR]

Thanks! Merging to master/4.0 (since the delay is not a trivial).