Initial support for parallel Soroban phase XDR.

[dmkozh]

Description

I've tried to minimize the scope of the changes; specifically this doesn't contain any actual logic for the parallel execution (such as data dependency validation and building parallel stages). However, there is still some refactoring that needed to happen in order to support new, more complex tx sets.

I've also removed some legacy surge pricing logic and the corresponding tests - we should no longer need the surge pricing logic that covers more than one source account per tx set.

Checklist

• Reviewed the contributing document
• Rebased on top of master (no merge commits)
• Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
• Compiles
• Ran all tests
• If change impacts performance, include supporting evidence per the performance document

[sisuresh]

Just an initial review

[marta-lokhova]

Thanks, parallel application of Soroban txs is exciting! I did an initial preliminary review. My main question is around scope and motivation:

• It's a bit hard to imagine how the refactored tx set is going to be used, because ledger close logic isn't wired up yet. As a result, I can't tell if the access/usage patterns are right in the context of multi-threading. Is there a reason to introduce apply-order changes before parallel application?
• (related) can you clarify the scope of this PR? if it's just refactoring tx set to be more general, then it'd be easier to make TxSetFrame more generalized internally, but continue exposing old API to make it easy for consumers. Changes to application order should probably be an isolated protocol change. This should help ship this work incrementally.
• There are some changes that I think should be factored out into a separate PR (surge pricing, test cleanup). There is no reason to block those changes on tx set refactor.

[marta-lokhova]

(I should mention that this is my first time tapping into soroban parallelization work, so please bear with me if I'm missing context)

[dmkozh]

It's a bit hard to imagine how the refactored tx set is going to be used, because ledger close logic isn't wired up yet. As a result, I can't tell if the access/usage patterns are right in the context of multi-threading. Is there a reason to introduce apply-order changes before parallel application?

This PR actually has to go first, because we can't properly wire up the parallel application logic if we don't have a proper schedule to use. FWIW there is no multi-threading here at all; the only meaningful interface change is that we'll expose the 'parallel' application schedule from the respective phase (instead of a transaction sequence). I probably mentioned this elsewhere, but to be clear we have to build a tx set in a proper fashion in order to be able to apply it in parallel (specifically we need to run the transactions with data dependencies in the same thread or different stages).

(related) can you clarify the scope of this PR? if it's just refactoring tx set to be more general, then it'd be easier to make TxSetFrame more generalized internally, but continue exposing old API to make it easy for consumers. Changes to application order should probably be an isolated protocol change. This should help ship this work incrementally.

The idea here is to provide support for a new phase type in the TxSetFrame; I've tried to limit the API changes as much as possible and basically the main change is that instead of exposing all the transactions at once we expose the phases separately (that just required an additional loop in a few places).

I realize that some stuff could be moved to separate PRs; I'll at least move all the surge-pricing related stuff. That said, I'm not sure the interface changes would make much sense without at least the stub for supporting the new XDR (otherwise it's hard to tell what's the motivation of the changes and whether they look reasonable). I could try separating that as well, but I'm not sure it will help reviewing the design decisions.

[dmkozh]

I've rebased and updated the PR. There is no rush to merge it into the minor release, so I suppose it'll need to wait more.

[dmkozh]

This PR should be rebased on top of the most recent changes, including the tx set nomination tests that should add some confidence in the refactoring.

[sisuresh]

Do you have a testcase that pushes a vnext valid parallel tx set in v22?

[dmkozh]

Do you have a testcase that pushes a vnext valid parallel tx set in v22?

Parallel tx sets are not supported in v22, but if you meant v23, there are several tests covering it: generalized tx set XDR conversion for basic XDR conversion, generalized tx set fees for the basic nomination, tx_set/v_next.csv changes for randomized nomination tests (that demonstrate that only structure changes, but not the overall contents).

[sisuresh]

Do you have a testcase that pushes a vnext valid parallel tx set in v22?

Parallel tx sets are not supported in v22, but if you meant v23, there are several tests covering it: generalized tx set XDR conversion for basic XDR conversion, generalized tx set fees for the basic nomination, tx_set/v_next.csv changes for randomized nomination tests (that demonstrate that only structure changes, but not the overall contents).

Once we deploy a version with v23 xdr, but before the v23 protocol upgrade, it'll be possible to receive a parallel tx set over the wire that we should toss right? That's the scenario I'm referring to.

[dmkozh]

Once we deploy a version with v23 xdr, but before the v23 protocol upgrade, it'll be possible to receive a parallel tx set over the wire that we should toss right? That's the scenario I'm referring to.

Yeah, good catch, I've added a test that verifies the phase format<->protocol version correspondence.

[marta-lokhova]

lgtm! I've resolved all my comments