Improve the in SDK fork testing experience

[leighmcculloch]

If we are to make the SDKs fork testing a better experience, it needs a fast data source of a stable ledger. The way we have addressed the need for historical state today in the Soroban Rust SDK and stellar-cli's snapshot testing is for the stellar-cli to download a history archive, which takes considerable time and requires significant bandwidth.

This issue is to explore improving the fork testing experience by having the SDK act independently to retrieve ledger entries from data sources as it needs them. That data source is not yet defined, but ideally it would be an RPC, or some RPC-compatible service, or possibly but not preferred continued use of history archives.

The way the SDK fork testing works today is:

sequenceDiagram
    user->>+cli: create snapshot with<br/>accountIDs and contract IDs
    cli->>+historyarchive: req full ledger
    historyarchive->>-cli: full ledger (GBs, slow)
    cli->>cli: filter ledger down to<br/>accountIDs and contract IDs<br/>(painful dev experience)
    cli->>-user: snapshot (KBs)
    
    user->>+sdk: write test importing snapshot for fork testing
    sdk->>sdk: loads snapshot and uses it as storage starting point
    sdk->>-user: 

Loading

The way it would ideally work if the SDK could independently load a stable ledger state from a data-source. RPC is listed as the data source, but imagine it could be any data-source that provides random access of a stable ledger state, via an RPC-compatible API:

sequenceDiagram
    user->>+sdk: write test fork testing ledger N
    loop 
    sdk->>sdk: test running
    alt local snapshot exists (cache)
    sdk->>sdk: load ledger entry for key K from snapshot
    else local snapshot does not exist
    sdk->>+rpc: req ledger entry for key K ledger N
    rpc->>-sdk: entry for K (Bytes)
    end
    end
    opt if new entries were downloaded
    sdk->>sdk: write local snapshot for starting point
    end
    sdk->>-user: 

Loading

This idea was first discussed here:

• Support forking from mainnet (or any target network) quickstart#625 (comment)

Dependent on:

• Serve recent historical state stellar-rpc#375 - This issue was opened to explore the possibility that RPC, or an RPC-compatible service, could serve stable recent historical state in a fashion that could be randomly accessed.
  • Add proof generation endpoint and Archival snapshot query support stellar-core#4397
    • Add state archival getledgerentry endpoint stellar-core#4623

Note this work is very much in the exploratory phase.

[leighmcculloch]

Todo:

• Memory profiling (Serve recent historical state stellar-rpc#375 (comment))
• Spike vertical experience