Add option to use CATCHUP_SKIP_KNOWN_RESULTS_FOR_TESTING in PubnetParallelCatchupV2

[ThomasBrady]

Overview

• Adds --catchup-use-known-results-for-testing cli flag to supercluster, which results PubnetParallelCatchupV2 running with CATCHUP_SKIP_KNOWN_RESULTS_FOR_TESTING=true in its config
• Adds worker.catchup_skip_known_results_for_testing kube set option, which is set to true if supercluster was run with --catchup-use-known-results-for-testing
• The catchup workers config volume is sourced from stellar-core-config configMap
• if worker.use_known_results_config is true, the data of the stellar-core-config configMap includes CATCHUP_SKIP_KNOWN_RESULTS_FOR_TESTING = true

Testing:

• Currently, the catchup worker invocation is set to --ll=INFO to allow for log message which validate results are downloaded and used for catchup
• Locally I have run PubnetParallelCatchupV2 via helm and been able to see the --use_known_results_config flag resulting in Results being downloaded via logs

stellar_core_image: docker-registry.services.stellar-ops.com/dev/stellar-core:22.0.1-2178.fb725ab9b.focal-do-not-use-in-prd-buildtests
  use_known_results_config: true

• Next step is to test via the supercluster test pipeline with my changes to stellar-supercluster

Future

It would be nice to have a dynamic (or templated) generation of config files for this mission. We use a TOML library in the other missions to generate config files, perhaps we can do something similar for PubnetParallelCatchupV2?

[jayz22]

This approach is okay for now. But I think after the experimental phase is done we should properly move this option to stellar core command line.
Conditionally appending options to the configmap can get out of hand quickly and be hard to maintain.
Let's not forget to create a followup issue for that.

[ThomasBrady]

I was able to run with this option enabled in the test-stellar-supercluster pipeline https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-supercluster-test/81/

[jayz22]

I was able to run with this option enabled in the test-stellar-supercluster pipeline https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-supercluster-test/81/

Looks good, do you want to switch the default flag to false as @anupsdf suggested, or do that on the Jenkins side?

[ThomasBrady]

I was able to run with this option enabled in the test-stellar-supercluster pipeline https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-supercluster-test/81/

Looks good, do you want to switch the default flag to false as @anupsdf suggested, or do that on the Jenkins side?

Thanks. I just pushed a change to make this default to true. I also have this PR to update stellar-supercluster.

Do you have permissions to edit the Jenkins schedule to ensure we do a once weekly non-skipping catchup run?

[jayz22]

I was able to run with this option enabled in the test-stellar-supercluster pipeline https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-supercluster-test/81/

Looks good, do you want to switch the default flag to false as @anupsdf suggested, or do that on the Jenkins side?

Thanks. I just pushed a change to make this default to true. I also have this PR to update stellar-supercluster.

Do you have permissions to edit the Jenkins schedule to ensure we do a once weekly non-skipping catchup run?

No you need to set it up. There is also the policy question of when and how we use this mode (which is separate from this PR). I'm not sure if we have a time-based CI job currently.

Just want to double check, are we sure about setting default to true. Or should we instead have this option be opted-in, i.e. configured from outside. (which is what @MonsieurNicolas has suggested at the beginning). Since that issue is closed I suppose all the work mentioned has been completed and it is okay to turn this on by default?

[ThomasBrady]

I was able to run with this option enabled in the test-stellar-supercluster pipeline https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-supercluster-test/81/

Looks good, do you want to switch the default flag to false as @anupsdf suggested, or do that on the Jenkins side?

Thanks. I just pushed a change to make this default to true. I also have this PR to update stellar-supercluster.
Do you have permissions to edit the Jenkins schedule to ensure we do a once weekly non-skipping catchup run?

No you need to set it up. There is also the policy question of when and how we use this mode (which is separate from this PR). I'm not sure if we have a time-based CI job currently.

Just want to double check, are we sure about setting default to true. Or should we instead have this option be opted-in, i.e. configured from outside. (which is what @MonsieurNicolas has suggested at the beginning). Since that issue is closed I suppose all the work mentioned has been completed and it is okay to turn this on by default?

The linked issue is referencing stellar-core catchup. This mode is opt-in in stellar-core (in fact one can only opt-in when running catchup on a test build). As discussed with Anup in standup this morning we want all supercluster runs to use this mode by default, with a less frequent weekly non-skip ParallelCatchupV2 run. Based on that, I think it makes sense to default this mode being enabled in our supercluster runs.

[jayz22]

The linked issue is referencing stellar-core catchup. This mode is opt-in in stellar-core (in fact one can only opt-in when running catchup on a test build). As discussed with Anup in standup this morning we want all supercluster runs to use this mode by default, with a less frequent weekly non-skip ParallelCatchupV2 run. Based on that, I think it makes sense to default this mode being enabled in our supercluster runs.

Sounds good, thanks