test(dursto): add a standalone long tests binary which runs in CI

[victor-dumitrescu]

Relates to RV-984.

What

Adds a standalone binary for the Database long test with 2 commands:

• test to start a test with the given parameters
• replay to replay a failing epoch recorded by a previous test run at a given path

It also adds a new Durable Storage Long Tests pull request job which runs the long test for 10 minutes. I chose this time limit so that, given parallel execution, the overall time to complete a PR CI run is not impacted.

Next:

• test(dursto): add snapshots pruning option for long tests #1090
• test(dursto): run Database long test nightly #1092

Why

Binary added so that the long test can be deployed independently.

Manually Testing

To run a short test through the binary:

cargo run --release --features rocksdb,unstable-test-utils --bin database_long_test -- test --epochs 10 --ops-per-epoch 100 --cases-per-epoch 20

To test the replay command you need to generate a failure first, for example by changing behaviour for one database instance but not the others in durable-storage/src/long_test/run_case.rs:

         check_and_apply_value_operation(&mut targets.in_memory_db, &targets.model, op);
         check_and_apply_value_operation(&mut targets.persistent_db, &targets.model, op);
-        check_and_apply_value_operation(&mut targets.production_db, &targets.model, op);
+        if !prove {
+            check_and_apply_value_operation(&mut targets.production_db, &targets.model, op);
+        }

Running the test again should now result in a failure and the command to replay it should be printed:

failure artifacts written to XXXX
replay with:
cargo run --release --features rocksdb,unstable-test-utils --bin database_long_test -- replay XXXX

Tasks for the Author

• Link all Linear issues related to this MR using magic words (e.g. part of, relates to, closes).
• Eliminate dead code and other spurious artefacts introduced in your changes.
• Document new public functions, methods and types.
• Make sure the documentation for updated functions, methods, and types is correct.
• Add tests for bugs that have been fixed.
• Explain changes to regression test captures when applicable.
• Write commit messages in agreement with our guidelines.
• Self-review your changes to ensure they are high-quality.
• Complete all of the above before assigning this MR to reviewers.

[codecov]

Codecov Report

❌ Patch coverage is 26.38889% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.95%. Comparing base (39f076e) to head (7325f47).

Files with missing lines	Patch %	Lines
durable-storage/src/long_test/mod.rs	40.42%	18 Missing and 10 partials ⚠️
durable-storage/src/bin/database_long_test.rs	0.00%	25 Missing ⚠️

❌ Your patch check has failed because the patch coverage (26.38%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1085      +/-   ##
==========================================
- Coverage   90.14%   89.95%   -0.20%     
==========================================
  Files         134      135       +1     
  Lines       29155    29217      +62     
  Branches    29155    29217      +62     
==========================================
  Hits        26282    26282              
- Misses       2058     2109      +51     
- Partials      815      826      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Benchmark results for revision 6ee73e4:

Metric	Duration	TPS
Mean	1.469866878s	27.213
Worst	1.478257983s	27.059
Best	1.464837872s	27.307
Standard Deviation	±3.172118ms	±0.059

Full results

Run	Transfers	Duration	TPS
1	40	1.473824759s	27.140
2	40	1.46954991s	27.219
3	40	1.474316551s	27.131
4	40	1.478257983s	27.059
5	40	1.46902629s	27.229
6	40	1.471620778s	27.181
7	40	1.464837872s	27.307
8	40	1.470426098s	27.203
9	40	1.470077746s	27.209
10	40	1.465743735s	27.290
11	40	1.47105497s	27.191
12	40	1.465955489s	27.286
13	40	1.4732362s	27.151
14	40	1.467339187s	27.260
15	40	1.467803981s	27.252
16	40	1.467121195s	27.264
17	40	1.469604209s	27.218
18	40	1.469450689s	27.221
19	40	1.468337756s	27.242
20	40	1.469752172s	27.215

Compare the results above with those for the default branch.