test(dursto): add the machinery for a long-running Database test

[victor-dumitrescu]

Relates to RV-984.

What

Adds the components required for running long end to end tests over Database. Similar to the existing integration tests, the long test runs randomly sampled operations over multiple instances of Database and a reference model and checks for correctness and consistency. The main differences are:

• The reference model also informs the key strategy, in order to make sure that hot keys and recently deleted keys are also generated in addition to fresh keys.
• The test is applied over 3 Database instances: In addition to the 2 traced databases, it also tracks a "production" Database<PersistenceLayer, Normal>.
• The test is split into epochs, which are self-contained property-based tests. At the end of an epoch, one of the test cases is used to advance the state for the next epoch. In case of a test failure, everything needed to reproduce is saved to disk, so only the last epoch needs to be replayed, not the full test.

For now, a short version of this test is run as part of make test.

• test(dursto): add a standalone long tests binary which runs in CI #1085 packages it as a separate binary and adds PR check which runs the test for 10 minutes
• test(dursto): add snapshots pruning option for long tests #1090
• Next, it will also run as a nightly test.

Why

To gain a higher degree of confidence in the correctness and consistency of the durable storage.

How

Most of the functionality is added in durable-storage/src/long_test. In addtion, the in-memory repo is now serialisable with rkyv so that an in-memory database can be committed to disk and restored for failure replay.

Manually Testing

make test

To see a printed log of the test:

cargo test database_long_test_restricted -- --no-capture

Tasks for the Author

• Link all Linear issues related to this MR using magic words (e.g. part of, relates to, closes).
• Eliminate dead code and other spurious artefacts introduced in your changes.
• Document new public functions, methods and types.
• Make sure the documentation for updated functions, methods, and types is correct.
• Add tests for bugs that have been fixed.
• Explain changes to regression test captures when applicable.
• Write commit messages in agreement with our guidelines.
• Self-review your changes to ensure they are high-quality.
• Complete all of the above before assigning this MR to reviewers.

[codecov]

Codecov Report

❌ Patch coverage is 87.39837% with 93 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (acb364f) to head (9152d8a).

Files with missing lines	Patch %	Lines
durable-storage/src/long_test/mod.rs	78.75%	32 Missing and 33 partials ⚠️
durable-storage/src/storage/in_memory.rs	82.65%	10 Missing and 7 partials ⚠️
durable-storage/src/long_test/run_case.rs	94.87%	6 Missing ⚠️
durable-storage/src/long_test/model.rs	92.30%	3 Missing and 1 partial ⚠️
durable-storage/src/test_helpers.rs	99.01%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1084      +/-   ##
==========================================
- Coverage   90.17%   90.14%   -0.04%     
==========================================
  Files         130      134       +4     
  Lines       28394    29064     +670     
  Branches    28394    29064     +670     
==========================================
+ Hits        25605    26199     +594     
- Misses       2025     2056      +31     
- Partials      764      809      +45     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Benchmark results for revision ce1652b:

Metric	Duration	TPS
Mean	1.471670698s	27.183
Worst	1.533830142s	26.079
Best	1.462770797s	27.345
Standard Deviation	±14.517017ms	±0.258

Full results

Run	Transfers	Duration	TPS
1	40	1.474205403s	27.133
2	40	1.533830142s	26.079
3	40	1.468887757s	27.231
4	40	1.469622783s	27.218
5	40	1.468042697s	27.247
6	40	1.471847882s	27.177
7	40	1.470699158s	27.198
8	40	1.462770797s	27.345
9	40	1.471107055s	27.190
10	40	1.465892259s	27.287
11	40	1.467249649s	27.262
12	40	1.468824417s	27.233
13	40	1.466882675s	27.269
14	40	1.468889609s	27.231
15	40	1.466464229s	27.276
16	40	1.467466686s	27.258
17	40	1.465683087s	27.291
18	40	1.464986719s	27.304
19	40	1.467143742s	27.264
20	40	1.472917217s	27.157

Compare the results above with those for the default branch.