Replace '...' with actual random data in simulation workloads (#12221)

[akankshamahajan15]

Cherry-pick #12221

• Replace '...' with actual data

• Add zeroPaddingRatio Knob in simulation workloads.

• Refactor randomValue API to include in KVWorkload

• Addressed comments - Add const and remove valueString - not needed

100k Correctness tests Running:

20250716-203553-ak_7.4_randomValue-17b9377078b84932 compressed=True data_size=41224799 duration=2287 ended=71 fail_fast=10 max_runs=100000 pass=71 priority=100 remaining=1 day, 22:31:26 runtime=0:01:59 sanity=False started=180 submitted=20250716-203553 timeout=5400 username=ak_7.4_randomValue

---

Replace this text with your description here...

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 792e70d
• Duration 0:38:05
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 792e70d
• Duration 0:49:21
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 792e70d
• Duration 1:01:49
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 792e70d
• Duration 1:02:58
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 792e70d
• Duration 1:15:47
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 792e70d
• Duration 1:17:20
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[spraza]

Actually CI is showing joshua failures:

[Container] 2025/07/16 21:48:43.218591 Running command python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID}
  20250716-211551-pr12254-792e70de-12380-6e02b845a2401658 compressed=True data_size=60353788 duration=511607 ended=8125 fail=10 fail_fast=10 max_runs=10000 pass=8115 priority=100 remaining=0 runtime=0:32:52 sanity=False started=9324 stopped=20250716-214843 submitted=20250716-211551 timeout=5400 username=pr12254-792e70de-12380

Due to some issue, I can not see the actual failures, so hard to trust the summary.

Looks like @akankshamahajan15 ran 100K against the change in 7.4 branch: 20250716-203553-ak_7.4_randomValue-17b9377078b84932. The results for those are:

20250716-203553-ak_7.4_randomValue-17b9377078b84932 compressed=True data_size=41224799 duration=5468606 ended=100000 fail=4 fail_fast=10 max_runs=100000 pass=99996 priority=100 remaining=0 runtime=1:22:57 sanity=False started=100000 stopped=20250716-215850 submitted=20250716-203553 timeout=5400 username=ak_7.4_randomValue

Again, some joshua issue, but the only failure from above run is tests/fast/BackupToDBCorrectness.toml with ssd-2 storage engine.

Akanksha, can you confirm the issue is not related to your PR? We should also check if the CI failing fast at 10 so quickly is expected.

[akankshamahajan15]

Looks like @akankshamahajan15 ran 100K against the change in 7.4 branch: 20250716-203553-ak_7.4_randomValue-17b9377078b84932. The results for those are:

20250716-203553-ak_7.4_randomValue-17b9377078b84932 compressed=True data_size=41224799 duration=5468606 ended=100000 fail=4 fail_fast=10 max_runs=100000 pass=99996 priority=100 remaining=0 runtime=1:22:57 sanity=False started=100000 stopped=20250716-215850 submitted=20250716-203553 timeout=5400 username=ak_7.4_randomValue

Again, some joshua issue, but the only failure from above run is tests/fast/BackupToDBCorrectness.toml with ssd-2 storage engine.

Akanksha, can you confirm the issue is not related to your PR? We should also check if the CI failing fast at 10 so quickly is expected.

Yes, it shows me one failure in BackupToDBCorrectness.toml whereas command says fail=4.
This PR didn't make any changes related to BackupToDBCorrectness.toml. I reran that test locally with same seed with 7.4 (without my changes) and it's failing.

For CI I don't see any logs as to why it failed.

[Container] 2025/07/16 21:12:55.278877 Running command ENSEMBLE_ID=$(python3 -m joshua.joshua list | awk -v username=${CORRECTNESS_USERNAME} '$0~username {print $1}')

[Container] 2025/07/16 21:12:55.575427 Running command python3 -m joshua.joshua tail --errors --xml ${ENSEMBLE_ID}
Results for test ensemble: 20250716-211252-pr12254-clang-792e70de-1233-77362437c642530d
<Trace>Ensemble stopped
</Trace>

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 1ca0c28
• Duration 0:50:17
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 0:56:51
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 0:58:54
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 1:11:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[akankshamahajan15]

The new run is passing and Michael is looking at why logs are not showing up in CI

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 1ca0c28
• Duration 0:49:37
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 0:58:59
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 1:03:10
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 3:00:04
• Result: ❌ FAILED
• Error: Build has timed out.
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 1ca0c28
• Duration 0:38:11
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 1ca0c28
• Duration 0:48:48
• Result: ❌ FAILED
• Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 1ca0c28
• Duration 1:01:15
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 1:01:34
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 1:02:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 1ca0c28
• Duration 1:03:12
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[akankshamahajan15]

Result of foundationdb-pr-clang on Linux RHEL 9 Failure (Not related to this PR)

RandomSeed="3361421525" SourceVersion="1ca0c2837b7f03602a0e2b10fc6c2b91e4d97d0d" Time="1753065783" BuggifyEnabled="0" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/restarting/from_7.3.0/SnapTestSimpleRestart-2.toml"

Output="Assertion res.empty() failed @ /codebuild/output/src2572302754/src/github.com/apple/foundationdb/fdbserver/workloads/RandomRangeLock.actor.cpp 195:"/

[akankshamahajan15]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

42/67 Test #45: fdb_c_api_test_CApiCorrectnessCallbacksOnExtThr .....................***Failed    4.09 sec

CApiCorrectnessCallbacksOnExtThr.toml

No errors found in logs

I don't see any error in the logs for this test.

[spraza]

Result of foundationdb-pr-clang on Linux RHEL 9 Failure (Not related to this PR)

RandomSeed="3361421525" SourceVersion="1ca0c2837b7f03602a0e2b10fc6c2b91e4d97d0d" Time="1753065783" BuggifyEnabled="0" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/restarting/from_7.3.0/SnapTestSimpleRestart-2.toml"

Output="Assertion res.empty() failed @ /codebuild/output/src2572302754/src/github.com/apple/foundationdb/fdbserver/workloads/RandomRangeLock.actor.cpp 195:"/

This is a known issue. I am fixing it.

[spraza]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

42/67 Test #45: fdb_c_api_test_CApiCorrectnessCallbacksOnExtThr .....................***Failed    4.09 sec

CApiCorrectnessCallbacksOnExtThr.toml

No errors found in logs

I don't see any error in the logs for this test.

Discussed offline. This should not be related to this PR. This test passed in the PR earlier and since then we have not made any changes.