[DO NOT MERGE] Test concurrent mocks3 ctests #12480

saintstack · 2025-10-20T20:13:06Z

See the .md file included with this PR for how to run the test. Here is a run with ten concurrent s3client runs:

 ./fdbclient/tests/simple_concurrent_test.sh
2025-10-20T13:09:00-07:00 Testing concurrent s3client execution
2025-10-20T13:09:00-07:00 Starting 10 concurrent instances...
2025-10-20T13:09:00-07:00 Starting instance 1 with scratch: /tmp/s3test_1.A3Kj6i
2025-10-20T13:09:00-07:00 Instance 1 started with PID 1189
2025-10-20T13:09:01-07:00 Starting instance 2 with scratch: /tmp/s3test_2.ciEgav
2025-10-20T13:09:01-07:00 Instance 2 started with PID 1210
2025-10-20T13:09:02-07:00 Starting instance 3 with scratch: /tmp/s3test_3.BUOve0
2025-10-20T13:09:02-07:00 Instance 3 started with PID 1228
2025-10-20T13:09:03-07:00 Starting instance 4 with scratch: /tmp/s3test_4.UFhfY5
2025-10-20T13:09:03-07:00 Instance 4 started with PID 1266
2025-10-20T13:09:04-07:00 Starting instance 5 with scratch: /tmp/s3test_5.TdTL2s
2025-10-20T13:09:04-07:00 Instance 5 started with PID 1350
2025-10-20T13:09:05-07:00 Starting instance 6 with scratch: /tmp/s3test_6.QzVQws
2025-10-20T13:09:05-07:00 Instance 6 started with PID 1451
2025-10-20T13:09:06-07:00 Instance 1 completed successfully
2025-10-20T13:09:06-07:00 Starting instance 7 with scratch: /tmp/s3test_7.S919BO
2025-10-20T13:09:06-07:00 Instance 7 started with PID 1608
2025-10-20T13:09:07-07:00 Starting instance 8 with scratch: /tmp/s3test_8.FHzNFv
2025-10-20T13:09:07-07:00 Instance 8 started with PID 1695
2025-10-20T13:09:08-07:00 Instance 2 completed successfully
2025-10-20T13:09:08-07:00 Starting instance 9 with scratch: /tmp/s3test_9.xtLBTr
2025-10-20T13:09:08-07:00 Instance 9 started with PID 1800
2025-10-20T13:09:10-07:00 Starting instance 10 with scratch: /tmp/s3test_10.I8BLxW
2025-10-20T13:09:10-07:00 Instance 10 started with PID 1952
2025-10-20T13:09:10-07:00 Instance 3 completed successfully
2025-10-20T13:09:11-07:00 All instances started, waiting for completion...
2025-10-20T13:09:11-07:00 Instance 1 (PID 1189) completed successfully
2025-10-20T13:09:11-07:00 Instance 2 (PID 1210) completed successfully
2025-10-20T13:09:11-07:00 Instance 3 (PID 1228) completed successfully
2025-10-20T13:09:11-07:00 Instance 6 completed successfully
2025-10-20T13:09:12-07:00 Instance 4 completed successfully
2025-10-20T13:09:12-07:00 Instance 4 (PID 1266) completed successfully
2025-10-20T13:09:13-07:00 Instance 7 completed successfully
2025-10-20T13:09:14-07:00 Instance 5 completed successfully
2025-10-20T13:09:14-07:00 Instance 5 (PID 1350) completed successfully
2025-10-20T13:09:14-07:00 Instance 6 (PID 1451) completed successfully
2025-10-20T13:09:14-07:00 Instance 7 (PID 1608) completed successfully
2025-10-20T13:09:16-07:00 Instance 8 completed successfully
2025-10-20T13:09:16-07:00 Instance 8 (PID 1695) completed successfully
2025-10-20T13:09:18-07:00 Instance 9 completed successfully
2025-10-20T13:09:18-07:00 Instance 9 (PID 1800) completed successfully
2025-10-20T13:09:20-07:00 Instance 10 completed successfully
2025-10-20T13:09:20-07:00 Instance 10 (PID 1952) completed successfully
2025-10-20T13:09:20-07:00 Checking for port conflicts in logs...
2025-10-20T13:09:20-07:00 Instance 1: No port conflicts detected
2025-10-20T13:09:20-07:00 Instance 2: Port conflict detected and handled
Port 8080 already in use, trying next port
MockS3Server ready on port 8081
2025-10-20T13:09:20-07:00 Instance 3: Port conflict detected and handled
Port 8080 already in use, trying next port
Port 8081 already in use, trying next port
MockS3Server ready on port 8082
2025-10-20T13:09:20-07:00 Instance 4: Port conflict detected and handled
Port 8080 already in use, trying next port
Port 8081 already in use, trying next port
Port 8082 already in use, trying next port
MockS3Server ready on port 8083
2025-10-20T13:09:20-07:00 Instance 5: Port conflict detected and handled
Port 8080 already in use, trying next port
Port 8081 already in use, trying next port
Port 8082 already in use, trying next port
Port 8083 already in use, trying next port
MockS3Server ready on port 8084
2025-10-20T13:09:20-07:00 Instance 6: No port conflicts detected
2025-10-20T13:09:20-07:00 Instance 7: Port conflict detected and handled
Port 8080 already in use, trying next port
MockS3Server ready on port 8081
2025-10-20T13:09:20-07:00 Instance 8: Port conflict detected and handled
Port 8080 already in use, trying next port
Port 8081 already in use, trying next port
MockS3Server ready on port 8082
2025-10-20T13:09:20-07:00 Instance 9: Port conflict detected and handled
Port 8080 already in use, trying next port
Port 8081 already in use, trying next port
Port 8082 already in use, trying next port
MockS3Server ready on port 8083
2025-10-20T13:09:20-07:00 Instance 10: Port conflict detected and handled
Port 8080 already in use, trying next port
Port 8081 already in use, trying next port
Port 8082 already in use, trying next port
Port 8083 already in use, trying next port
MockS3Server ready on port 8084
2025-10-20T13:09:20-07:00 Test completed!

foundationdb-ci · 2025-10-20T20:37:34Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: ff118fc
Duration 0:24:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T20:53:13Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: ff118fc
Duration 0:39:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T20:59:19Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: ff118fc
Duration 0:46:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T21:01:48Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: ff118fc
Duration 0:48:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T21:09:42Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: ff118fc
Duration 0:56:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T21:18:08Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: ff118fc
Duration 1:04:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T21:21:31Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: ff118fc
Duration 1:08:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack · 2025-10-20T22:59:02Z

Added new script to run ten concurrent tests of the s3client ctest.... up to 1k times...

...

2025-10-20T15:14:44-07:00 🎉 SUCCESS: ALL 10,000 TESTS COMPLETED!
2025-10-20T15:14:44-07:00 ===== FINAL PROOF OF 10K COMPLETION =====
Start time: Mon Oct 20 14:47:24 PDT 2025
End time: Mon Oct 20 15:14:44 PDT 2025
Total duration: 1640 seconds (27 minutes)
Tests completed: 1000
Tests failed: 0
Port conflicts handled: 900
Success rate: 100%
Batches completed: 100

Log files created:     1000
Expected: 1000
✅ VERIFIED: Exactly 1000 test logs exist
✅ Perfect cleanup - no orphaned processes

PROOF FILE: /tmp/s3client_10k_proof.txt
LOG FILES: /tmp/test10k_*.log
PORT CONFLICT CHECK: grep -l 'Port.*already in use' /tmp/test10k_*.log | wc -l

foundationdb-ci · 2025-10-20T23:10:00Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 613b0b4
Duration 0:11:34
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:22:37Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 613b0b4
Duration 0:24:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:39:49Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 613b0b4
Duration 0:41:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:42:49Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 613b0b4
Duration 0:44:21
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:48:35Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 613b0b4
Duration 0:50:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:51:40Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 613b0b4
Duration 0:53:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T00:03:29Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 613b0b4
Duration 1:05:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

gxglass

quick review. I guess the proof is in the pudding

saintstack · 2025-10-21T18:58:43Z

Script to run the bigger backup ctest. Here is running it 1k times w/ parallelism of 3 (its heavyweight test)

...
[2025-10-21 11:43:11] :white_check_mark: Test 998 PASSED (998/1000)
[2025-10-21 11:43:11] Started test 1000/1000 (PID: 66842)
[2025-10-21 11:43:55] :white_check_mark: Test 999 PASSED (999/1000)
[2025-10-21 11:43:55] Waiting for final 1 tests to complete...
[2025-10-21 11:44:09] :white_check_mark: Test 1000 PASSED (1000/1000)
[2025-10-21 11:44:09] Progress: 1000/1000 completed (1 tests/minute)
[2025-10-21 11:44:09] :tada: ALL 1000 TESTS COMPLETED SUCCESSFULLY! :tada:
[2025-10-21 11:44:09] Total time: 13h 21m 35s
[2025-10-21 11:44:09] Success rate: 100%
[2025-10-21 11:44:09] Parallelism: 2 concurrent tests
[2025-10-21 11:44:09] Proof written to: /tmp/s3backup_1k_parallel_proof.txt

foundationdb-ci · 2025-10-21T19:29:09Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: d5eaa85
Duration 0:31:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T19:43:39Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: d5eaa85
Duration 0:46:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T19:59:16Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 1c14157
Duration 0:24:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T19:59:41Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: d5eaa85
Duration 1:02:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:00:03Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: d5eaa85
Duration 1:02:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:02:05Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: d5eaa85
Duration 1:04:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:13:21Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: d5eaa85
Duration 1:16:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-21T20:14:57Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 1c14157
Duration 0:40:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:19:50Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 1c14157
Duration 0:45:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:25:02Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 1c14157
Duration 0:50:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:26:37Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 1c14157
Duration 0:52:16
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T20:41:18Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 1c14157
Duration 1:06:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack changed the title ~~Test concurrency script~~ Test concurrent mocks3 ctests Oct 20, 2025

saintstack marked this pull request as draft October 20, 2025 20:14

saintstack changed the title ~~Test concurrent mocks3 ctests~~ [DO NOT MERGE] Test concurrent mocks3 ctests Oct 20, 2025

saintstack requested a review from gxglass October 20, 2025 20:16

gxglass approved these changes Oct 21, 2025

View reviewed changes

stack added 2 commits October 21, 2025 08:25

Test concurrency script

9274b15

Add new tests

d5eaa85

saintstack force-pushed the ctest_concurrency branch from 613b0b4 to d5eaa85 Compare October 21, 2025 18:57

Include this script for backup

1c14157

Uh oh!

[DO NOT MERGE] Test concurrent mocks3 ctests #12480

Are you sure you want to change the base?

[DO NOT MERGE] Test concurrent mocks3 ctests #12480

Uh oh!

Conversation

saintstack commented Oct 20, 2025

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

saintstack commented Oct 20, 2025

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

gxglass left a comment

Choose a reason for hiding this comment

Uh oh!

saintstack commented Oct 21, 2025

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x