Fix BackupS3BlobCorrectness race condition with backup workers #12482

saintstack · 2025-10-20T22:13:39Z

The test was failing intermittently because checkAndDisableBackupWorkers() (called automatically after submitBackup) would disable backup workers before the snapshot could be created, causing restore_invalid_version errors.

Fixes:

Make test run longer
Set start of backup workers before submit
Check restorable before moving on.

saintstack · 2025-10-20T22:16:03Z

Here is running the single test only 100k times:

20251020-194415-stack-338befe4b127693b compressed=True data_size=55739272 duration=1811305 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:31:48 sanity=False started=100000 stopped=20251020-201603 submitted=20251020-194415 timeout=5400 username=stack

saintstack · 2025-10-20T22:22:42Z

Convert to draft till sure of this fix (100k runs of the test doesn't guarantee 100k of mixed tests will pass)

foundationdb-ci · 2025-10-20T22:25:08Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 5e47f8e
Duration 0:11:14
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T22:38:36Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 5e47f8e
Duration 0:24:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T22:54:58Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 5e47f8e
Duration 0:41:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T22:59:09Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 5e47f8e
Duration 0:45:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:04:23Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 5e47f8e
Duration 0:50:35
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:08:09Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 5e47f8e
Duration 0:54:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-20T23:25:30Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 5e47f8e
Duration 1:11:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-21T00:24:23Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: cf02d2c
Duration 0:29:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T00:36:38Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: cf02d2c
Duration 0:41:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T00:40:40Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: cf02d2c
Duration 0:45:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T00:50:33Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: cf02d2c
Duration 0:55:20
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T00:56:35Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: cf02d2c
Duration 1:01:21
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T00:59:10Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: cf02d2c
Duration 1:03:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T01:06:18Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: cf02d2c
Duration 1:11:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-21T05:25:04Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: aebfe72
Duration 0:28:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T05:37:14Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: aebfe72
Duration 0:41:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T05:40:51Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: aebfe72
Duration 0:44:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T05:48:42Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: aebfe72
Duration 0:52:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T06:02:59Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: aebfe72
Duration 1:06:42
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T06:05:39Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: aebfe72
Duration 1:09:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

The test was failing intermittently because checkAndDisableBackupWorkers() (called automatically after submitBackup) would disable backup workers before the snapshot could be created, causing restore_invalid_version errors. Fixes: - Re-enable backup workers immediately after submitBackup() to ensure snapshot creation completes - Set waitForQuiescenceBegin=false to avoid unnecessary quietDatabase() delay

…tBackup so backup agents don't go away

foundationdb-ci · 2025-10-21T22:18:43Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 1eb6639
Duration 1:07:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-21T22:25:15Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 1eb6639
Duration 0:41:31
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T22:28:34Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 1eb6639
Duration 0:44:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T22:29:20Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 1eb6639
Duration 0:45:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T22:41:48Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 1eb6639
Duration 0:23:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77

A general question on how to debug mock S3 tests: how can you check the files stored on mock S3 after the test failure? In the past, I've been using the backup files left on the disk to aid debugging. For mock S3, the files are cached in memory? Then it might be hard to debug its usage. Is there anyway to make it easy?

jzhou77 · 2025-10-21T22:43:59Z

fdbserver/workloads/BackupS3BlobCorrectness.actor.cpp

+				// Wait for backup to become restorable if it's still in progress
+				// This handles cases where cluster recoveries delay snapshot completion


Can you make this a separate actor for clarity?

jzhou77 · 2025-10-21T22:44:59Z

fdbserver/workloads/BackupS3BlobCorrectness.actor.cpp

 			                               true,
 			                               StopWhenDone{ !stopDifferentialDelay },
-			                               UsePartitionedLog::False,
+			                               UsePartitionedLog::True,


So if this was False, then we are using backup V1, which doesn't need backup workers. Why would the test fail?

I found that having it false, after calling submitBackup, when we came back, the backup workers were being shut down which disrupted snapshot range tasks... they wouldn't complete... No snapshot manifest was written so recovery would never run (If you have an alternative, I should try, I'd be happy to give it a go).

I just tried without setting UsePartitionedLog to true and removing the backup worker stuff and

20251022-020938-stack_backup_9-04e2387024d4e9d8 compressed=True data_size=55712465 duration=6193024 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:06:49 sanity=False started=100000 stopped=20251022-031627 submitted=20251022-020938 timeout=5400 username=stack_backup_9

foundationdb-ci · 2025-10-21T22:48:59Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 1eb6639
Duration 1:05:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T22:50:25Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 1eb6639
Duration 1:06:44
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-21T22:53:44Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 1eb6639
Duration 1:10:02
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T23:03:14Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 1eb6639
Duration 0:45:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T23:10:08Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 1eb6639
Duration 0:52:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T23:10:40Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 1eb6639
Duration 0:52:35
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-21T23:23:27Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 1eb6639
Duration 1:05:23
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack · 2025-10-21T23:25:39Z

A general question on how to debug mock S3 tests: how can you check the files stored on mock S3 after the test failure? In the past, I've been using the backup files left on the disk to aid debugging. For mock S3, the files are cached in memory? Then it might be hard to debug its usage. Is there anyway to make it easy?

Yeah. Mocks3 is memory based. Tests verify what was stored on s3 by variations on reading back what they wrote (in the s3client tests its explicit; in the backup tests its backup, clear, restore, and then verify the chain of keyvalues we've been writing). Up to this, if test fails, I've been looking at http back and forth headers or dumping content if necessary to figure where we went wrong. I can look at adding persistence. Intentionally I've avoided it up to this in effort at making sure we have constant read/write so tests can pass determinism checks.

…he UsePartitionedLog=True code

foundationdb-ci · 2025-10-22T03:42:19Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: de9e2ec
Duration 0:24:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T04:00:01Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: de9e2ec
Duration 0:42:23
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T04:03:34Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: de9e2ec
Duration 0:45:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T04:03:35Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: de9e2ec
Duration 0:45:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T04:21:47Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: de9e2ec
Duration 1:04:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T04:23:47Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: de9e2ec
Duration 1:06:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T04:26:37Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: de9e2ec
Duration 1:08:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack · 2025-10-22T05:05:08Z

Ran 250k

20251022-031848-stack_backup_9-04e2387024d4e9d8 compressed=True data_size=55712465 duration=11734990 ended=250000 fail=2 fail_fast=10 max_runs=250000 pass=249998 priority=100 remaining=0 runtime=1:23:47 sanity=False started=250000 stopped=20251022-044235 submitted=20251022-031848 timeout=5400 username=stack_backup_9

Two failures:

MetaclusterManagement.toml Timed out after 2100 seconds with an assertion failure in MetaclusterManagementWorkload.actor.cpp:776
BackupS3BlobCorrectness.toml w/ seed: 2574247734 ... but it passes when run standalone. Looking...

foundationdb-ci · 2025-10-22T15:58:14Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: efdbbb8
Duration 0:11:40
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T16:10:20Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: efdbbb8
Duration 0:23:43
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T16:30:08Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: efdbbb8
Duration 0:43:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T16:31:20Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: efdbbb8
Duration 0:44:44
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack marked this pull request as draft October 20, 2025 22:22

saintstack marked this pull request as ready for review October 21, 2025 00:19

saintstack marked this pull request as draft October 21, 2025 00:33

saintstack force-pushed the BackupS3BlobCorrectness branch from cf02d2c to aebfe72 Compare October 21, 2025 04:56

michael stack added 4 commits October 21, 2025 10:33

Wait on backup complete state to be written and use transaction submi…

6668668

…tBackup so backup agents don't go away

Simple cluster config.

81c07ec

Reenable BackupS3BlobCorrectness test

f56c3d9

saintstack closed this Oct 21, 2025

saintstack reopened this Oct 21, 2025

jzhou77 reviewed Oct 21, 2025

View reviewed changes

Break out the wait on completion as standalone actor and then strip t…

de9e2ec

…he UsePartitionedLog=True code

Fix for flakey early test exit

efdbbb8

		// Wait for backup to become restorable if it's still in progress
		// This handles cases where cluster recoveries delay snapshot completion

Fix BackupS3BlobCorrectness race condition with backup workers #12482

Are you sure you want to change the base?

Fix BackupS3BlobCorrectness race condition with backup workers #12482

Conversation

saintstack commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saintstack commented Oct 20, 2025

Uh oh!

saintstack commented Oct 20, 2025

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 20, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

saintstack commented Oct 20, 2025 •

edited

Loading