An exception during runs with many instances of suborchestrators and activities

[pascalnaber]

We are using durable functions and start thousands of suborchestrators and many more activities.
This was never a problem. But all of a sudden we get exceptions and we don't understand why.

The result is problematic. Messages are picked up multiple times. So we get a lot more results then we expected.
Right before the exception we always see this warning.
How should we deal with this exception?
The exception itself doesn't say a lot. What result was received? What result was expected? We don't know.

We are using nuget 2.9.1

[Warning]   98aa9fe85fd14e25b3b72f5aa6afe66c:78: Split brain detected: Another worker already updated the history for this instance - the 4 history event result(s) will be discarded
2023-02-22T21:35:23Z   [Error]   An unexpected failure occurred while processing instance '98aa9fe85fd14e25b3b72f5aa6afe66c:78': DurableTask.AzureStorage.Storage.DurableTaskStorageException: Element 4 in the batch returned an unexpected response code.
 ---> Microsoft.WindowsAzure.Storage.StorageException: Element 4 in the batch returned an unexpected response code.
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
Request Information
RequestID:4c24328a-f002-0074-5b05-47a70a000000
RequestDate:Wed, 22 Feb 2023 21:35:21 GMT
StatusMessage:4:The update condition specified in the request was not satisfied.
ErrorCode:
ErrorMessage:4:The update condition specified in the request was not satisfied.
RequestId:4c24328a-f002-0074-5b05-47a70a000000
Time:2023-02-22T21:35:22.7062321Z

   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 140
   at DurableTask.AzureStorage.Storage.Table.ExecuteBatchAsync(TableBatchOperation batchOperation, String batchType) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 172
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UploadHistoryBatch(String instanceId, String sanitizedInstanceId, String executionId, TableBatchOperation historyEventBatch, StringBuilder historyEventNamesBuffer, Int32 numberOfTotalEvents, Int32 episodeNumber, Int32 estimatedBatchSizeInBytes, String eTagValue, Boolean isFinalBatch) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 1290
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, String eTagValue) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 1086
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1055
2023-02-22T21:35:23Z   [Error]   TaskOrchestrationDispatcher-5d086598f53342da931c5df3738df545-0: Unhandled exception with work item '98aa9fe85fd14e25b3b72f5aa6afe66c:78': DurableTask.AzureStorage.Storage.DurableTaskStorageException: Element 4 in the batch returned an unexpected response code.
 ---> Microsoft.WindowsAzure.Storage.StorageException: Element 4 in the batch returned an unexpected response code.
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
Request Information
RequestID:4c24328a-f002-0074-5b05-47a70a000000
RequestDate:Wed, 22 Feb 2023 21:35:21 GMT
StatusMessage:4:The update condition specified in the request was not satisfied.
ErrorCode:
ErrorMessage:4:The update condition specified in the request was not satisfied.
RequestId:4c24328a-f002-0074-5b05-47a70a000000
Time:2023-02-22T21:35:22.7062321Z

   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 140
   at DurableTask.AzureStorage.Storage.Table.ExecuteBatchAsync(TableBatchOperation batchOperation, String batchType) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 172
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UploadHistoryBatch(String instanceId, String sanitizedInstanceId, String executionId, TableBatchOperation historyEventBatch, StringBuilder historyEventNamesBuffer, Int32 numberOfTotalEvents, Int32 episodeNumber, Int32 estimatedBatchSizeInBytes, String eTagValue, Boolean isFinalBatch) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 1290
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, String eTagValue) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 1086
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1055
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 551
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 194
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

Backing off for 10 seconds until 5 successful operations
2023-02-22T21:35:23Z   [Warning]   98aa9fe85fd14e25b3b72f5aa6afe66c:78: Abandoning [ExecutionStarted] message back to coda15-control-01 and setting a visibility delay of 16ms 

[pascalnaber]

We can reproduce this exception.
For now we work around it by passing less data to suborchestrators.
It looks like the amount of memory being used was to high (because of the amount of data we passed). Although I'm only assuming this. There are no warnings or information in the logging about this.

[cgillum]

Split-brain errors aren't something you can mitigate yourself. They are an internal error that often indicate something unexpected happened. For example, a critical failure that occurs in the middle of a storage checkpoint that causes state to be partially committed can result in this if some of that state included scheduling sub-orchestrations. Out-of-memory errors can be one example. However, the system should be able to automatically recover from these. It may result in duplicate executions, however, which I assume you are seeing.

If you search further back in time (no more than 10 minutes), are there any other warnings or errors that you see?