Orchestrations are forced to wait for non-awaited tasks (forced fan-in)

[vgpro54321]

Description

If orchestration A starts awaits sub-orchestration B which in turn starts asynchronous operation C (without await fire-and-forget style), await for sub orchestration B returns no earlier than operation C finishes. This makes it impossible to start independent fire-and forget orchestrations

Example:

Orchestration A

await context.CallSubOrchestration("B");
logger.LogInformation("Ok, we are done");

Orchestration B

// No await here, just scheduling fire and forget task 
_ = context.CallSubOrchestration("C")


Orchestration C
{
  await context.CreateTimer(TimeSpan.FromMinutes(1));
}

"We re done" will not be printed until after 1 minute even though Orchestraion B is not blocked by Operation C nominally.

Expected behavior

"We re done" in the example should be printed immediately after Orchestration B finished executing body.

Actual behavior

Please see example.

Relevant source code snippets

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.Logging;

namespace DurableFuncExperiment;

public static class FireAndForgetFunctions
{
    [Function(nameof(FireAndForgetFunctions))]
    public static async Task RunOrchestrator(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        await ValueTask.CompletedTask;

        ILogger logger = context.CreateReplaySafeLogger(nameof(FireAndForgetFunctions));
        logger.LogInformation("Saying hello.");
        var outputs = new List<string>();

        await context.CallSubOrchestratorAsync(nameof(FireAndForgetFunctionsNonAwaiter), "Tokyo");

        logger.LogInformation("FireAndForgetFunctionsNonAwaiter fully completed.");

    }

    [Function(nameof(FireAndForgetFunctionsNonAwaiter))]
    public static async Task FireAndForgetFunctionsNonAwaiter([OrchestrationTrigger] TaskOrchestrationContext context, string name)
    {
        ILogger logger = context.CreateReplaySafeLogger(nameof(FireAndForgetFunctions));
        _ = context.CreateTimer(TimeSpan.FromSeconds(10), CancellationToken.None);
        logger.LogInformation("NonAwaiter is done.");
    }



    [Function(nameof(FireAndForgetFunctionsSayHello))]
    public static async Task FireAndForgetFunctionsSayHello([OrchestrationTrigger] TaskOrchestrationContext taskOrchestrationContext, string name)
    {
        ILogger logger = taskOrchestrationContext.CreateReplaySafeLogger(nameof(FireAndForgetFunctionsSayHello));
        await taskOrchestrationContext.CreateTimer(TimeSpan.FromSeconds(10), CancellationToken.None);
        logger.LogInformation("Saying hello to {name}.", name);
    }

    [Function("FireAndForgetFunctions_HttpStart")]
    public static async Task<HttpResponseData> HttpStart(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
        [DurableClient] DurableTaskClient client,
        FunctionContext executionContext)
    {
        ILogger logger = executionContext.GetLogger("FireAndForgetFunctions_HttpStart");

        // Function input comes from the request content.
        string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
            nameof(FireAndForgetFunctions));

        logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

        // Returns an HTTP 202 response with an instance management payload.
        // See https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-http-api#start-orchestration
        return await client.CreateCheckStatusResponseAsync(req, instanceId);
    }
}

Known workarounds

A workaround is to move scheduling of fire-and-forget task into an activity to be scheduled as independent orchestration.

App Details

• Durable Functions extension version (e.g. v1.8.3): 2
• Azure Functions runtime version (1.0 or 2.0): local
• Programming language used: C#

Screenshots

Note the time of when "FireAndForgetFunctionsNonAwaiter fully completed" is printed.

For detailed output, run func with --verbose flag.
[2024-11-22T01:03:16.297Z] Host lock lease acquired by instance ID '0000000000000000000000000C75A72E'.
[2024-11-22T01:03:17.097Z] Executing 'Functions.FireAndForgetFunctions_HttpStart' (Reason='This function was programmatically called via the host APIs.', Id=a9c0a9a6-ecf1-498d-b907-6e32896208d3)
[2024-11-22T01:03:17.533Z] Scheduling new FireAndForgetFunctions orchestration with instance ID '600944a2b1d64ac2b2efabf3966d5b6b' and 0 bytes of input data.
[2024-11-22T01:03:17.720Z] Started orchestration with ID = '600944a2b1d64ac2b2efabf3966d5b6b'.
[2024-11-22T01:03:17.807Z] Executed 'Functions.FireAndForgetFunctions_HttpStart' (Succeeded, Id=a9c0a9a6-ecf1-498d-b907-6e32896208d3, Duration=734ms)
[2024-11-22T01:03:17.819Z] Executing 'Functions.FireAndForgetFunctions' (Reason='(null)', Id=629d082d-6dc5-49f1-a999-a5448291acec)
[2024-11-22T01:03:17.941Z] Saying hello.
[2024-11-22T01:03:17.981Z] Executed 'Functions.FireAndForgetFunctions' (Succeeded, Id=629d082d-6dc5-49f1-a999-a5448291acec, Duration=178ms)
[2024-11-22T01:03:18.043Z] Executing 'Functions.FireAndForgetFunctionsNonAwaiter' (Reason='(null)', Id=b63e060c-1518-4b1f-86b9-c1fbd47b1814)
[2024-11-22T01:03:18.067Z] NonAwaiter is done.
[2024-11-22T01:03:18.074Z] Executed 'Functions.FireAndForgetFunctionsNonAwaiter' (Succeeded, Id=b63e060c-1518-4b1f-86b9-c1fbd47b1814, Duration=33ms)
[2024-11-22T01:03:32.391Z] Executing 'Functions.FireAndForgetFunctionsNonAwaiter' (Reason='(null)', Id=657d5cca-e8de-48f9-a51b-7f56021ab6de)
[2024-11-22T01:03:32.413Z] Executed 'Functions.FireAndForgetFunctionsNonAwaiter' (Succeeded, Id=657d5cca-e8de-48f9-a51b-7f56021ab6de, Duration=22ms)
[2024-11-22T01:03:32.479Z] Executing 'Functions.FireAndForgetFunctions' (Reason='(null)', Id=9fb57156-46c2-45d9-a9b3-1074a7b0d5b2)
[2024-11-22T01:03:32.495Z] FireAndForgetFunctionsNonAwaiter fully completed.
[2024-11-22T01:03:32.503Z] Executed 'Functions.FireAndForgetFunctions' (Succeeded, Id=9fb57156-46c2-45d9-a9b3-1074a7b0d5b2, Duration=26ms)

If deployed to Azure

We have access to a lot of telemetry that can help with investigations. Please provide as much of the following information as you can to help us investigate!

• Timeframe issue observed:
• Function App name:
• Function name(s):
• Azure region:
• Orchestration instance ID(s):
• Azure storage account name:

If you don't want to share your Function App or storage account name GitHub, please at least share the orchestration instance ID. Otherwise it's extremely difficult to look up information.

[cgillum]

This is expected behavior. An orchestration never transitions into a completed state until all outstanding tasks are either completed or cancelled (cancellation currently only applies to timers). If you want true fire-and-forget, you'll need to start the background task from an activity function.

In your case, change FireAndForgetFunctions to call an activity, have that activity bind to [DurableClient] DurableTaskClient client, and then use client.ScheduleNewOrchestrationInstanceAsync(nameof(FireAndForgetFunctionsNonAwaiter)) from the activity.

[vgpro54321]

I suspect there is intention and effort to do that. It should be quite simple for the DF engine to see that function terminated without ContinueAsNew and move on. Instead engine refrains from signaling orchestration completion and starts waiting for sub orchestrations. This behavior is not consistent with async paradigm mimicked otherwise in the framework. I am wondering, what is cause and purpose of this?

[cgillum]

This behavior has been in place for the Durable Task Framework since before I became a maintainer, and I've been reluctant to change it since I'm not positive that I know what the implications would be. That said, we're adjusting this behavior in some of the newer SDKs, like Java, and we may end up doing the same with the .NET Isolated SDK after we remove its internal dependency on the Durable Task Framework (something we are planning to do as it's creating a lot of unnecessary complexity for us).

[vgpro54321]

This behavior is different from async/await pattern tradition, it is artificial and limiting. I would say even the first argument by itself should be enough to change the behavior.

As an half-way option, a TaskOptions parameter for sub orchestrations could be a compromise, similar to WithInstanceId.