fan-out 100k activities / batching?

[XBeg9]

Hi Team,

There is a little bit of initial conversation, but I want to post my observations after testing. Looks like durable-functions lack of batching functionality which is presented when you have eventhub trigger with batching enabled. My scenario is to parse a csv file with ~100+k records, process them with azure function (could be in batch/chunks) and be sure that all rows processed via orchestrator state change. With just using eventhub it takes up to minute, with durable functions it can take up to an hour, any suggestions on batching techniques around durable functions?

Thanks

[cgillum]

Event Hubs is hyper-optimized for being able to process large numbers of events with minimal overhead. Durable Functions with the Azure Storage backend, on the other hand, requires each message to be created, dequeued, and deleted individually, requiring substantially higher amounts of I/O. This is why normal event processing scenarios are generally going to favor Event Hubs compared to orchestrations or entities.

That said, the Netherite storage provider for Durable Functions uses Event Hubs in its implementation to get the same benefits of more efficient I/O. It internally uses batching to avoid too many small I/Os, though the actual programming model in functions remains the same. Have you tried testing your ~100K workload with Durable Functions and Netherite?

[XBeg9]

Yes, I did and got an increase, but it was insignificant... I will recheck it; I probably needed something in the EventHub configuration. Can Durable Functions support batching with the Netherite backend?

[cgillum]

Netherite does batching internally on the backend to reduce I/O but will still invoke your function code one-by-one.

/cc @sebastianburckhardt

[sebastianburckhardt]

Based on your description I would have expected Netherite to improve the throughput significantly.

Still, if the amount of processing you do on a single line is extremely lightweight then the overhead of invoking one function per line is going to be a problem no matter which backend you use, and calling 100k activities from a single orchestrator is also a problem (because of the history size).

Have you tried batching the input/output of the activity function? Like, passing in an array of lines as input, as opposed to a single line. For example, if you set the batch size to 1000, your orchestrator would now call 100 activities, and each activity processes 1000 lines. I would expect DF to handle that a lot better.