[12.x] Add Http::batch

[WendellAdriel]

Overview

Inspired by the hooks from the Bus::batch, this PR adds a new Http::batch with the same hooks that are available in the Bus::batch.

Use Case

Imagine that your application needs to connect to 3 different 3rd-party services to setup configurations for your customers, and you want to track the progress of the overall and also take actions after each one of these calls finish (handling success and error cases) and after everything finished you want to send a notification to the client.

With the current Http::pool, this is not possible. But with the new Http::batch, you can do that easily by setting up hooks that can run to cover all of these needs.

Example

$responses = Http::batch(fn (Batch $batch) => [
    $batch->get('http://localhost/first'),
    $batch->get('http://localhost/second'),
    $batch->get('http://localhost/third'),
])->before(function (Batch $batch) {
    // This runs before the first HTTP request is executed.
})->progress(function (Batch $batch, int|string $key, Response $response) {
    // This runs after each successful HTTP request from the Batch.
})->catch(function (Batch $batch, int|string $key, Response|RequestException $response) {
    // This runs after each failed HTTP request from the Batch.
})->then(function (Batch $batch, array $results) {
    // This runs ONLY IF all the HTTP requests from the Batch are successful and the batch is not cancelled.
})->finally(function (Batch $batch, array $results) {
    // This runs after all the HTTP requests from the Batch finish and the batch is not cancelled.
})->send();

Helper Methods/Properties

Besides the Hooks, the \Illuminate\Http\Client\Batch class also provides some helper methods and properties to check details from the batch of requests:

$batch->totalRequests gives you the count of requests in the batch.

$batch->pendingRequests gives you the count of pending requests in the batch.

$batch->failedRequests gives you the count of failed requests in the batch.

$batch->createdAt gives you the datetime that the batch was created.

$batch->cancelledAt gives you the datetime that the batch was cancelled.

$batch->finishedAt gives you the datetime that the batch finished to execute all the HTTP requests.

$batch->processedRequests() gives you the total number of requests that have been processed by the batch.

$batch->completion() gives you the percentage of requests that have been processed (between 0-100).

$batch->hasFailures() returns true if the batch has at least one failed request.

$batch->finished() returns true if the batch has finished executing.

$batch->cancel() to cancel the batch (this will stop the batch to execute the pending requests).

$batch->cancelled() returns true if the batch was cancelled.

[hafezdivandari]

I wonder if it would be possible/better to have a syntax similar to Bus::batch, such as:

Http::pool(fn (Pool $pool) => [
    $pool->get('http://localhost/first'),
    $pool->get('http://localhost/second'),
    $pool->get('http://localhost/third'),
])->before(function (Pool $pool) {
    // Logic here...
})->progress(function (Pool $pool, int|string $key, Response $response) {
    // Logic here...
})->then(function (Pool $pool, array $results) {
    // Logic here...
})->catch(function (Pool $pool, int|string $key, Response|RequestException $response) {
    // Logic here...
})->finally(function (Pool $pool, array $results) {
    // Logic here...
});

[WendellAdriel]

@hafezdivandari I thought about that.
I tried some approaches, but it was going to add a breaking change, and I didn't want to.
Let's see what Taylor says. I could either investigate more on it or add it as a breaking change and point to master instead.

[WendellAdriel]

@taylorotwell an alternative approach to this implementation, that wouldn't create a breaking change can be creating a different method like Http::lazyPool where I can add the hooks plus the dispatch method to init the requests:

Http::lazyPool(fn (LazyPool $pool) => [
    $pool->get('http://localhost/first'),
    $pool->get('http://localhost/second'),
    $pool->get('http://localhost/third'),
])->before(function (LazyPool $pool) {
    // Logic here...
})->progress(function (LazyPool $pool, int|string $key, Response $response) {
    // Logic here...
})->then(function (LazyPool $pool, array $results) {
    // Logic here...
})->catch(function (LazyPool $pool, int|string $key, Response|RequestException $response) {
    // Logic here...
})->finally(function (LazyPool $pool, array $results) {
    // Logic here...
})->dispatch();

If you prefer this approach, I can update this PR towards this approach.
I like both approaches TBH, so it's more what you think it's best for the Framework.

[hafezdivandari]

Maybe Http::pendingPool()->send()?
We also have Process::pool()->start() with a similar approach for inspiration.

[WendellAdriel]

@hafezdivandari I like the names.
I'll just wait for feedback (to see if this idea is something they would accept for the framework) before I change the implementation.

[taylorotwell]

You could probably just call it Http::batch? And I think I would use send instead of dispatch maybe. Can you also explain your real-world use case a bit that made you want to add this and how you plan to use the callbacks?

Please mark as ready for review when the requested changes have been made. 🫡

[WendellAdriel]

Thanks @taylorotwell
I'm not at home today, I'll work on this during the weekend and add both the changes and explain the use cases!

[WendellAdriel]

Hey there @taylorotwell
I marked this PR to be reviewed again.

I updated the implementation and also updated the PR description to reflect the changes in the implementation and the use case that was faced, which I think this implementation allows an elegant and simple way to solve it.

[negoziator]

Pull requests like this 🫶

Great work @WendellAdriel 👌

[WendellAdriel]

Pull requests like this 🫶

Great work @WendellAdriel 👌

Thank you so much @negoziator 🫶🏼

[nexxai]

Can the batch be added to dynamically later? Or is it that once it's created, the batch size must stay the same and a new batch may be created?

[WendellAdriel]

Can the batch be added to dynamically later? Or is it that once it's created, the batch size must stay the same and a new batch may be created?

Since you have access to the Batch in each of the hooks, you can add more calls by calling like:

$batch->get(...);

[MeiKatz]

Not sure about this one but since Laravel 12 requires PHP 8.2 wouldn't it be good if the public properties would be read-only? And have type declarations as close as possible?

[WendellAdriel]

Not sure about this one but since Laravel 12 requires PHP 8.2 wouldn't it be good if the public properties would be read-only? And have type declarations as close as possible?

That's a good question I had when creating the code.
I saw most places were not using property types and I didn't add them, however I do agree that this could be used.
I'd be happy to add typings if it's ok.

[taylorotwell]

@WendellAdriel In job batches, progress returns the completion percentage when have a batch instance, whereas here it is used only to set the callback, and getting the completion percentage is done via the completion method. This is because jobs have the concept of a PendingBatch and a Batch whereas you are having Batch in HTTP world serve both purposes.

[WendellAdriel]

@taylorotwell yeah, I saw that.
However, I didn't come up with a better naming, that's why I used completion for the status.
What do you suggest? Should I rename the progress callback to something else?
Or maybe update the implementation of the progress when no parameters are passed to do the logic from the completion method instead?

[timacdonald]

Does progress work as expected? I just tried this and found weird results.

Http::batch(fn ($batch) => [
    $batch->get('https://httpbin.org/delay/1'),
    $batch->get('https://httpbin.org/delay/10'),
])->before(function () {
    echo 'Before: '.date('Y-m-d H:i:s').PHP_EOL;
})->progress(function () {
    echo 'Progress: '.date('Y-m-d H:i:s').PHP_EOL;
})->send();

I'd expect the output the be something like:

Before: 2000-01-01 00:00:00
Progress: 2000-01-01 00:00:01  # 1 second after before
Progress: 2000-01-01 00:00:10  # 10 seconds after before

But instead I consistently see something like:

Before: 2000-01-01 00:00:00
Progress: 2000-01-01 00:00:10  # 10 seconds after before
Progress: 2000-01-01 00:00:10  # 10 seconds after before

I believe there is a problem in the foreach loop where the promises have already been combined into a single promise. Maybe I'm missing something, though.

[timacdonald]

And even if we were to fix that, I am wondering is the progress method is going to do what we really want.

Imagine I have the following, where I run a slow and a fast request.

Http::batch(fn ($batch) => [
    $batch->get('https://httpbin.org/delay/10'),
    $batch->get('https://httpbin.org/delay/1'),
])->before(function () {
    echo 'Before: '.date('Y-m-d H:i:s').PHP_EOL;
})->progress(function () {
    echo 'Progress: '.date('Y-m-d H:i:s').PHP_EOL;
})->send();

Due to PHP, I assume we would be resolving these sync, so the progress closure for the fast request would not trigger until the long request has finished?

I understand that curl multi is likely processing both of the requests under the hood at the same time, but when it comes to iterating over them in PHP, what output would we expect see?

I assume we would see:

Before: 2000-01-01 00:00:00
Progress: 2000-01-01 00:00:10  # 10 seconds after before
Progress: 2000-01-01 00:00:10  # 10 seconds after before

But I wonder if a user of the method would expect:

Before: 2000-01-01 00:00:00
Progress: 2000-01-01 00:00:10  # 1 second after before
Progress: 2000-01-01 00:00:10  # 10 seconds after before

Due to the issue I raised above, I can't test this myself. Once that is resolved I'm happy to play with it and find out.

[WendellAdriel]

Oh, that's a good catch @timacdonald
Thanks for that, I'll take a look into it.
I'm going to move this PR to draft while I take a look and I'll ping you when I have something if that's ok.
cc: @taylorotwell

[WendellAdriel]

@timacdonald @taylorotwell
I think that I have found a way of making it work well with the progress hook.
IDK if I overcomplicated it a little tho 😅

But with this approach, we have two things that I want to check if it's ok and we can move on like this or if we should dig more into.

1 - For making the progress hook work correctly, the way it handles the promises, there's no way to cancel the batch because it will resolve things in a non-standard order + it will dispatch all promises at the same time. Is it ok if I remove the cancel feature?

2 - Also because of this, adding new calls to the batch won't work as expected as far as I could check.

I didn't find a way to make these two things work with this new approach.
Let me know your thoughts on how I should proceed.

[timacdonald]

@WendellAdriel, Taylor might not see this while it is in draft. I would recommend opening it to get his feedback.

[WendellAdriel]

Thanks @timacdonald!
Were you able to play around with the new implementation?

@taylorotwell I opened again if you want to take a look on what I said above.

This seems to work good, but we would need to take out the cancel mechanism and adding new calls to the already initiated batch.

Let me know how you want me to proceed, and I'll be happy to work on any needed changes! 🔥