RFC: Provide equivalence of MPICH_ASYNC_PROGRESS

[hominhquan]

This PR is follow-up of #13074.

[janjust]

@hominhquan thanks for the PR, do you have any performance data and/or testing data?

[bosilca]

A step in the right direction. I still have an issue with the fact that the thread is not bound and cannot be bound, but we can address that later.

[hominhquan]

@hominhquan thanks for the PR, do you have any performance data and/or testing data?

@janjust Yes, I shared in #10374 that we observed a gain of upto x1.4 in OSU_Ireduce. Note that most OSU benchmarks only run on two MPI processes, so memory contention and CPU occupation is not stressed enough to their limit.

[devreal]

@hominhquan What is the impact on collective operations, both in shared and distributed memory? I imagine there to be more contention...

[hppritcha]

There are many other things to consider in terms of performance but in terms of changes to just support the async thread capability this looks okay. Low risk since its buy in.

[hppritcha]

This is okay for now but does leave a hole to plug for the sessions model, but since this is an buy-in option for the application user it should be okay for how.

[bosilca]

Do you mind elaborate on this @hppritcha, I fail to see the issue with the session model.

[hominhquan]

@hominhquan What is the impact on collective operations, both in shared and distributed memory? I imagine there to be more contention...

I only compared with OSU_Ireduce. I'll run the whole OSU collective suite this night for a larger view, but only on a single-node shared-mem config, since I don't have access to any cluster system.

[hominhquan]

@hominhquan What is the impact on collective operations, both in shared and distributed memory? I imagine there to be more contention...

I only compared with OSU_Ireduce. I'll run the whole OSU collective suite this night for a larger view, but only on a single-node shared-mem config, since I don't have access to any cluster system.

@devreal below is our results on a single-node Grace CPU, in which we see the sweet-spot around 128-256 KB per rank. There was degradation on three operations: osu_iallgather, osu_iallgatherv and osu_ialltoall

[devreal]

Is that a 3x slowdown for small messages? That would be very concerning. Any idea what would cause that?

[hominhquan]

Is that a 3x slowdown for small messages? That would be very concerning. Any idea what would cause that?

It would come from the synchronization overhead between the progress thread (now executing opal_progress()) and the main thread (waiting for the job done at each MPI_Test() or MPI_Wait()). This incompressible cost turns out to be important on small communications (Amdahl's law).

[devreal]

It's not Amdahl's law if the execution gets slower when you add more resources :) In a perfect world, the thread calling MPI_Iallreduce would trigger the progress thread, which then immediately goes ahead and executes the required communications before returning control to the main thread. This looks like many 10s of microseconds synchronization overhead, which is more than I would expect.

[hominhquan]

As said @bosilca in #13074, (back in 2010-2014), it was hard to get an optimal solution for all message sizes and to all use-cases, and I confirm this conclusion. Spawning a thread introduces many side effects that we can hardly measure their impact. The idea behind this patch is to (re)open the door to further improvement and fine-tuning (core-binding ? time-based yield/progression ? work-stealing ?).

[bosilca]

These results looks suspicious. OSU doesn't do overlap, it basically posts the non-blocking and wait for it.

For small messages I could understand 10 to 20 percent performance degradation, but not 3x. And for large messages on an iallgather a 2x increase in performance ? Where is that extra bandwidth coming from ?

What exactly is the speedup you report on this graph ?

[hominhquan]

Recent version of OSU added, for example osu_ireduce which does overlap and measure the overall time of MPI_Ireduce + dummy_compute(latency_seq) + MPI_Wait, in which dummy_compute() simulates a computation period of latency_seq = elapsed(MPI_Ireduce + MPI_Wait). The goal was to see whether the MPI library manages to overlap 100% the non-blocking collective with the dummy_compute(...). They also did the same with other osu_iallreduce.c, osu_ialltoall or osu_ibcast.c etc..

The results I showed above is on the overall_time of those non-blocking operations with OPAL_ASYNC_PROGRESS=[0|1], on the latest OSU benchmark (>= 7.x) https://mvapich.cse.ohio-state.edu/benchmarks/

The speedup is expected from the fact that some tasks (e.g. Ireduce or purely data bcast/scatter/gather) are done really in parallel of dummy_compute() by the new progress thread, and not at the end of communication (MPI_Wait()).

I tested with MPICH's MPICH_ASYNC_PROGRESS=1 and also observed some performance gain "à l'époque" (numbers to be refreshed by new runs).

Again, I know the limitation of OSU of only using two MPI processes, where resource contention is not stressed much far. They mimic anyway a real well-written overlapped non-blocking schema.

[hjelmn]

I have been out of the loop for awhile but I thought the idea was to do more targeted progress rather than the hammer that is just looping on opal_progress (which is predictably bad-- we knew this years ago). The concept I remember was signaled sends that could then wake up the remote process that would then make progress. Did that go nowhere?

The way I remember it, the reason this speeds up large messages is we can progress the RNDV without entering MPI. This is what signaled sends in the BTL were supposed to address. It would send the RTS which would trigger an RDMA get, RTR, whatever then go back to sleep.

[hominhquan]

Ping, please tell me if this PR needs more discussion and has its merit to be merged ?