Sea ice model not optimised for distributed GPU runs

[taimoorsohail]

Hello,

I am trying to run a 1/6th-degree global ocean-sea-ice model with ClimaOcean+ClimaSeaIce here.

When I run the coupled simulation with distributed GPUs and a sea ice model, I get a 7x slowdown compared to no sea ice.

@simone-silvestri mentioned the sea ice model is not optimised for distributed GPU runs, but I thought I would flag here this outstanding issue which currently makes super high resolution global simulations with ClimaSeaIce difficult.

FYI - I am setting up some distributed ClimaOcean+ClimaSeaIce tests here which could be used for benchmarking down the track.

cc @navidcy @xkykai

[simone-silvestri]

I suspect the performance bottleneck is the fill_halo_regions! happening at every substep of the dynamics algorithm.
One way to solve the issue is to implement the same optimization we developed for the split explicit free surface algorithm described here (section 3.3) that allows us to avoid the fill halo regions. Something like that was attempted in PR #88.

However, before optimizing, it is best not to assume we know the performance bottleneck but profile the execution. So, the first thing to do is to provide a detailed profile of the code to pinpoint the bottleneck, to know where we can go and operate. You can do it using the nsys profiler, which should already be installed on a server that has CUDA support.
The way to profile is by wrapping the execution under the nsys profiler as so:

nsys profile --tracer=cuda,mpi --ouptut=my_profile julia --project --checkbounds=no minimum_working_example.jl

If you are running under MPI, you can profile each core independently with

mpiexec -np 4 nsys profile --tracer=cuda,mpi --ouptut=my_profile${SLURM_PROCID} julia --project --checkbounds=no minimum_working_example.jl

To generate a statistically relevant profile, you would need to run at least 100 - 200 iterations.
If you can post the output file here (it should be a file with the .nsys-rep extension), we can analyze it and optimize the performance