[HPC Cluster] Unable to utilize multiple GPUs per node with correct domain decomposition in HPC environment

[LightLifel]

Hi everyone,

I successfully installed the GPU version of OpenSn on an HPC cluster. However, when running examples, I am unable to utilize multiple GPUs within a single computing node correctly.

Environment:

• Cluster Node Configuration:
  • CPU: 2 Sockets (64 cores per socket, 128 cores total)
  • GPU: 8 GPUs per node

Issue Description:
I have tried two ways to run the tests, but neither achieved the expected multi-GPU parallelism with domain decomposition.

Attempt 1: Submit job via CLI with Interactive Node

Command:

mpiexec -n 4 python3 transport_3d_1b_ortho_gpu.py

I applied an interactive computing node , and run tests

The program successfully recognizes 4 processes and divides the global cells into 4 local cells. But, 4 processes run on a single GPU (GPU 0). It does not distribute the workload across multiple GPUs, just like this:

Attempt 2: Job Submission with sbatch
Command:

sbatch run.sh

run.sh is srun -n 4 python3 transport_3d_1b_ortho_gpu.py
The job appears to utilize all GPUs. However, it just repeat the one process on each GPU independently rather than running as a coordinated MPI job.Consequently, the global cell count equals the local cell count (no domain decomposition occurs).

Question :
Is there a recommended way to submit tasks in an HPC cluster to utilize multiple GPUs per node while ensuring correct domain decomposition?

What's more, I also used mpiexec -n 4 path/to/opensn/build/python/opensn -i path/to/opensn/test/python/modules/linear_boltzmann_solvers/transport_steady/transport_3d_1b_ortho_gpu.py and it is the same as the last command.

[quocdang1998]

Hi,

Right now, we haven't finished the implementation for 1 rank -> multi-GPUs. A PR in the near future will likely fix this issue.

OpenSn relies on the GPU assignment of SLURM.

For multi-GPU usage, you must run with multi-threading. In your case, I presume that there is 128 CPU cores and 8 GPUs per node, and any CPU in the node can access any GPU in the same node. This averages to 16 (=128/8) CPU cores per GPU. You can try running the code with:

srun -N 1 -c 16 --gpus-per-task=1 -n 8 python3 .... # for 1 node
srun -N 2 -c 16 --gpus-per-task=1 -n 16 python3 .... # for 2 nodes

Or with sbatch:

#SBATCH --nodes=1
#SBATCH --gpus-per-task=1
#SBATCH --exclusive

srun -n 8 -c 16 python ...

Please let us know if there is any issue.