first commit

Author: harry

Dockerfile

@@ -0,0 +1,14 @@
+FROM nvcr.io/nvidia/pytorch:23.11-py3
+
+WORKDIR /
+
+# change the download source of apt, comment it out if you are abroad
+COPY sources.list /etc/apt/sources.list
+RUN apt-get update && \
+    apt-get install -y openssh-server vim curl inetutils-ping net-tools telnet lsof
+
+COPY start.sh /start.sh
+COPY sshd_config /etc/ssh/sshd_config
+COPY nccl-tests /nccl-tests
+
+CMD ["/bin/bash", "start.sh"]

README.md

@@ -1,2 +1,73 @@
-# build-nccl-tests-with-pytorch
-This is a dockerfile to build PyTorch executing NCCL-Tests. 
+# Build-NCCL-Tests-With-PyTorch
+
+![license](https://img.shields.io/hexpm/l/plug.svg)
+[![docker](https://img.shields.io/docker/pulls/mayooot/nccl-tests-with-pytorch.svg)](https://hub.docker.com/r/mayooot/nccl-tests-with-pytorch)
+
+# Overview
+
+Build [NCCL-Tests](https://github.com/NVIDIA/nccl-tests) and configure SSHD in PyTorch container to help you test NCCL
+faster!
+
+PyTorch Version: 23.11
+
+# Quick Start
+
+~~~shell
+docker pull mayooot/nccl-tests-with-pytorch:v0.0.1
+~~~
+
+# Build From Source
+
+~~~shell
+git clone https://github.com/mayooot/build-nccl-tests-with-pytorch
+cd build-nccl-tests-with-pytorch
+
+docker build -t nccl-tests-with-pytorch:latest .
+~~~
+
+# Usage
+
+The default values for `PORT` and `PASS` are 12345, you can replace them with `-e`.
+
+In addition, you need to mount the host's `id_rsa` and `id_rsa.pub` to the container.
+
+~~~shell
+docker run --name foo \
+  -d -it \
+  --network=host \
+  -e PORT=1998 -e PASS=P@88w0rd \
+  -v /tmp/id_rsa:/root/.ssh/id_rsa \
+  -v /tmp/id_rsa.pub:/root/.ssh/id_rsa.pub \
+  --gpus all --shm-size=1g \
+  --cap-add=IPC_LOCK --device=/dev/infiniband \
+  mayooot/nccl-tests-with-pytorch:v0.0.1 
+~~~
+
+The code and executable for NCCL-Tests is located in `/nccl-tests`, so let me show you how to use it,
+using `all_reduce_perf` as an example.
+
+Before using `all_reduce_perf`, you need to configure SSH intercommunication.
+
+~~~shell
+ssh-copy-id -p 1998 root@all_cluster_ip
+~~~
+
+Please replace `--host cluster_ip1,cluster_ip2,...` to the real cluster's IP address.
+
+~~~shell
+docker exec -it foo bash
+
+cd /nccl-tests
+
+mpirun --allow-run-as-root \
+  -mca plm_rsh_args "-p 1998" \
+  -x NCCL_DEBUG=INFO \
+  -x NCCL_IB_HCA=mlx5_10,mlx5_11,mlx5_12,mlx5_13,mlx5_14,mlx5_15,mlx5_16,mlx5_17 \
+  --host cluster_ip1,cluster_ip2,... \
+  ./build/all_reduce_perf \
+  -b 1G -e 4G -f 2 -g 8
+~~~
+
+# Contribute
+
+Feel free to open issues and pull requests. Any feedback is highly appreciated!

nccl-tests/LICENSE.txt

@@ -0,0 +1,27 @@
+
+ Copyright (c) 2016-2017, NVIDIA CORPORATION.  All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+  * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+  * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in the
+    documentation and/or other materials provided with the distribution.
+  * Neither the name of NVIDIA CORPORATION, nor the names of their
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
+ EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

nccl-tests/Makefile

@@ -0,0 +1,23 @@
+#
+# Copyright (c) 2017, NVIDIA CORPORATION. All rights reserved.
+#
+# See LICENCE.txt for license information
+#
+
+BUILDDIR ?= build
+override BUILDDIR := $(abspath $(BUILDDIR))
+
+.PHONY: all clean
+
+default: src.build
+
+TARGETS=src
+
+all:   ${TARGETS:%=%.build}
+clean: ${TARGETS:%=%.clean}
+
+%.build:
+	${MAKE} -C $* build BUILDDIR=${BUILDDIR}
+
+%.clean:
+	${MAKE} -C $* clean BUILDDIR=${BUILDDIR}

nccl-tests/README.md

@@ -0,0 +1,72 @@
+# NCCL Tests
+
+These tests check both the performance and the correctness of [NCCL](http://github.com/nvidia/nccl) operations.
+
+## Build
+
+To build the tests, just type `make`.
+
+If CUDA is not installed in /usr/local/cuda, you may specify CUDA\_HOME. Similarly, if NCCL is not installed in /usr, you may specify NCCL\_HOME.
+
+```shell
+$ make CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl
+```
+
+NCCL tests rely on MPI to work on multiple processes, hence multiple nodes. If you want to compile the tests with MPI support, you need to set MPI=1 and set MPI\_HOME to the path where MPI is installed.
+
+```shell
+$ make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl
+```
+
+## Usage
+
+NCCL tests can run on multiple processes, multiple threads, and multiple CUDA devices per thread. The number of process is managed by MPI and is therefore not passed to the tests as argument. The total number of ranks (=CUDA devices) will be equal to (number of processes)\*(number of threads)\*(number of GPUs per thread).
+
+### Quick examples
+
+Run on 8 GPUs (`-g 8`), scanning from 8 Bytes to 128MBytes :
+```shell
+$ ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8
+```
+
+Run with MPI on 10 processes (potentially on multiple nodes) with 4 GPUs each, for a total of 40 GPUs:
+```shell
+$ mpirun -np 10 ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 4
+```
+
+### Performance
+
+See the [Performance](doc/PERFORMANCE.md) page for explanation about numbers, and in particular the "busbw" column.
+
+### Arguments
+
+All tests support the same set of arguments :
+
+* Number of GPUs
+  * `-t,--nthreads <num threads>` number of threads per process. Default : 1.
+  * `-g,--ngpus <GPUs per thread>` number of gpus per thread. Default : 1.
+* Sizes to scan
+  * `-b,--minbytes <min size in bytes>` minimum size to start with. Default : 32M.
+  * `-e,--maxbytes <max size in bytes>` maximum size to end at. Default : 32M.
+  * Increments can be either fixed or a multiplication factor. Only one of those should be used
+    * `-i,--stepbytes <increment size>` fixed increment between sizes. Default : 1M.
+    * `-f,--stepfactor <increment factor>` multiplication factor between sizes. Default : disabled.
+* NCCL operations arguments
+  * `-o,--op <sum/prod/min/max/avg/all>` Specify which reduction operation to perform. Only relevant for reduction operations like Allreduce, Reduce or ReduceScatter. Default : Sum.
+  * `-d,--datatype <nccltype/all>` Specify which datatype to use. Default : Float.
+  * `-r,--root <root/all>` Specify which root to use. Only for operations with a root like broadcast or reduce. Default : 0.
+* Performance
+  * `-n,--iters <iteration count>` number of iterations. Default : 20.
+  * `-w,--warmup_iters <warmup iteration count>` number of warmup iterations (not timed). Default : 5.
+  * `-m,--agg_iters <aggregation count>` number of operations to aggregate together in each iteration. Default : 1.
+  * `-a,--average <0/1/2/3>` Report performance as an average across all ranks (MPI=1 only). <0=Rank0,1=Avg,2=Min,3=Max>. Default : 1.
+* Test operation
+  * `-p,--parallel_init <0/1>` use threads to initialize NCCL in parallel. Default : 0.
+  * `-c,--check <check iteration count>` perform count iterations, checking correctness of results on each iteration. This can be quite slow on large numbers of GPUs. Default : 1.
+  * `-z,--blocking <0/1>` Make NCCL collective blocking, i.e. have CPUs wait and sync after each collective. Default : 0.
+  * `-G,--cudagraph <num graph launches>` Capture iterations as a CUDA graph and then replay specified number of times. Default : 0.
+
+## Copyright
+
+NCCL tests are provided under the BSD license. All source code and accompanying documentation is copyright (c) 2016-2021, NVIDIA CORPORATION. All rights reserved.
+