[mini] document GPU-aware MPI for various clusters (#992)

* document GPU aware MPI for JUWELS and Maxwell

* Apply suggestions from code review

* add lumi and perlmutter, fix maxwell

* fix perlmutter doc

* Update docs/source/building/platforms/perlmutter_nersc.rst

* Update docs/source/building/platforms/perlmutter_nersc.rst

more unique naming

docs/source/building/platforms/booster_jsc.rst

@@ -63,6 +63,12 @@ You can then create your directory in your ``$SCRATCH_<project id>``, where you
 
 and use it to submit a simulation.
 
+.. tip::
+   Parallel simulations can be largely accelerated by using GPU-aware MPI.
+   To utilize GPU-aware MPI, the input parameter ``hipace.comms_buffer_on_gpu = 1`` must be set.
+
+   Note that using GPU-aware MPI may require more GPU memory.
+
 Running on CPU
 --------------
 

docs/source/building/platforms/lumi_csc.rst

@@ -93,3 +93,13 @@ You can then create your directory in ``/project/project_<project id>``, where y
 
 
 and use it to submit a simulation.
+
+.. tip::
+   Parallel simulations can be largely accelerated by using GPU-aware MPI.
+   To utilize GPU-aware MPI, the input parameter ``hipace.comms_buffer_on_gpu = 1`` must be set and the following flag must be passed in the job script:
+
+   .. code-block:: bash
+
+      export FI_MR_CACHE_MAX_COUNT=0
+
+   Note that using GPU-aware MPI may require more GPU memory.

docs/source/building/platforms/maxwell_desy.rst

@@ -56,3 +56,13 @@ submission script:
 The ``-npernode`` must be set to ``GPUS_PER_NODE``, otherwise not all GPUs are used correctly.
 There are nodes with 4 GPUs and 1 GPU (see the `Maxwell documentation on compute infrastructure <https://confluence.desy.de/display/MXW/Compute+Infrastructure>`__.
 for more details and the required constraints). Please set the value accordingly.
+
+.. tip::
+   Parallel simulations can be largely accelerated by using GPU-aware MPI.
+   To utilize GPU-aware MPI, the input parameter ``hipace.comms_buffer_on_gpu = 1`` must be set and the following flag must be passed in the job script:
+
+   .. code-block:: bash
+
+      export UCX_MEMTYPE_CACHE=n
+
+   Note that using GPU-aware MPI may require more GPU memory.

docs/source/building/platforms/perlmutter_nersc.rst

@@ -65,22 +65,35 @@ You can then create your directory in your ``$PSCRATCH``, where you can put your
     #SBATCH -q regular
     #SBATCH -C gpu
     #SBATCH -c 32
-    #SBATCH --ntasks-per-gpu=1
+    #SBATCH --exclusive
+    #SBATCH --gpu-bind=none
     #SBATCH --gpus-per-node=4
     #SBATCH -o hipace.o%j
     #SBATCH -e hipace.e%j
 
-    # GPU-aware MPI
-    export MPICH_GPU_SUPPORT_ENABLED=1
-
-    # expose one GPU per MPI rank
-    export CUDA_VISIBLE_DEVICES=$SLURM_LOCALID
-
     # path to executable and input script
     EXE=$HOME/src/hipace/build/bin/hipace
     INPUTS=inputs
 
-    srun ${EXE} ${INPUTS}
+    # pin to closest NIC to GPU
+    export MPICH_OFI_NIC_POLICY=GPU
+
+    # for GPU-aware MPI use the first line
+    #HIPACE_GPU_AWARE_MPI="hipace.comms_buffer_on_gpu=1"
+    HIPACE_GPU_AWARE_MPI=""
+
+    # CUDA visible devices are ordered inverse to local task IDs
+    #   Reference: nvidia-smi topo -m
+    srun --cpu-bind=cores bash -c "
+        export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
+        ${EXE} ${INPUTS} ${HIPACE_GPU_AWARE_MPI}" \
+      > output.txt
 
 
 and use it to submit a simulation. Note, that this example simulation runs on 8 GPUs, since `-N = 2` yields 2 nodes with 4 GPUs each.
+
+.. tip::
+   Parallel simulations can be largely accelerated by using GPU-aware MPI.
+   To utilize GPU-aware MPI, the input parameter ``hipace.comms_buffer_on_gpu = 1`` must be set (see the job script above).
+
+   Note that using GPU-aware MPI may require more GPU memory.

docs/source/run/parameters.rst

@@ -795,7 +795,7 @@ Use ``hipace/tools/read_insitu_diagnostics.py`` to read the files using this for
 
 
 Additional physics
-----------------
+------------------
 
 Additional physics describe the physics modules implemented in HiPACE++ that go beyond the standard electromagnetic equations.
 This includes ionization (see plasma parameters), binary collisions, and radiation reactions. Since all of these require the actual plasma density,