Skip to content

Commit af96b33

Browse files
committed
Fix docs
1 parent 8fa25e5 commit af96b33

File tree

1 file changed

+16
-20
lines changed

1 file changed

+16
-20
lines changed

website/software_install.md

+16-20
Original file line numberDiff line numberDiff line change
@@ -231,8 +231,7 @@ $ mpiexecjl -n 4 -host localhost julia --project ./hello_mpi.jl
231231
```
232232
}
233233

234-
235-
<!-- For running Julia at scale on Piz Daint, refer to the [Julia MPI GPU on Piz Daint](#julia_mpi_gpu_on_piz_daint) section. -->
234+
For running Julia at scale on Piz Daint, refer to the [Julia MPI GPU on Piz Daint](#julia_mpi_gpu_on_piz_daint) section.
236235

237236
## GPU computing on Piz Daint
238237

@@ -447,12 +446,14 @@ If you do not want to use an interactive session you can use the `sbatch` comman
447446
#SBATCH --constraint=gpu
448447
#SBATCH --account class04
449448

450-
# activate julia env
451-
. $SCRATCH/../julia/daint-gpu-nocudaaware/activate
449+
# make sure to activate julia env before executing this script.
450+
# DO NOT add this `. $SCRATCH/../julia/daint-gpu-nocudaaware/activate` in here
452451

453-
srun julia <my_julia_gpu_script.jl>
452+
srun julia --project <my_julia_gpu_script.jl>
454453
```
455454

455+
\warn{Make sure to have activated Julia env (`. $SCRATCH/../julia/daint-gpu-nocudaaware/activate`) **before** executing the `sbatch` command.}
456+
456457
### JupyterLab access on Piz Daint
457458
Some tasks and homework, are prepared as Jupyter notebook and can easily be executed within a JupyterLab environment. CSCS offers a convenient [JupyterLab access](https://user.cscs.ch/tools/interactive/jupyterlab/#access-and-setup).
458459

@@ -486,43 +487,41 @@ fusermount -u -z /home/$USER/mnt_daint
486487
```
487488
For convenience it is suggested to also symlink to the home-directory `ln -s ~/mnt/daint/users/<your username on daint> ~/mnt/daint_home`. (Note that we mount the root directory `/` with `sshfs` such that access to `/scratch` is possible.)
488489

489-
<!--
490490
### Julia MPI GPU on Piz Daint
491491
The following step should allow you to run distributed memory parallelisation application on multiple GPU nodes on Piz Daint.
492492
1. Make sure to have the Julia GPU environment loaded
493493
```sh
494-
module load daint-gpu
495-
module load Julia/1.9.3-CrayGNU-21.09-cuda
494+
. $SCRATCH/../julia/daint-gpu-nocudaaware/activate
496495
```
496+
497497
2. Then, you would need to allocate more than one node, let's say 4 nodes for 2 hours, using `salloc`
498498
```
499499
salloc -C'gpu' -Aclass04 -N4 -n4 --time=02:00:00
500500
```
501+
501502
3. To launch a Julia (GPU) MPI script on 4 nodes (GPUs) using MPI, you can simply use `srun`
502503
```sh
503-
srun -n4 julia -O3 <my_mpi_script.jl>
504+
srun -n4 julia --project <my_mpi_script.jl>
504505
```
505506

506507
#### CUDA-aware MPI on Piz Daint
507508
\warn{There is currently an issue on the Daint software stack with CuDA-aware MPI. For now, make sure **not to run** with CUDA-aware MPI, i.e., having both `MPICH_RDMA_ENABLED_CUDA` and `IGG_CUDAAWARE_MPI` set to 0.}
508509

509-
You may want to leverage CUDA-aware MPI, i.e., passing GPU pointers directly through the MPI-based update halo functions, then make sure to export the appropriate `ENV` variables
510+
<!-- You may want to leverage CUDA-aware MPI, i.e., passing GPU pointers directly through the MPI-based update halo functions, then make sure to export the appropriate `ENV` variables
510511
```sh
511512
export MPICH_RDMA_ENABLED_CUDA=1
512513
export IGG_CUDAAWARE_MPI=1
513514
```
515+
and to activate the Julia env on the login node
514516
515517
In the CUDA-aware MPI case, a more robust launch procedure may be to launch a shell script via `srun`. You can create, e.g., a [`runme_mpi_daint.sh`](https://github.com/eth-vaw-glaciology/course-101-0250-00/blob/main/scripts/l8_scripts/l8_runme_mpi_daint.sh) script containing:
516518
```sh
517519
#!/bin/bash -l
518520
519-
module load daint-gpu
520-
module load Julia/1.9.3-CrayGNU-21.09-cuda
521-
522521
export MPICH_RDMA_ENABLED_CUDA=1
523522
export IGG_CUDAAWARE_MPI=1
524523
525-
julia -O3 <my_script.jl>
524+
julia --project <my_script.jl>
526525
```
527526
528527
Which you then launch using `srun` upon having made it executable (`chmod +x runme_mpi_daint.sh`)
@@ -543,13 +542,10 @@ If you do not want to use an interactive session you can use the `sbatch` comman
543542
#SBATCH --constraint=gpu
544543
#SBATCH --account class04
545544
546-
module load daint-gpu
547-
module load Julia/1.9.3-CrayGNU-21.09-cuda
548-
549545
export MPICH_RDMA_ENABLED_CUDA=1
550546
export IGG_CUDAAWARE_MPI=1
551547
552-
srun -n4 bash -c 'julia -O3 <my_julia_mpi_gpu_script.jl>'
553-
```
548+
srun -n4 bash -c 'julia --project <my_julia_mpi_gpu_script.jl>'
549+
``` -->
554550

555-
\note{The 2 scripts above can be found in the [scripts](https://github.com/eth-vaw-glaciology/course-101-0250-00/blob/main/scripts/l8_scripts/) folder.} -->
551+
\note{The scripts above can be found in the [scripts](https://github.com/eth-vaw-glaciology/course-101-0250-00/blob/main/scripts/l8_scripts/) folder.}

0 commit comments

Comments
 (0)