You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<!-- For running Julia at scale on Piz Daint, refer to the [Julia MPI GPU on Piz Daint](#julia_mpi_gpu_on_piz_daint) section. -->
234
+
For running Julia at scale on Piz Daint, refer to the [Julia MPI GPU on Piz Daint](#julia_mpi_gpu_on_piz_daint) section.
236
235
237
236
## GPU computing on Piz Daint
238
237
@@ -447,12 +446,14 @@ If you do not want to use an interactive session you can use the `sbatch` comman
447
446
#SBATCH --constraint=gpu
448
447
#SBATCH --account class04
449
448
450
-
# activate julia env
451
-
.$SCRATCH/../julia/daint-gpu-nocudaaware/activate
449
+
#make sure to activate julia env before executing this script.
450
+
# DO NOT add this `. $SCRATCH/../julia/daint-gpu-nocudaaware/activate` in here
452
451
453
-
srun julia <my_julia_gpu_script.jl>
452
+
srun julia --project <my_julia_gpu_script.jl>
454
453
```
455
454
455
+
\warn{Make sure to have activated Julia env (`. $SCRATCH/../julia/daint-gpu-nocudaaware/activate`) **before** executing the `sbatch` command.}
456
+
456
457
### JupyterLab access on Piz Daint
457
458
Some tasks and homework, are prepared as Jupyter notebook and can easily be executed within a JupyterLab environment. CSCS offers a convenient [JupyterLab access](https://user.cscs.ch/tools/interactive/jupyterlab/#access-and-setup).
For convenience it is suggested to also symlink to the home-directory `ln -s ~/mnt/daint/users/<your username on daint> ~/mnt/daint_home`. (Note that we mount the root directory `/` with `sshfs` such that access to `/scratch` is possible.)
488
489
489
-
<!--
490
490
### Julia MPI GPU on Piz Daint
491
491
The following step should allow you to run distributed memory parallelisation application on multiple GPU nodes on Piz Daint.
492
492
1. Make sure to have the Julia GPU environment loaded
493
493
```sh
494
-
module load daint-gpu
495
-
module load Julia/1.9.3-CrayGNU-21.09-cuda
494
+
.$SCRATCH/../julia/daint-gpu-nocudaaware/activate
496
495
```
496
+
497
497
2. Then, you would need to allocate more than one node, let's say 4 nodes for 2 hours, using `salloc`
498
498
```
499
499
salloc -C'gpu' -Aclass04 -N4 -n4 --time=02:00:00
500
500
```
501
+
501
502
3. To launch a Julia (GPU) MPI script on 4 nodes (GPUs) using MPI, you can simply use `srun`
502
503
```sh
503
-
srun -n4 julia -O3 <my_mpi_script.jl>
504
+
srun -n4 julia --project<my_mpi_script.jl>
504
505
```
505
506
506
507
#### CUDA-aware MPI on Piz Daint
507
508
\warn{There is currently an issue on the Daint software stack with CuDA-aware MPI. For now, make sure **not to run** with CUDA-aware MPI, i.e., having both `MPICH_RDMA_ENABLED_CUDA` and `IGG_CUDAAWARE_MPI` set to 0.}
508
509
509
-
You may want to leverage CUDA-aware MPI, i.e., passing GPU pointers directly through the MPI-based update halo functions, then make sure to export the appropriate `ENV` variables
510
+
<!--You may want to leverage CUDA-aware MPI, i.e., passing GPU pointers directly through the MPI-based update halo functions, then make sure to export the appropriate `ENV` variables
510
511
```sh
511
512
export MPICH_RDMA_ENABLED_CUDA=1
512
513
export IGG_CUDAAWARE_MPI=1
513
514
```
515
+
and to activate the Julia env on the login node
514
516
515
517
In the CUDA-aware MPI case, a more robust launch procedure may be to launch a shell script via `srun`. You can create, e.g., a [`runme_mpi_daint.sh`](https://github.com/eth-vaw-glaciology/course-101-0250-00/blob/main/scripts/l8_scripts/l8_runme_mpi_daint.sh) script containing:
516
518
```sh
517
519
#!/bin/bash -l
518
520
519
-
module load daint-gpu
520
-
module load Julia/1.9.3-CrayGNU-21.09-cuda
521
-
522
521
export MPICH_RDMA_ENABLED_CUDA=1
523
522
export IGG_CUDAAWARE_MPI=1
524
523
525
-
julia -O3 <my_script.jl>
524
+
julia --project <my_script.jl>
526
525
```
527
526
528
527
Which you then launch using `srun` upon having made it executable (`chmod +x runme_mpi_daint.sh`)
@@ -543,13 +542,10 @@ If you do not want to use an interactive session you can use the `sbatch` comman
\note{The 2 scripts above can be found in the [scripts](https://github.com/eth-vaw-glaciology/course-101-0250-00/blob/main/scripts/l8_scripts/) folder.}-->
551
+
\note{The scripts above can be found in the [scripts](https://github.com/eth-vaw-glaciology/course-101-0250-00/blob/main/scripts/l8_scripts/) folder.}
0 commit comments