diff --git a/docs/tutorials/Implementing_Azure_HPC_OnDemand_Platform_Lab_2.md b/docs/tutorials/Implementing_Azure_HPC_OnDemand_Platform_Lab_2.md index 527cf9d4f..17cc9dcad 100644 --- a/docs/tutorials/Implementing_Azure_HPC_OnDemand_Platform_Lab_2.md +++ b/docs/tutorials/Implementing_Azure_HPC_OnDemand_Platform_Lab_2.md @@ -20,7 +20,12 @@ - [Task 1: Build OPM](#task-1-build-opm) - [Task 2: Retrieve test data and run a flow job](#task-2-retrieve-test-data-and-run-a-flow-job) - [Task 3: View the results of the OPM job by using ResInsight](#task-3-review-the-results-of-the-opm-job-by-using-resinsight) - - [Exercise 6: Deprovision Azure HPC OnDemand Platform environment](#exercise-6-deprovision-azure-hpc-ondemand-platform-environment) + - [Exercise 6: Build, run, and analyze OpenFOAM](#exercise-6-build-run-and-analyze-openfoam) + - [Task 1: Build OpenFOAM](#task-1-build-openfoam) + - [Task 2: Running the motorbike tutorial on a single node](#task-2-running-the-motorbike-tutorial-on-a-single-node) + - [Task 3: Running the motorbike tutorial on multiple nodes](#task-3-running-the-motorbike-tutorial-on-multiple-nodes) + - [Task 4: Visualize the motorbike tutorial result](#task-4-visualize-the-motorbike-tutorial-result) + - [Exercise 7: Deprovision Azure HPC OnDemand Platform environment](#exercise-7-deprovision-azure-hpc-ondemand-platform-environment) - [Task 1: Deprovision the Azure resources](#task-1-deprovision-the-azure-resources) @@ -102,7 +107,7 @@ In this exercise, you will review the main features of the Azure HPC OnDemand Pl > Note: Examine the output of the command and verify that the submitted job is in the queue. -1. Switch to the browser tab with the **Azure CycleCloud for Azure HPC On-Demand Platform** page. +1. Switch to the browser tab with the **Azure CycleCloud for Azure HPC On-Demand Platform** page. After some time (less than a minute), a new **execute** instance is created. 1. Review the newly created job's progress, including the new VM creation. ### Task 3: Running interactive apps using Code Server and Linux Desktop @@ -154,7 +159,7 @@ In this exercise, you will review the main features of the Azure HPC OnDemand Pl > Note: Wait until the status of the node changes to **Ready**. This should take about 10 minutes. - > Note: The **viz3d** node provisioning will fail if your subscription doesn't offer **Standard_NV12s_v3** SKU in the Azure region hosting your az-hop deployment. + > Note: The **viz3d** node provisioning will fail if your subscription doesn't offer **Standard_NV6** SKU in the Azure region hosting your az-hop deployment. 1. Switch back to the **My Interactive Sessions** page, and then verify that the corresponding job's status has changed to **Running**. 1. Use the **Delete** button to delete the **Code Server** job by selecting **Confirm** when prompted. @@ -320,6 +325,7 @@ In this exercise, you will install and configure Spack from Code Server, as docu 1. Run the following commands to confirm the list of defined compilers: ```bash + . ~/spack/share/spack/setup-env.sh spack compilers ``` @@ -353,7 +359,7 @@ Duration: 30 minutes ### Task 2: Create the run script -1. At the root of the home directoryn, create a file named **osu_benchmarks.sh** with this content +1. At the root of the home directory, create a file named **osu_benchmarks.sh** with this content ```bash #!/bin/bash BENCH=${1:-osu_latency} @@ -399,6 +405,7 @@ Duration: 60 minutes 1. Review the following script and run it to create the az-hop Spack repo: ```bash + . ~/spack/share/spack/setup-env.sh ~/azurehpc/experimental/azhop/azhop-spack/install.sh ``` @@ -729,7 +736,7 @@ Duration: 60 minutes ### Task 3: Review the results of the OPM job by using ResInsight 1. On the lab computer, in the browser window, switch back to the **Azure HPC On-Demand Platform** portal, and then in the **Interactive Apps** section, select **Linux Desktop**. -1. On the **Linux Desktop** launching page, from the **Session target** drop-down list, ensure that **With GPU** entry is selected. In the **Maximum duration of your remote session** field, enter **1**, and then select **Launch**. +1. On the **Linux Desktop** launching page, from the **Session target** drop-down list, ensure that **With GPU** entry is selected. In the **Maximum duration of your remote session** field, enter **2**, and then select **Launch**. > Note: In case your subscription doesn't have a sufficient number of quotas for the **Standard_NV6** SKU, choose the **Without GPU** entry instead. @@ -745,7 +752,6 @@ Duration: 60 minutes 1. In the **Terminal Emulator** window, at the **[clusteradmin@vis-1 ~]$** prompt, run the following command to launch ResInsight: ```bash - exit vglrun ResInsight ``` @@ -761,7 +767,132 @@ Duration: 60 minutes - Rotation can be maintaining with the right mouse button 1. In the button ribbon, play the simulation with the **>** button. -## Exercise 6: Deprovision Azure HPC OnDemand Platform environment +## Exercise 6: Build, run, and analyze OpenFOAM + +In this exercise, you will build, run, and analyze CFD simulation using OpenFOAM. + +Duration: 60 minutes + +### Task 1: Build OpenFOAM + +1. On the lab computer, in the browser window displaying the Code Server, in the **Terminal** pane, at the **[clusteradmin@hb120v2-1 ~]$** prompt, run the following command to build OpenFOAM 8: + + ```bash + . ~/spack/share/spack/setup-env.sh + module use /usr/share/Modules/modulefiles + spack install openfoam-org@8 + ``` + > Note: Wait for the build to complete. This might take about 30 minutes. +### Task 2: Running the motorbike tutorial on a single node + +1. On the lab computer, in the browser window displaying the Code Server, in the **Terminal** pane, at the **[clusteradmin@hb120v2-1 ~]$** prompt, run the following command to load OpenFOAM modules: + ```bash + spack load openfoam-org@8 + ``` + +1. Copy the motorbike tutorial into scratch space + ```bash + mkdir -p /lustre/$USER + cd /lustre/$USER + cp -r $WM_PROJECT_DIR/tutorials/incompressible/simpleFoam/motorBike . + ``` + +1. Run the case + ```bash + cd motorBike + ./Allrun + ``` + +1. Review all log files `log.*` for any errors + +### Task 3: Running the motorbike tutorial on multiple nodes + +The following updates are needed: + +- Add FOAM_MPIRUN_FLAGS to the mpirun command when using runParallel (needed for all version of OpenFOAM) +- Reconstruct the single partition after the solve + +1. On the lab computer, in the browser window displaying the Code Server, in the **Terminal** pane, at the **[clusteradmin@hb120v2-1 ~]$** prompt, run the following commands: + + ```bash + cd /lustre/$USER/motorBike + sed -i '/RunFunctions/a source <(declare -f runParallel | sed "s/mpirun/mpirun \\\$FOAM_MPIRUN_FLAGS/g")' Allrun + sed -i 's#/bin/sh#/bin/bash#g' Allrun + sed -i 's/# runApplication reconstructPar/runApplication reconstructPar/g' Allrun + ``` + +1. Now, create a PBS submit script, **submit.sh** withe the following content. + +When running on multiple nodes it is necessary to export all the OpenFOAM environment variables (unless you add loading the modules in `.bashrc`). This is done with the `FOAM_MPIRUN_FLAGS` that are added to the `runParallel` in the last step. The script will run for the number of cores specified to PBS (`select` x `mpiprocs`) + +```bash +#!/bin/bash +. ~/spack/share/spack/setup-env.sh +module use /usr/share/Modules/modulefiles +spack load openfoam-org@8 +ranks_per_numa=4 +export FOAM_MPIRUN_FLAGS="-hostfile $PBS_NODEFILE $(env |grep 'WM_\|FOAM' | cut -d'=' -f1 | sed 's/^/-x /g' | tr '\n' ' ') -x MPI_BUFFER_SIZE --report-bindings --map-by ppr:${ranks_per_numa}:numa" +$PBS_O_WORKDIR/Allrun -cores $(wc -l <$PBS_NODEFILE) +``` + +1. Save that file under the `/lustre/$USER/motorBike` directory + +1. Run the OpenFOAM job + +```bash +rm log.* +qsub -l select=2:slot_type=hb120v2:ncpus=120:mpiprocs=120,place=scatter:excl submit.sh +``` + +1. Monitor the job and wait for the job to be finished + +### Task 4: Visualize the motorbike tutorial result + +1. On the lab computer, in the browser window, switch back to the **Azure HPC On-Demand Platform** portal, and then in the **Interactive Apps** section, select **Linux Desktop**. +1. On the **Linux Desktop** launching page, from the **Session target** drop-down list, ensure that **With GPU** entry is selected. In the **Maximum duration of your remote session** field, enter **2**, and then select **Launch**. + + > Note: In case your subscription doesn't have a sufficient number of quotas for the **Standard_NV6** SKU, choose the **Without GPU** entry instead. + + > Note: This will begin compute node provisioning of the type you specified. This also creates a new job with its **Queued** status displaying on the same page. + +1. Switch back to the **Linux Desktop** launching page, and then verify that the corresponding job's status has changed to **Running**. +1. Adjust **Compression** and **Image quality** according to your preferences, and then select **Launch Linux Desktop**. + + > Note: This will open another browser tab displaying the Linux Desktop session. + +1. Within the Linux Desktop session, start **Terminal Emulator**. + +1. Install the **Paraview** viewer + +```bash +wget "https://www.paraview.org/paraview-downloads/download.php?submit=Download&version=v5.10&type=binary&os=Linux&downloadFile=ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz" -O ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz + +tar xvf ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz + +``` + +1. Create a case file and launch Paraview + +```bash +touch /lustre/$USER/motorBike/case.foam +vglrun ./ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/paraview +``` + +1. Open the model + +Within **Paraview** open the case `/lustre/$USER/motorBike/case.foam` + +When the model is loaded, you can view the geometry like this: +- In the bottom left pane, in the "Mesh Regions" list, unselect "internalMesh" and select "group/motorBikeGroup". +- Click "Apply" above the list. +- You should now see the model geometry, and you can move/rotate/zoom using the mouse. + +Next, you can visualize the simulation results. +- Click the "Play" button on the toolbar at the top of the window to advance to the end of the simulation. +- On the Active Variables Coontrol toolbar you will find a drop down box where you can select variables. For example, select "p" for pressure. + + +## Exercise 7: Deprovision Azure HPC OnDemand Platform environment Duration: 5 minutes