This section walks you through a brief introduction to using incflo.
+
+Downloading the code
+incflo is built on top of the AMReX framework. In order to run
+incflo, you must download separate git modules for incflo, AMReX
+and AMReX-Hydro.
+First, make sure that git is installed on your machine.
+
+Download the AMReX repository by typing:
+git clone https://github.com/AMReX-Codes/amrex.git
+
+
+This will create a folder called amrex/
on your machine.
+For additional information on downloading the AMReX source code see AMReX user guide - Downloading the code.
+
+Download the AMReX-Hydro repository by typing:
+git clone https://github.com/AMReX-Fluids/AMReX-Hydro.git
+
+
+This will create a folder called AMReX-Hydro/
on your machine.
+
+Download the incflo repository by typing:
+git clone https://github.com/AMReX-Fluids/incflo.git
+
+
+This will create a folder called incflo/
on your machine.
+
+
+You will want to periodically update each of these repositories
+by typing git pull
within each repository.
+Next, you can choose to either Building with GNU Make or Building with CMake.
+
+
+Building with GNU Make
+Here, we walk you through compiling an incflo executable.
+Decide whether you want to build the executable in 2-D vs 3-D,
+with or without EB (embedded boundaries / cut cells), and choose the corresponding
+directory: test_2d
, test_3d
, test_no_eb_2d
, or test_no_eb
.
+For this example, let’s say you want to build in 2-D with cut cells.
+Then to build the code:
+
+cd
to the desired build directory; e.g.
+
+
+Edit the GNUmakefile
:
+Set AMREX_HOME to be the path to the directory where you have put amrex. NOTE: when setting AMREX_HOME
in the GNUmakefile
, be aware that ~
does not expand, so AMREX_HOME=~/amrex/
will yield an error.
+Alternatively, the path to AMReX can be set up as an environment variable, AMREX_HOME
, on your machine to point to the path name where you have put AMReX. For example, if you are using the bash shell, you can add this to your .bashrc
as:
+export AMREX_HOME=/path/to/amrex
+
+
+alternatively, in tcsh one can set
+setenv AMREX_HOME /path/to/amrex
+
+
+Common options that you can set in the GNUMakefile include
+
+
+Option name |
+Description |
+Possible values |
+Default
+value |
+
+
+
+COMP |
+Compiler (gnu or intel) |
+gnu / intel |
+None |
+
+USE_MPI |
+Whether to enable MPI |
+TRUE / FALSE |
+FALSE |
+
+USE_OMP |
+Whether to enable OpenMP |
+TRUE / FALSE |
+FALSE |
+
+USE_CUDA |
+Whether to enable CUDA |
+TRUE / FALSE |
+FALSE |
+
+DEBUG |
+Whether to use DEBUG mode |
+TRUE / FALSE |
+FALSE |
+
+PROFILE |
+Include profiling info |
+TRUE / FALSE |
+FALSE |
+
+TINY_PROFILE |
+Include tiny profiling info |
+TRUE / FALSE |
+FALSE |
+
+COMM_PROFILE |
+Include comm profiling info |
+TRUE / FALSE |
+FALSE |
+
+TRACE_PROFILE |
+Include trace profiling info |
+TRUE / FALSE |
+FALSE |
+
+
+
+
+
Note
+
Do not set both USE_OMP and USE_CUDA to true.
+
+Additional options and information on using other compilers can be found in the
+AMReX documentation at
+https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html .
+
+Make the executable:
+Now type
+
+The name of the resulting executable (generated by the make system) encodes several of the build characteristics, including dimensionality of the problem, compiler name, and whether MPI and/or OpenMP were linked with the executable.
+Thus, several different build configurations may coexist simultaneously in a problem folder.
+For example, the default build in incflo/test_2d
will look
+like incflo2d.gnu.MPI.EB.ex
, indicating that this is a 2-d version of the code, made with
+COMP=gnu
, USE_MPI=TRUE
and support for embedded boundaries (EB).
+
+
+
+
+Building with CMake
+CMake build is a two-steps process. First cmake
is invoked to create
+configuration files and makefiles in a chosen directory (builddir
).
+Next, the actual build is performed by invoking make
from within builddir
.
+The CMake build process is summarized as follows:
+mkdir /path/to/builddir
+cd /path/to/builddir
+cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/incflo
+make
+
+
+In the above snippet, [options]
indicates one or more options for the
+customization of the build, as described later in this section.
+If the option CMAKE_BUILD_TYPE
is omitted,
+CMAKE_BUILD_TYPE=Release
is assumed.
+There are two modes to build incflo with CMake:
+
+o SUPERBUILD (recommended): AMReX and AMReX-Hydro are built as part
+of the incflo build process. This method is strongly encouraged as it
+ensures that the configuration options are consistent.
+
o STANDALONE: incflo source code is built separately and linked to an existing
+AMReX installation. This is ideal for continuous integration severs (CI)
+and regression testing applications. AMReX library version and configuration options
+must meet incflo requirements.
+
+
+
Note
+
incflo requires CMake 3.14 or higher.
+
+
+SUPERBUILD Instructions (recommended)
+In this mode, incflo CMake inherents AMReX CMake targets and configuration options, as well as AMReX-Hydro CMake targets; that is, incflo, AMReX and AMReX-Hydro are configured and built as a single entity.
+First, make sure you’ve downloaded all the source codes (see Downloading the code).
+Next, build the incflo executable:
+
+> cd incflo
+> mkdir build
+> cd build
+> cmake -DAMREX_HOME=/path/to/amrex/source/directory [amrex options] -DAMREX_HYDRO_HOME=/path/to/AMReX-Hydro/source/directory -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..
+> make -j
+
+
+
+where AMREX_HOME
is a CMake variable pointing to the top-level source directory of the AMReX distribution you downloaded earlier; [amrex options]
is a list of any of the AMReX configuration options listed in the AMReX user guide - Building with CMake; AMREX_HYDRO_HOME
is a CMake variable pointing to the top-level source directory of the AMReX-Hydro distribution you downloaded earlier; and ..
is the relative path pointing to the top-level source directory of incflo or this example.
+
+
+STANDALONE instructions
+For a stand alone build, incflo CMake will look for an existing
+AMReX installation on the system and link the incflo binaries against it.
+
+Building AMReX
+First, make sure you’ve downloaded all the source codes (see Downloading the code).
+Next, configure, build and install AMReX as follows:
+> cd /path/to/amrex
+> mkdir build
+> cd build
+> cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir -DAMREX_HYDRO_HOME=/path/to/AMReX-Hydro/source/directory ..
+> make install
+
+
+[amrex options]
is a list of any of the AMReX configuration options listed in the
+AMReX user guide - Building with CMake.
+We suggest to always use the option -DUSE_XSDK_DEFAULTS=yes
when building AMReX for incflo. AMREX_HYDRO_HOME
is a CMake variable pointing to the top-level source directory of the AMReX-Hydro distribution you downloaded earlier
+
+
+Building incflo
+Build incflo:
+> cd /path/to/incflo
+> mkdir build
+> cd build
+> cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [incflo options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..
+> make -j
+
+
+Passing -DAMReX_ROOT=/absolute/path/to/amrex/installdir
instructs CMake to search
+/absolute/path/to/amrex/installdir
before searching system paths
+for an available AMReX installation.
+AMReX_ROOT
can also be set as an environmental variable instead of passing it as a command line option.
+[incflo options]
indicates any of the configuration option listed in the table below.
+
+
+Option name |
+Description |
+Possible values |
+Default
+value |
+
+
+
+CMAKE_CXX_FLAGS |
+User-defined C++ flags |
+valid C++
+compiler flags |
+None |
+
+CMAKE_CUDA_FLAGS |
+User-defined CUDA flags |
+valid CUDA
+compiler flags |
+None |
+
+INCFLO_DIM |
+Dimensionality of the build |
+2/3 |
+3 |
+
+INCFLO_MPI |
+Enable build with MPI |
+no/yes |
+yes |
+
+INCFLO_OMP |
+Enable build with OpenMP |
+no/yes |
+no |
+
+INCFLO_CUDA |
+Enable build with CUDA |
+no/yes |
+no |
+
+INCFLO_EB |
+Build Embedded Boundary
+support |
+no/yes |
+no |
+
+INCFLO_HYPRE |
+Enable HYPRE support |
+no/yes |
+no |
+
+INCFLO_FPE |
+Build with Floating-Point
+Exceptions checks |
+no/yes |
+no |
+
+
+
+
+
+
+Few more notes on building incflo
+The system default C++ compiler can be overwritten as follows:
+> cmake -DCMAKE_CXX_COMPILER=<c++-compiler> [options] ..
+
+
+When building on a platform that uses the module
utility, use either
+the above command (with full path to the compilers) or the following:
+> cmake -DCMAKE_CXX_COMPILER=CC [options] ..
+
+
+incflo uses the same compiler flags used to build AMReX, unless
+CMAKE_CXX_FLAGS
is explicitly provided, or
+the environmental variables CXXFLAGS
is set.
+
+
+Building incflo for Summit (OLCF)
+For the Summit cluster at OLCF, you first need to load/unload modules required to build incflo.
+> module unload xalt
+> module unload darshan
+> module load gcc
+> module load cmake/3.14.0
+
+
+Now incflo can be built following the SUPERBUILD Instructions (recommended).
+To build incflo for GPUs, you need to load cuda module:
+> module load cuda/10.1.105
+
+
+To compile for GPUs:
+> cd incflo
+> mdkir build
+> cd build
+> cmake -DCMAKE_CXX_COMPILER=g++ -DINCFLO_CUDA=yes
+> make -j
+
+
+An example of a submission_script for using the GPUs on Summit can be found in incflo/summit_script.sh
.
+For more information about Summit cluster: https://www.olcf.ornl.gov/for-users/system-user-guides/summit/
+
+
+
+Running the code
+incflo takes an input file as its first command-line argument. The file may
+contain a set of parameter definitions that will override defaults set in the code.
+Many example inputs files can be found in the incflo/test_XXX
directories.
+For example, to run the Taylor-Green vortices example in incflo/test_2d
, assuming your executable is named incflo.ex
+and is located in the test_2d
directory, then you can simply type:
+> cd incflo/test_2d
+> ./incflo.ex benchmark.taylor_green_vortices
+
+
+For more information on inputs options, see the Run-time Inputs section.
+incflo can generate subfolders in the current folder
+named plt00000
, plt00010
, etc, and chk00000
,
+chk00010
, etc. These are called plotfiles and checkpoint
+files. The plotfiles are used for visualization of derived fields; the checkpoint
+files are used for restarting the code. For more information see the AMReX Documentation on
+I/O (Plotfile, Checkpoint).
+
+