deploy: bf363d8

AMReX-Astro · Jun 18, 2024 · 9809eb6 · 9809eb6
1 parent cc87bd7
commit 9809eb6
Show file tree

Hide file tree

Showing 138 changed files with 468 additions and 57 deletions.
diff --git a/docs/dev/AMR.html b/docs/dev/AMR.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/ConvertCheckpoint.html b/docs/dev/ConvertCheckpoint.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/EOS.html b/docs/dev/EOS.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/FlowChart.html b/docs/dev/FlowChart.html
@@ -82,6 +82,7 @@
 </ul>
 </li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>
@@ -179,7 +180,7 @@ <h2>Introduction<a class="headerlink" href="#introduction" title="Link to this h
 time using a simple quadrature rule, and integrates the hydro
 explicitly and reactions implicitly to the next time node.
 Iterations allow each process to see one another and achieve
-high-order in time convergence.  This is described in <span id="id3">[<a class="reference internal" href="zreferences.html#id49" title="M. Zingale, M. P. Katz, J. B. Bell, M. L. Minion, A. J. Nonaka, and W. Zhang. Improved Coupling of Hydrodynamics and Nuclear Reactions via Spectral Deferred Corrections. Astrophysical Journal, 886(2):105, Dec 2019. arXiv:1908.03661, doi:10.3847/1538-4357/ab4e1d.">75</a>]</span>.</p></li>
+high-order in time convergence.  This is described in <span id="id3">[<a class="reference internal" href="zreferences.html#id49" title="M. Zingale, M. P. Katz, J. B. Bell, M. L. Minion, A. J. Nonaka, and W. Zhang. Improved Coupling of Hydrodynamics and Nuclear Reactions via Spectral Deferred Corrections. Astrophysical Journal, 886(2):105, Dec 2019. arXiv:1908.03661, doi:10.3847/1538-4357/ab4e1d.">76</a>]</span>.</p></li>
 </ul>
 </li>
 </ul>
@@ -702,7 +703,7 @@ <h3>Single Step Flowchart<a class="headerlink" href="#single-step-flowchart" tit
 <p>The code must be compiled with <code class="docutils literal notranslate"><span class="pre">USE_TRUE_SDC</span> <span class="pre">=</span> <span class="pre">TRUE</span></code> to use this
 evolution type.</p>
 </div>
-<p>The SDC solver follows the algorithm detailed in <span id="id8">[<a class="reference internal" href="zreferences.html#id49" title="M. Zingale, M. P. Katz, J. B. Bell, M. L. Minion, A. J. Nonaka, and W. Zhang. Improved Coupling of Hydrodynamics and Nuclear Reactions via Spectral Deferred Corrections. Astrophysical Journal, 886(2):105, Dec 2019. arXiv:1908.03661, doi:10.3847/1538-4357/ab4e1d.">75</a>]</span>.
+<p>The SDC solver follows the algorithm detailed in <span id="id8">[<a class="reference internal" href="zreferences.html#id49" title="M. Zingale, M. P. Katz, J. B. Bell, M. L. Minion, A. J. Nonaka, and W. Zhang. Improved Coupling of Hydrodynamics and Nuclear Reactions via Spectral Deferred Corrections. Astrophysical Journal, 886(2):105, Dec 2019. arXiv:1908.03661, doi:10.3847/1538-4357/ab4e1d.">76</a>]</span>.
 We write our evolution equation as:</p>
 <div class="math notranslate nohighlight">
 \[\frac{\partial \Ub}{\partial t} = {\bf A}(\Ub) + {\bf R}(\Ub)\]</div>

diff --git a/docs/dev/Hydrodynamics.html b/docs/dev/Hydrodynamics.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>
@@ -1306,7 +1307,7 @@ <h2>Hybrid Momentum<a class="headerlink" href="#hybrid-momentum" title="Link to
 ultimately doing the conservative update in terms of the cylindrical momentum.  Additional
 source terms appear in this formulation, which are written out in <span id="id17">[<a class="reference internal" href="zreferences.html#id21" title="Z. D. Byerly, B. Adelstein-Lelbach, J. E. Tohline, and D. C. Marcello. A Hybrid Advection Scheme for Conserving Angular Momentum on a Refined Cartesian Mesh. Astrophysical Journal Supplement Series, 212:23, June 2014. arXiv:1404.5942, doi:10.1088/0067-0049/212/2/23.">25</a>]</span>.</p>
 <p>The <code class="docutils literal notranslate"><span class="pre">rotating_torus</span></code> problem gives a good test for this.  This problem
-originated with <span id="id18">[<a class="reference internal" href="zreferences.html#id22" title="J. C. B. Papaloizou and J. E. Pringle. The dynamical stability of differentially rotating discs with constant specific angular momentum. Monthly Notices of the Royal Astronomical Society, 208:721-750, June 1984. URL: http://adsabs.harvard.edu/abs/1984MNRAS.208..721P.">59</a>]</span>.  The
+originated with <span id="id18">[<a class="reference internal" href="zreferences.html#id22" title="J. C. B. Papaloizou and J. E. Pringle. The dynamical stability of differentially rotating discs with constant specific angular momentum. Monthly Notices of the Royal Astronomical Society, 208:721-750, June 1984. URL: http://adsabs.harvard.edu/abs/1984MNRAS.208..721P.">60</a>]</span>.  The
 problem is initialized as a torus with constant specific angular
 momentum, as shown below:</p>
 <figure class="align-default" id="id23">

diff --git a/docs/dev/Introduction.html b/docs/dev/Introduction.html
@@ -70,6 +70,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/MAESTRO_restart.html b/docs/dev/MAESTRO_restart.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/Particles.html b/docs/dev/Particles.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/Preface.html b/docs/dev/Preface.html
@@ -68,6 +68,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>

diff --git a/docs/dev/Verification.html b/docs/dev/Verification.html
@@ -66,6 +66,7 @@
 <li class="toctree-l1"><a class="reference internal" href="mpi_plus_x.html">Running Options: CPUs and GPUs</a></li>
 <li class="toctree-l1"><a class="reference internal" href="FlowChart.html">Flowchart</a></li>
 <li class="toctree-l1"><a class="reference internal" href="software.html">Software Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="gpu.html">GPU Programming Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="problem_setups.html">Distributed Problem Setups</a></li>
 <li class="toctree-l1"><a class="reference internal" href="timestepping.html">Timestepping and Retries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="creating_a_problem.html">Setting Up Your Own Problem</a></li>
@@ -170,7 +171,7 @@ <h3>Sod’s Problem (and Other Shock Tube Problems)<a class="headerlink" href="#
 exact Riemann solver from Toro <span id="id1">[<a class="reference internal" href="zreferences.html#id39" title="E. F. Toro. Riemann Solvers and Numerical Methods for Fluid Dynamics. Springer, 1997.">13</a>]</span>, Chapter 4.</p>
 <section id="sods-problem">
 <h4>Sod’s Problem<a class="headerlink" href="#sods-problem" title="Link to this heading"></a></h4>
-<p>The Sod problem <span id="id2">[<a class="reference internal" href="zreferences.html#id40" title="G. A. Sod. A survey of several finite difference methods for systems of nonlinear hyperbolic conservation la ws. Journal of Computational Physics, 27:1-31, April 1978. doi:10.1016/0021-9991(78)90023-2.">62</a>]</span> is a simple shock tube problem that
+<p>The Sod problem <span id="id2">[<a class="reference internal" href="zreferences.html#id40" title="G. A. Sod. A survey of several finite difference methods for systems of nonlinear hyperbolic conservation la ws. Journal of Computational Physics, 27:1-31, April 1978. doi:10.1016/0021-9991(78)90023-2.">63</a>]</span> is a simple shock tube problem that
 exhibits a shock, contact discontinuity, and a rarefaction wave.
 The initial conditions are:</p>
 <div class="math notranslate nohighlight">
@@ -396,7 +397,7 @@ <h3>Sedov Problem<a class="headerlink" href="#sedov-problem" title="Link to this
 test problem. A large amount of energy is placed into a very small
 volume, driving a spherical (or cylindrical in 2-d Cartesian
 coordinates) blast wave. Analytic solutions were found by Sedov
-<span id="id5">[<a class="reference internal" href="zreferences.html#id41" title="L. I. Sedov. Similarity and Dimensional Methods in Mechanics. Academic Press, 1959. translated from the 4th Russian Ed.">61</a>]</span>.</p>
+<span id="id5">[<a class="reference internal" href="zreferences.html#id41" title="L. I. Sedov. Similarity and Dimensional Methods in Mechanics. Academic Press, 1959. translated from the 4th Russian Ed.">62</a>]</span>.</p>
 <p>A cylindrical blast wave (e.g. a point explosion in a 2-d plane) can
 be modeled in 2-d Cartesian coordinates. A spherical blast wave can
 be modeled in 1-d spherical, 2-d axisymmetric (cylindrical <span class="math notranslate nohighlight">\(r\)</span>-<span class="math notranslate nohighlight">\(z\)</span>), or 3-d
@@ -441,7 +442,7 @@ <h3>Sedov Problem<a class="headerlink" href="#sedov-problem" title="Link to this
 density, <span class="math notranslate nohighlight">\(\rho_\mathrm{ambient}\)</span>, and pressure, <span class="math notranslate nohighlight">\(p_\mathrm{ambient}\)</span>.
 Initializing the problem can be difficult because the small volume is
 typically only a cell in extent. This can lead to grid imprinting in
-the solution. A standard solution (see for example <span id="id7">[<a class="reference internal" href="zreferences.html#id43" title="M. Omang, S. Børve, and J. Trulsen. SPH in spherical and cylindrical coordinates. Journal of Computational Physics, 213:391-412, March 2006. doi:10.1016/j.jcp.2005.08.023.">58</a>]</span>
+the solution. A standard solution (see for example <span id="id7">[<a class="reference internal" href="zreferences.html#id43" title="M. Omang, S. Børve, and J. Trulsen. SPH in spherical and cylindrical coordinates. Journal of Computational Physics, 213:391-412, March 2006. doi:10.1016/j.jcp.2005.08.023.">59</a>]</span>
 and the references therein)
 is to convert the explosion energy into a pressure contained within a
 certain volume, <span class="math notranslate nohighlight">\(V_\mathrm{init}\)</span>, of radius <span class="math notranslate nohighlight">\(r_\mathrm{init}\)</span> as</p>
@@ -592,7 +593,7 @@ <h3>Radiation Source Problem<a class="headerlink" href="#radiation-source-proble
 each case, the gas energy and radiation field will evolve until
 thermal equilibrium is achieved.</p>
 <p>Our implementation of this problem follows that of
-<span id="id9">[<a class="reference internal" href="zreferences.html#id45" title="F. D. Swesty and E. S. Myra. A Numerical Algorithm for Modeling Multigroup Neutrino-Radiation Hydrodynamics in Two Spatial Dimensions. Astrophysical Journal Supplement, 181:1-52, March 2009. doi:10.1088/0067-0049/181/1/1.">65</a>]</span>.</p>
+<span id="id9">[<a class="reference internal" href="zreferences.html#id45" title="F. D. Swesty and E. S. Myra. A Numerical Algorithm for Modeling Multigroup Neutrino-Radiation Hydrodynamics in Two Spatial Dimensions. Astrophysical Journal Supplement, 181:1-52, March 2009. doi:10.1088/0067-0049/181/1/1.">66</a>]</span>.</p>
 <figure class="align-center" id="id20">
 <a class="reference internal image-reference" href="_images/radiating_source.png"><img alt="radiatin source" src="_images/radiating_source.png" style="width: 5in;" /></a>
 <figcaption>
@@ -616,7 +617,7 @@ <h3>Radiating Sphere<a class="headerlink" href="#radiating-sphere" title="Link t
 radiation energy as a function of energy group at a specified time and
 distance from the radiating sphere.</p>
 <p>Our implementation of this problem is in Exec/radiation_tests/RadSphere and
-follows that of <span id="id11">[<a class="reference internal" href="zreferences.html#id45" title="F. D. Swesty and E. S. Myra. A Numerical Algorithm for Modeling Multigroup Neutrino-Radiation Hydrodynamics in Two Spatial Dimensions. Astrophysical Journal Supplement, 181:1-52, March 2009. doi:10.1088/0067-0049/181/1/1.">65</a>]</span>. The routine that computes
+follows that of <span id="id11">[<a class="reference internal" href="zreferences.html#id45" title="F. D. Swesty and E. S. Myra. A Numerical Algorithm for Modeling Multigroup Neutrino-Radiation Hydrodynamics in Two Spatial Dimensions. Astrophysical Journal Supplement, 181:1-52, March 2009. doi:10.1088/0067-0049/181/1/1.">66</a>]</span>. The routine that computes
 the analytic solution is provided as analytic.f90.</p>
 <figure class="align-default" id="id21">
 <a class="reference internal image-reference" href="_images/radiating_sphere.png"><img alt="radiating sphere" src="_images/radiating_sphere.png" style="width: 5in;" /></a>

diff --git a/docs/dev/_sources/gpu.rst.txt b/docs/dev/_sources/gpu.rst.txt
@@ -0,0 +1,58 @@
+*********************
+GPU Programming Model
+*********************
+
+CPUs and GPUs have separate memory, which means that working on both
+the host and device may involve managing the transfer of data between
+the memory on the host and that on the GPU.
+
+In Castro, the core design when running on GPUs is that all of the compute
+should be done on the GPU.
+
+When we compile with ``USE_CUDA=TRUE`` or ``USE_HIP=TRUE``, AMReX will allocate
+a pool of memory on the GPUs and all of the ``StateData`` will be stored there.
+As long as we then do all of the computation on the GPUs, then we don't need
+to manage any of the data movement manually.
+
+.. note::
+
+   We can tell AMReX to allocate the data using managed-memory by
+   setting:
+
+   ::
+
+      amrex.the_arena_is_managed = 1
+
+   This is generally not needed.
+
+The programming model used throughout Castro is C++-lambda-capturing
+by value.  We access the ``FArrayBox`` stored in the ``StateData``
+``MultiFab`` by creating an ``Array4`` object.  The ``Array4`` does
+not directly store a copy of the data, but instead has a pointer to
+the data in the ``FArrayBox``.  When we capture the ``Array4`` by
+value in the GPU kernel, the GPU gets access to the pointer to the
+underlying data.
+
+
+Most AMReX functions will work on the data directly on the GPU (like
+``.setVal()``).
+
+In rare instances where we might need to operate on the data on the
+host, we can force a copy to the host, do the work, and then copy
+back.  For an example, see the reduction done in  ``Gravity.cpp``.
+
+.. note::
+
+   For a thorough discussion of how the AMReX GPU offloading works
+   see :cite:`amrex-ecp`.
+
+
+Runtime parameters
+------------------
+
+The main exception for all data being on the GPUs all the time are the
+runtime parameters.  At the moment, these are allocated as managed
+memory and stored in global memory.  This is simply to make it easier
+to read them in and initialize them on the CPU at runtime.
+
+
diff --git a/docs/dev/_sources/index.rst.txt b/docs/dev/_sources/index.rst.txt
@@ -21,6 +21,7 @@ https://github.com/amrex-astro/Castro
    mpi_plus_x
    FlowChart
    software
+   gpu
    problem_setups
    timestepping
    creating_a_problem