Skip to content

Commit

Permalink
Docs first draft
Browse files Browse the repository at this point in the history
  • Loading branch information
@EZoni
EZoni committed Sep 3, 2024
1 parent ba017dc commit af246bc
Showing 1 changed file with 124 additions and 89 deletions.
213 changes: 124 additions & 89 deletions Docs/source/developers/testing.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,39 +3,22 @@
Testing the code
================

When adding a new feature, you want to make sure that (i) you did not break the existing code and (ii) your contribution gives correct results. While existing capabilities are tested regularly remotely (when commits are pushed to an open PR on CI, and every night on local clusters), it can also be useful to run tests on your custom input file. This section details how to use both automated and custom tests.
When adding a new feature, you want to make sure that (i) you did not break the existing code and (ii) your contribution gives correct results. While the code is tested regularly remotely (on the cloud when commits are pushed to an open PR, and every night on local clusters), it can also be useful to run tests on your custom input file. This section details how to use both automated and custom tests.

Continuous Integration in WarpX
-------------------------------

Configuration
^^^^^^^^^^^^^

Our regression tests are using the suite published and documented at `AMReX-Codes/regression_testing <https://github.com/AMReX-Codes/regression_testing>`__.
Our regression tests are run with `CTest <https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing%20With%20CMake%20and%20CTest.html#>`__, an executable that comes with CMake.

Most of the configuration of our regression tests happens in ``Regression/Warpx-tests.ini``.
We slightly modify this file in ``Regression/prepare_file_ci.py``.
The test suite is ready to run once you have configured and built WarpX with CMake, following the instructions that you find in our :ref:`Users <install-cmake>` or :ref:`Developers <building-cmake>` sections.

For example, if you like to change the compiler to compilation to build on Nvidia GPUs, modify this block to add ``-DWarpX_COMPUTE=CUDA``:
A test that requires a build option that was not configured and built will be skipped automatically. For example, if you configure and build WarpX in 1D only, any test of dimensionality other than 1D, which would require WarpX to be configured and built in the corresponding dimensionality, will be skipped automatically.

.. code-block:: ini
[source]
dir = /home/regtester/AMReX_RegTesting/warpx
branch = development
cmakeSetupOpts = -DAMReX_ASSERTIONS=ON -DAMReX_TESTING=ON -DWarpX_COMPUTE=CUDA
We also support changing compilation options via the usual :ref:`build environment variables <building-cmake-envvars>`.
For instance, compiling with ``clang++ -Werror`` would be:

.. code-block:: sh
export CXX=$(which clang++)
export CXXFLAGS="-Werror"
Run Pre-Commit Tests Locally
----------------------------
How to run pre-commit tests locally
-----------------------------------

When proposing code changes to Warpx, we perform a couple of automated stylistic and correctness checks on the code change.
You can run those locally before you push to save some time, install them once like this:
Expand All @@ -47,97 +30,149 @@ You can run those locally before you push to save some time, install them once l
See `pre-commit.com <https://pre-commit.com>`__ and our ``.pre-commit-config.yaml`` file in the repository for more details.

How to run automated tests locally
----------------------------------

Once your new feature is ready, there are ways to check that you did not break anything.
WarpX has automated tests running every time a commit is pushed to an open pull request.
The input files and scripts used by the automated tests can be found in the `Examples <https://github.com/ECP-WarpX/WarpX/tree/development/Examples>`__ directory, either under `Physics_applications <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications>`__ or `Tests <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests>`__.

For easier debugging, it can be convenient to run the tests on your local machine by executing CTest as illustrated in the examples below (where we assume that WarpX was configured and built in the directory ``build``):

* List tests available for the current build options:

.. code-block:: sh
ctest --test-dir build -N
* Run tests available for the current build options:

.. code-block:: sh
ctest --test-dir build
* Run tests available for the current build options in parallel (while preserving existing dependencies between tests):

.. code-block:: sh
ctest --test-dir build -j 2
Run the test suite locally
* Run tests available for the current build options and output anything outputted by the test program if the test should fail:

.. code-block:: sh
ctest --test-dir build --output-on-failure
* Run tests available for the current build options with verbose output:

.. code-block:: sh
ctest --test-dir build --verbose
* Run tests matching the regular expression ``laser_acceleration``:

.. code-block:: sh
ctest --test-dir build -R laser_acceleration
* Run tests except those matching the regular expression ``laser_acceleration``:

.. code-block:: sh
ctest --test-dir build -E laser_acceleration
Once the execution of CTest is completed, you can find all files associated with each test in its corresponding directory under ``build/bin/``.
For example, if you run the single test ``test_3d_laser_acceleration``, you can find all files associated with this test in the directory ``build/bin/test_3d_laser_acceleration/``.

If you modify the code base locally and want to assess the effects of your code changes on the automated tests, you need to first rebuild WarpX including your code changes and then rerun CTest.

How to add automated tests
--------------------------

Once your new feature is ready, there are ways to check that you did not break anything.
WarpX has automated tests running every time a commit is added to an open pull request.
The list of automated tests is defined in `./Regression/WarpX-tests.ini <https://github.com/ECP-WarpX/WarpX/blob/development/Regression/WarpX-tests.ini>`__.
As mentioned above, the input files and scripts used by the automated tests can be found in the `Examples <https://github.com/ECP-WarpX/WarpX/tree/development/Examples>`__ directory, either under `Physics_applications <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications>`__ or `Tests <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests>`__.

For easier debugging, it can be convenient to run the tests on your local machine by executing the script
`./run_test.sh <https://github.com/ECP-WarpX/WarpX/blob/development/run_test.sh>`__ from WarpX's root folder, as illustrated in the examples below:
Each test directory must contain a file named ``CMakeLists.txt`` where all tests associated with the input files and scripts in that directory must be listed.

.. code-block:: sh
A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as illustrated in the examples below:

# Example:
# run all tests defined in ./Regression/WarpX-tests.ini
./run_test.sh
* Add the **regular test** ``test_1d_laser_acceleration``, with input file and analysis script in ``Examples/Tests/laser_acceleration/``:

# Example:
# run only the test named 'pml_x_yee'
./run_test.sh pml_x_yee
.. code-block:: sh
# Example:
# run only the tests named 'pml_x_yee', 'pml_x_ckc' and 'pml_x_psatd'
./run_test.sh pml_x_yee pml_x_ckc pml_x_psatd
add_warpx_test(
test_1d_laser_acceleration # name
1 # dims
2 # nprocs
OFF # eb
Examples/Tests/laser_acceleration/inputs_test_1d_laser_acceleration # inputs
Examples/Tests/laser_acceleration/analysis.py # analysis
diags/diag1000100 # output (plotfile)
OFF # dependency
)
Note that the script `./run_test.sh <https://github.com/ECP-WarpX/WarpX/blob/development/run_test.sh>`__ runs the tests with the exact same compile-time options and runtime options used to run the tests remotely.
* Add the **PICMI test** ``test_2d_laser_acceleration_picmi``, with input file and analysis script in ``Examples/Tests/laser_acceleration/``:

Moreover, the script `./run_test.sh <https://github.com/ECP-WarpX/WarpX/blob/development/run_test.sh>`__ compiles all the executables that are necessary in order to run the chosen tests.
The default number of threads allotted for compiling is set with ``numMakeJobs = 8`` in `./Regression/WarpX-tests.ini <https://github.com/ECP-WarpX/WarpX/blob/ad74bcbdd131a8797339ba38370b1195d0aecffb/Regression/WarpX-tests.ini#L20>`__.
However, when running the tests on a local machine, it is usually possible and convenient to allot more threads for compiling, in order to speed up the builds.
This can be accomplished by setting the environment variable ``WARPX_CI_NUM_MAKE_JOBS``, with the preferred number of threads that fits your local machine, e.g. ``export WARPX_CI_NUM_MAKE_JOBS=16`` (or less if your machine is smaller).
On public CI, we overwrite the value to ``WARPX_CI_NUM_MAKE_JOBS=2``, in order to avoid overloading the available remote resources.
Note that this will not change the number of threads used to run each test, but only the number of threads used to compile each executable necessary to run the tests.
.. code-block:: sh
Once the execution of `./run_test.sh <https://github.com/ECP-WarpX/WarpX/blob/development/run_test.sh>`__ is completed, you can find all the relevant files associated with each test in one single directory.
For example, if you run the single test ``pml_x_yee``, as shown above, on 04/30/2021, you can find all relevant files in the directory ``./test_dir/rt-WarpX/WarpX-tests/2021-04-30/pml_x_yee/``.
The content of this directory will look like the following (possibly including backtraces if the test crashed at runtime):
add_warpx_test(
test_2d_laser_acceleration_picmi # name
2 # dims
2 # nprocs
OFF # eb
Examples/Tests/laser_acceleration/inputs_test_2d_laser_acceleration_picmi.py # inputs
Examples/Tests/laser_acceleration/analysis.py # analysis
diags/diag1000100 # output (plotfile)
OFF # dependency
)
.. code-block:: sh
* Add the **restart test** ``test_3d_laser_acceleration_restart``, with input file in ``Examples/Tests/laser_acceleration/`` and default restart analysis script provided by WarpX:

.. code-block:: sh
$ ls ./test_dir/rt-WarpX/WarpX-tests/2021-04-30/pml_x_yee/
analysis_pml_yee.py # Python analysis script
inputs_2d # input file
main2d.gnu.TEST.TPROF.MTMPI.OMP.QED.ex # executable
pml_x_yee.analysis.out # Python analysis output
pml_x_yee.err.out # error output
pml_x_yee.make.out # build output
pml_x_yee_plt00000/ # data output (initialization)
pml_x_yee_plt00300/ # data output (last time step)
pml_x_yee.run.out # test output
add_warpx_test(
test_3d_laser_acceleration_restart # name
3 # dims
2 # nprocs
OFF # eb
Examples/Tests/laser_acceleration/inputs_test_3d_laser_acceleration_restart # inputs
Examples/analysis_default_restart.py # analysis
diags/diag1000100 # output (plotfile)
test_3d_laser_acceleration # dependency
)
Note that the restart has an explicit dependency, namely it can run only provided that the original test, from which the restart checkpoint files will be read, runs first.

Add a test to the suite
-----------------------
* A more complex example. Add the **PICMI test** ``test_rz_laser_acceleration_picmi``, with custom command-line arguments ``--test`` and ``dir``, input file and analysis script in ``Examples/Tests/laser_acceleration/``, and openPMD time series output:

There are three steps to follow to add a new automated test (illustrated here for PML boundary conditions):
.. code-block:: sh
* An input file for your test, in folder `Example/Tests/...`. For the PML test, the input file is at ``Examples/Tests/pml/inputs_2d``. You can also re-use an existing input file (even better!) and pass specific parameters at runtime (see below).
* A Python script that reads simulation output and tests correctness versus theory or calibrated results. For the PML test, see ``Examples/Tests/pml/analysis_pml_yee.py``. It typically ends with Python statement ``assert( error<0.01 )``.
* If you need a new Python package dependency for testing, add it in ``Regression/requirements.txt``
* Add an entry to ``Regression/WarpX-tests.ini``, so that a WarpX simulation runs your test in the continuous integration process, and the Python script is executed to assess the correctness. For the PML test, the entry is
add_warpx_test(
test_rz_laser_acceleration_picmi # name
RZ # dims
2 # nprocs
OFF # eb
"Examples/Tests/laser_acceleration/inputs_test_rz_laser_acceleration_picmi.py --test --dir 1" # inputs
Examples/Tests/laser_acceleration/analysis.py # analysis
diags/diag1/ # output (openPMD time series)
OFF # dependency
)
.. code-block::
If you need a new Python package dependency for testing, please add it in `Regression/requirements.txt <https://github.com/ECP-WarpX/WarpX/blob/development/Regression/requirements.txt>`__.

[pml_x_yee]
buildDir = .
inputFile = Examples/Tests/pml/inputs2d
runtime_params = warpx.do_dynamic_scheduling=0 algo.maxwell_solver=yee
dim = 2
addToCompileString =
cmakeSetupOpts = -DWarpX_DIMS=2
restartTest = 0
useMPI = 1
numprocs = 2
useOMP = 1
numthreads = 1
compileTest = 0
doVis = 0
analysisRoutine = Examples/Tests/pml/analysis_pml_yee.py
Sometimes, two tests share a large number of input parameters. The shared input parameters can be collected in a "base" input file that can be passed as a runtime parameter in the actual test input files through the parameter ``FILE``.

If you re-use an existing input file, you can add arguments to ``runtime_params``, like ``runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=100 plasma_e.zmin=-200.e-6``.
Naming conventions for automated tests
--------------------------------------

.. note::
Note that we currently obey the following snake\_case naming conventions for test names and test input files (which make automation tasks easier, e.g., parsing visually, parsing through code, sorting alphabetically, filtering tests in CTest via -R, etc.):

If you added ``analysisRoutine = Examples/analysis_default_regression.py``, then run the new test case locally and add the :ref:`checksum <developers-checksum>` file for the expected output.
#. **Regular test names** start with the string ``test_1d_``, ``test_2d_``, ``test_3d_`` or ``test_rz_``, followed by a string that is descriptive of the test. For example, ``test_3d_laser_acceleration``.

.. note::
#. **PICMI test names** start with the string ``test_1d_``, ``test_2d_``, ``test_3d_`` or ``test_rz_``, followed by a string that is descriptive of the test, and end with the string ``_picmi``. For example, ``test_3d_laser_acceleration_picmi``.

We run those tests on our continuous integration services, which at the moment only have 2 virtual CPU cores.
Thus, make sure that the product of ``numprocs`` and ``numthreads`` for a test is ``<=2``.
#. **Restart test names** end with the string ``_restart``. For example, ``test_3d_laser_acceleration_restart``.

#. **Test input files** start with the string ``inputs_`` followed by the test name. For example, ``inputs_test_3d_laser_acceleration`` or ``inputs_test_3d_laser_acceleration_picmi.py`` or ``inputs_test_3d_laser_acceleration_restart``.

Useful tool for plotfile comparison: ``fcompare``
-------------------------------------------------
Expand Down

0 comments on commit af246bc

Please sign in to comment.