ReFrame 2.18

ReFrame 2.18 Release Notes

New features and enhancements

• Print performance variable name in case of performance errors (#791)
• Print task allocation type (standard or flexible) for tests in detailed listing (#786)
• Do not use absolute paths in generated batch scripts (#779)
• Add option to print performance report (#733)
• Remove deprecated --logdir command-line option and the corresponding attribute in the logging configuration (#741)
• Enable environment variable substitution in shell code emitted by build systems (#744)

Bug fixes

• Fix crash when using syslog logging handler with the -v option (#808)
• Add version checks for TMod and TMod4 (#794)
• Fix exception message formatting in reservation node extraction (#793)
• Fix filename order in build error message (#790)
• Do not crash when invalid regular expressions are passed to the test selection options (#757)
• Force units in gperftools report (#759)
• Fix gdb4hpc failure with C++ (#747)
• Fix failure in Intel Advisor Roofline check (#745)
• Fix duplicate module commands emitted while executing test cases (#740)
• Fix typo in Likwid bandwidth benchmark (#735)
• Fix crash in flexible task allocation due to refactoring of SystemPartition (#729)
• Remove unnecessary module from the "Hello, World" tests (#726)
• Fix string error in header file of CUDA memtest (#718)
• Fix deprecation warning in check filter functions unit tests (#723)

Regression tests

• Revise MAGMA library benchmark (#783)
• Add simple FFTW benchmark (#776)
• Make OSU tests check generic (#801)
• Make the STREAM test generic (#799)
• Make alloc_speed check generic (#800)
• Make CUDA kernel latency tests generic (#796)
• Hardcode version in Intel Vtune test to prevent failure when default changes (#785)
• Add Roofline check of the GPP code using NVIDIA nvprof (#778)
• Add ert roofline check (gpu) (#775)
• Add ert roofline check (#773)
• Add sde roofline check (#774)
• Fix ddt+cuda/10.0 check (#772)
• Add CUDA shared memory bandwidth benchmark (#770)
• Adapt TensorFlow+Horovod test to CrayGNU-19.03 (#771)
• Update Tensorflow check to r1.12.0 (#768)
• Add flexible HPCG GPU benchmark test (#762)
• Update Score-P check for PE/19.03 (#767)
• Update IPM check for PE/19.03 (#766)
• Adapt Paraview GPU check to CLE7 (#758)
• Update magma tests (#754)
• Update Jacobi test for CLE7 (#727)
• Fix ldflags for nvprof, cuda_gdb tests (#751)
• Update boost python test to use the latest module (#748)
• Update libsci_acc_symlink tests for PE19.03 (#734)
• Update Nvidia35Resolve test for the new PE (#732)
• Update perftools check for CLE7 (#731)
• Rename for loop variable to fix warnings in kernel latency test (#750)
• Add test for gdb4hpc (#603)
• Enable small scale application tests also on Daint (#738)
• Sort cores list in affinity test before comparing (#739)
• Add Intel Advisor Roofline test (#721)
• Add mpiP parallel test (#720)
• Add gperftools parallel test (#719)

Other

• Require full tag names in genrelnotes.py script (#807)
• Expand Ault configuration (#782)
• Add Ault TDS to the config file (#777)
• Update deployment script (#755)
• Add PyPI badge to README (#752)
• Disable running modified tests on Daint (#749)
• Prepare ReFrame for deployment to PyPI (#743)
• Document the --performance-report command line option (#761)
• Fix reference to exclusive_access in tutorial (#742)
• Ongoing work for supporting containers (#787)
• Ongoing work for supporting test dependencies (#769, #797)

ReFrame 2.17

ReFrame 2.17 Release Notes

New features and enhancements

• Add support for selecting tests using regular expressions (#664)
• Add logging handler for sending records to syslog (#675)
• Remove support for passing options to the RegressionTest.compile() method (#697)
• Support command substitution in variable expansion (#683)
• Drop support for setting flags in environment (#678)
• Extend Job API for retrieving the list of nodes assigned to the regression test (#660)
• Add support for directories in keep_files attribute (#673)

Bug fixes

• Define the set of run environments correctly when -p option is used with regular expressions (#716)
• Fix Slurm backend failures when trying node lists with inexistent nodes (#713)
• Fix unit test failure with EBUSY when running on NFS filesystem (#715)
• Fix error message in case of performance errors (#704)
• Fix deprecation warnings with Python 3.7 (#711)
• Correct the units of the kernel latency test (#703)
• Fix flexible task allocation bug ignoring Slurm partition defined in access (#693)
• Use the default Slurm partition with --flex-alloc-tasks if none is specified (#676)
• Fix crash in failure report when test is killed due to timeout (#680)
• Fix bug in 'keep_files' unit tests (#672)

Regression tests

• Add OSU Allreduce test (#686)
• Add Intel Vtune test (#714)
• Add description to the collective halo tests (#690)
• Add Intel Inspector test (#710)
• Add Fulen as a supported system in CSCS config (#684)
• Add MPI halo cell exchange benchmark (#666)
• Enlarge buffer sizes for H2D and D2D multi GPU tests (#689)
• Extend DGEMM test to build with OpenBLAS (#691)
• Add test for COSMO post-processing with Fieldextra (#685)
• Change resources dir for GROMACS test (#695)
• Fix tags of flexible tests + other improvements (#681)
• Add test for gperftools (#670)
• Add CPU affinity test (#657)
• Add test for NERSC IPM (#631)
• Update maintenance tests (#679)
• Extend DGEMM test for using MKL (#662)

Other

• Reenable Dom in the Jenkinsfile (#709)
• Disable Dom from running in the ci (#696)
• Add documentation for regular expression support in test selection options (#717)
• Move documentation to readthedocs.org (#705)
• Remove link to prior to 2.6.1 documentation (#687)

ReFrame 2.16

ReFrame 2.16 Release Notes

New features and enhancements

• Add support for setting a minimum number of required tasks in tests with flexible task allocation (#652)
• Add command line option for increasing the verbosity level of output (#650)
• Add command line options for showing the system and environments configuration (#633)
• Print error information when a modules system operation fails (#628)
• Add support for units in performance references (#576)
• Add a new count_uniq() sanity function (#584)

Bug fixes

• Improve handling of bad tests to avoid frontend crashing (#623)
• Fix bug in variable names in Slurm backend (#639)
• Do not crash if Slurm node does not contain partition information (#621)
• Check if 'UnavailableNodes' are down to cancel job (#599)
• Fix deadlock in the flexible task allocation when the node list output is too large (#614)

Regression tests

• Port remaining tests to the new syntax (#656)
• Add Likwid benchmarks for cache, memory and cross-socket bandwidths (#647)
• Add new MPI_Init_thread() check (#620)
• Add benchmark measuring CUDA kernel launch latency (#646)
• Add HPX 1D stencil tests (#617)
• Add tests for SPEC-ACCEL benchmarks (#618)
• Add test for distributed training with TensorFlow and Horovod (#637)
• Add flexible HPCG test (#604)
• Add HPX hello world test (#616)
• Add allocation speed check (#597)
• Fail if test is not compiled with OpenACC (#615)
• Convert GpuBandwidthCheck to a flexible diagnostic check (#586)
• Testing the environment of the login nodes when accessed remotely over SSH (#583)
• Promote DGEMM to a multi-node check (#530)
• Enable NCO and CDO for the C2SM-RCM environments (#624)
• Make Alltoallv tests non-strict (#622)
• Add specific tag for MCH tests (#594)
• Remove ifdef USE_MPI from the MCH OpenACC/CUDA/MPI/C++ test (#585)
• Set exclusive_access for Kesch tests (#591)
• Adapt MCH regression tests to C2SM-RCM environments (#513)
• Adapt CP2K test (#596)

Other

• Make .pep8speaks.yml consistent with coding style (#649)
• Add option to disable CSCS CI tests (#648)
• Enable pep8speaks for code style checking in the repository (#626)
• Use integer comparison for ReFrame versions in deployment script (#600)
• Replace python 3.7-dev with 3.7 in .travis.yml (#588)
• Document verbosity option (#665)
• Add link for the automatic invites to Slack (#661)
• Update README with contact info (#642)
• Move SC18 presentation & demo to docs (#606)

ReFrame 2.15

ReFrame 2.15 Release Notes

New features and enhancements

• Update retries stage/output directory naming convention (#580)
• Full support for flexible task allocation (#522)
• Add new command-line option for more verbose test listing (#506)
• List only the checks that are supported on the current system (#487)
• Use different stage and output directories for different retries of the same test (#464)

Bug fixes

• Directly assign to num_tasks if --flex-alloc-tasks has an integer value (#558)
• Show also hidden partitions in Slurm, when trying to flexibly allocate tasks (#553)
• Fix redundant export statements (#448)
• Fix retry report when test fails before the setup phase (#535)
• Do not silently abort tasks in hold when all currently running tasks fail (#533)
• Remove the validation of the name of the test's base classes (#480)
• Remove time limit from build jobs (#474)

Regression tests

• Update NAMD test (#574)
• Run libsci_acc and libsci_resolve tests on compute nodes as well (#581)
• Update GROMACS test for Dom (#579)
• Ruby 2.2.2 NArray test (#575)
• Add test checking if CRAY_CPU_TARGET is set (#572)
• Update MAGMA tests (#571)
• Update CP2K test (#559)
• Adapt the GPU burn test for multi-gpu nodes (#564)
• Disable Trilinos tests which fail due to the corresponding module (#551)
• Adapt usertool tests to new syntax (#552)
• Update GROMACS performance numbers (#548)
• Remove Monch from supported systems (#537)
• Remove read-only files attribute from the CP2K check (#529)
• Remove wildcards from programming environment specification in tests (#534)
• Fix NetCDF test on Kesch (#523)
• Sync OpenMP versions for Dom/Daint in notool test (#519)
• Add MCH CUDA stress test (#514)
• Adapt boost tests to latest pe18.08 (#518)
• Adapt the TensorFlow module name for PE 18.08 (#516)
• Adapt Score-P modules to PE 18.08 (#515)
• Add MCH halo exchange benchmark (#509)
• Adapt IOR checks to build systems (#505)
• Add MCH alltoallv check (#347)
• Adjust LAMMPS performance values (#504)
• Adapt the LAMMPS test to PE18.08 (#502)
• Adapt QE test to PE18.08 (#501)
• Remove PrgEnv-gnu support from the Scalapack test on Kesch (#500)
• Adapt NetCDF test to Kesch (#482)
• New OpenACC/CUDA/C++ test from MCH (#342)
• Adapt Kesch tests to final CSCS prog. environments (#479)
• Publish MAGMA tests and adapt to PE 18.08 (#463)
• Remove the matrix file from version control (#484)
• Set required version for tests using build systems (#476)
• Add matrix file add20.rua in the src dir (#481)
• Adapt trilinos regression test to PE18.08 and build systems (#466)
• Adapt gpu direct test to build systems (#462)
• Adapt CUDA NVML test to build systems (#461)
• Increase GROMACS sanity check tolerance to 0.1% (#459)
• Replace FFLAGS with FCFLAGS in Makefiles (#457)

Other

• Add C2SM environments on Kesch (#512)
• Re-enable running modified tests on Daint (#517)
• Adapt Kesch settings (#477)
• Automate deployment and generation of release notes (#460)
• Document the --flex-alloc-tasks feature (#582)
• Add User Lab Day slides to documentation (#520)
• Update documentation on the topic of filtering tests by system (#492)

ReFrame 2.14

ReFrame 2.14 Release Notes

New features and enhancements

• Add support for Autotools-based projects (#452)
• Add support for CMake (#447)
• Add a new infrastructure for build systems (#340)
  • Setting compilation flags directly in the programming environment is now deprecated.
  • Passing arguments to the compile() method is now deprecated.
  • For more information on how to use the new build system feature, please refer to the tutorial and the reference guide.
• Evaluate and log all performance values before asserting them (#359)
• Allow purging modules before running regression tests (#399)

Bug fixes

• Fix reading of logdir setting from system config (#445)
• Convert message for inexistent test files to warning (#440)
• Do not load environments twice (#441)
• Add a persistent version of rmtree() (#437)
• Filter out empty strings before writing preamble (#410)
• Fix bug in unloading of conflicting modules (#405)
• Enable sphinx versioning only for releases (#361)
• Change the resources dir path (#358)
• Fix expected OpenMP versions for Dom (#350)
• Comment out Graylog handler from CSCS config (#352)
• Fix site_configuration in docs (#344)

Regression tests

• Adapt stream microbenchmark test to build systems (#420)
• Adapt prgenv test to build systems (#449)
• Fix sourcesdir for the scorep test (#455)
• Update OSU microbenchmark test to new syntax and build system (#442)
• Adapt GpuBandwidthTest to build_systems (#438)
• Adapt perftools to use build systems (#439)
• Adapt helloworld tests to build systems (#416)
• Adapt netcdf tests to build systems (#434)
• Adapt cdo test to PE 18.07 (#431)
• Adapt ddt regression test to PE 18.07 (#425)
• Adapt cuda_gdb to build systems (#428)
• Adapt notools check to build systems (#427)
• Adapt hdf5 tests to build systems (#422)
• Adapt libsci_resolve test to PE 18.07 (#424)
• Adapt scorep test to PE 18.07 (#426)
• Adapt cuda checks to build systems (#421)
• Adapt dgemm test to use build systems (#418)
• Adapt the scalapack tests to use build systems (#417)
• Adapt haswell_fma to buildsystems style (#415)
• Adapt nvprof test to PE 18.07 (#414)
• Adapt the nco test for PE 18.07 (#412)
• Adapt Tensorflow tests for PE 18.07 (#409)
• Adapt the Boost Python test for PE 18.07 (#408)
• Adapt Paraview check for PE 18.07 (#401)
• Adapt the petsc test for PE 18.07 (#404)
• Update PrgEnv-pgi definitions on Kesch (#348)
• New flexible CUDA memtest check (#346)

Other

• Add testing for python3.7 in travis (#356)
• Remove monch from the machinesList (#436)
• New programming environments for Kesch (#432)
• Temporarily disable running of checks on Daint (#406)
• Check added/modified regression tests w.r.t master (#353)
• Update documentation for the build systems feature (#433)

ReFrame 2.13

ReFrame 2.13 Release Notes

API Changes

• Enhanced syntax of regression tests using decorators that eliminate the need of boilerplate code (#264, #294)
• Support for specifying compatible ReFrame versions for tests (#287)
• Enhance behavior of pre_run and post_run attributes, allowing to specify multiple job steps (#333)

Framework

• Redesign of performance logging (#213)
  • Performance logging can now be configured from the configuration file
  • Two different backends: file logging, Graylog
• More logging attributes added for formatting log messages (#213, #337)
• Support for PBS job scheduler (#316)
• Support for TMod4 (#282)
• Include name of the regression test file to the list of checks (#288)
• Better filtering of tests (#319)
• Add a knob for controlling the 'sacct'/'squeue' ratio in Slurm 'sacct' backend (#289)
• Migrate to pytest (#231)
• Bug fixes: #293, #305, #307, #308, #312, #318, #323, #325

Regression tests

• New regression tests and enhancements: #281, #284, #286, #310, #311
• Bug fixes: #309, #330, #331, #332

ReFrame 2.12

ReFrame 2.12 Release Notes

API Changes

• post_run and pre_run attributes removed from the Job class
• Support for wildcards in valid_prog_environs

Framework

• Support for automatic retry of failing tests (new --max-retries option)
• Support for custom configuration files (new -C option)
• Poll rate of the asynchronous execution policy is now lowered
• Unique test names are now enforced
• Environment variables are expanded in the -c option
• Documentation fixes and updates
• Internal abstraction layer for treating ReFrame versions
• Bug fixes: #192, #207, #221, #266, #274, #279

Regression tests

• New regression test for TensorFlow
• New flexible MPI Hello World test
• Add numerical sanity checks for Amber, CP2K, CPMD, GROMACS, LAMMPS, NAMD
• Corrected use of scoped dict key updates in the multi-GPU CUDA test
• Bug fixes: #242

ReFrame 2.11

ReFrame 2.11 Release Notes

API Changes

None

Framework

• New feature: support for module mappings
• Submit jobs from the stage directory
• Allow thresholds >= 1.0 in assert_reference() sanity function
• Flexible allocation take into account the --exclude-nodes option
• Add prefix rfm_ to the job names of ReFrame tests
• Fix crash when an unknown modules system is set in the configuration
• Fix and update documentation on the check_info logging attribute
• Fix math domain error in the polling function of the asynchronous execution policy
• Fix failure of the asynchronous execution policy's unit tests
• Fix confusing error messages in serial execution policy
• Fix emission of Slurm's --exclusive flag
• Fix logging configuration in generic settings

Regression tests

• Add regression tests for GPU direct
• Add a G2G meteoswiss regression test
• Add ICON RRTMGP regression test
• Add numerical sanity check in quantum espresso test
• Enable the switches resource certain tests
• Fix deprecation warnings in regression tests

ReFrame 2.10

ReFrame 2.10 Release Notes

API Changes

• RegressionTest.sourcesdir is now always copied to the stage directory
• Support for specifying Git repositories in RegressionTest.sourcesdir
• Job.pre_run and Job.post_run fields are deprecated in favour of the new ones RegressionTest.pre_run and RegressionTest.post_run

Framework

• Support for the LMod modules system
• Support for module aliases by the internal modules system abstraction
• New more robust design of the regression test execution policies
• More robust and efficient error checking for the Tmod modules system backend
• Fix framework behaviour when trying to run interactive commands
• Fir random unit tests failures with Python 3.5
• Documentation fixes

ReFrame 2.9

ReFrame 2.9 Release Notes

API Changes

• Old sanity and performance patterns syntax completely removed
• New extended syntax of extra_resources
• Allow sourcesdir to be None even for compiled regression tests
• More advanced examples in documentation

Framework

• Redesign of the exception hierarchy
• Version information moved away from settings
• Support for flexible job allocations that span a full reservation
• Support for easy deprecation of fields
• Fix ungraceful exit when a job is blocked in in unrecoverable state
• Fix regex group references to be compatible with Python 3.5
• Fix random failures in unit tests
• Fix behaviour of --partition option to override any similar option passed in site configuration