ReFrame 3.0-dev3

v3.0-dev3

ReFrame 3.0-dev3

ReFrame 3.0-dev2

v3.0-dev2

ReFrame 3.0-dev2

ReFrame 3.0-dev1

v3.0-dev1

ReFrame 3.0-dev1

ReFrame 3.0-dev0

v3.0-dev0

ReFrame 3.0-dev0

ReFrame 2.21

ReFrame v2.21 Release Notes

New features and enhancements

• Print repository's revision hash when ReFrame's version is requested (#1107)
• Make job completion time available as a logging variable (#1090)
• Emit job options to aprun command line (#1067)
• Add nodelist in failure report (#1077)
• Improve error message when no action is specified (#1073)
• Add support for TMod version 3.1 (#935)
• Allow passing arbitrary options to a container platform (#1064)
• Add support for TACC's ibrun parallel job launcher (#1066)
• Add a print() sanity function (#1062)
• Change command line option --flex-alloc-tasks to --flex-alloc-nodes (#1058)
• Rename make_deferrable() to defer() and make it publicly available along with evaluate() (#1050)
• Support for running dependent tests with the asynchronous execution policy (#1022)
• Add support for test dependencies in serial execution policy (#1015)

Bug fixes

• Fix crash of local scheduler with flexible allocation (#1088)
• Fix unit test failure for scheduler backends when run alone (#1076)
• Ignore invalid Slurm nodes in flexible task allocation (#1054)

Regression tests

• Update maintainers list (#1096)
• Add check for Slurm error message when no constraint is given (#1104)
• Adapt HaloExchangeTest/Alltoallv for Kesch (#1105)
• Fix netCDF tests for Kesch (#1103)
• Fix sanity error in jacobi_perftools check (#1102)
• Update AllocSpeed check for Kesch (#1098)
• Remove slurm switch option from dom (#1086)
• Fix CCE flags in HelloWorld tests for Kesch (#1085)
• Add NumPy test (#1021)
• Add strided access bandwidth benchmark (#959)
• Add check for TensorFlow2+Horovod (#1072)
• Fix reference of the tensorflow+horovod test (#1071)
• Update automatic Fortran arrays check (#1045)
• Update compiler flag for Gperftools check (#1040)
• Update compiler version for Mpip check (#1041)
• Update tool version for Intel tools check (#1039)
• Add Tiger as valid system (#1035)
• Add JupyterHub submission test (#1033)
• Fix tags in Gridtools checks (#1034)
• Port Perftools check to CLE7 (#1025)
• Update alloc speed test (#1024)
• Update HPX "hello world" program name to v1.3.0 (#1020)

Other

• Use --flex-alloc-nodes option to run user tests in CI (#1100)
• Fix quotes in CI script (#1036)
• Document the test dependency feature (#1084)
• Stress out that hostnames are Python regexes (#1097)
• Add link to the contribution guide in the README page (#1080)
• Add slides from SC19 (#1079)

ReFrame 2.20

New features and enhancements

• Write the number of tasks in performance report (#976)
• Enable running containers inside the test pipeline (#853)
• Lazy evaluation of job-related attributes and environments (#961)
• Add SSH job launcher (#952)
• Add syntax for running functions after dependency resolution (#951)
• Enable easy testing of non-default Cray programming environments (#949)
• Replace reframe_module configuration option with command line option (#946)
• Improve and simplify the topological sorting of the test cases (#934)
• Add new syntax for attaching arbitrary regression test methods to pipeline stages (#920)
• Print compiler commands and flags when showing the environment configuration (#918)

Bug fixes

• Support inheritance in pipeline hooks (#994)
• Do not use login shell when submitting jobs (#998)
• Fix unit tests leaving stale files behind (#948)
• Use login shell when building executables (#911)

Regression tests

• Fix buffer decode error in MpiP check (#1013)
• Fix nvidia and libsci/mkl resolve tests (#1011)
• Bump PE version in module names (#1007)
• Fix Scorep_mpi_omp check (#1006)
• Minor fixes for CLE7 (#1003)
• Update Intel_VTuneAmplifier (#1004)
• Add PrgEnv-pgi to SPEC-ACCEL test (#999)
• Port GpuBandwidthCheck to Tiger (#1000)
• Remove unnecessary scheduler options from the DGEMM benchmark (#990)
• Port GPUDIRECTACC check to Tiger (#971)
• Port OSU check to Tiger (#981)
• Add mem-per-cpu resources to ALLOC_SPEED check for Tiger (#977)
• Remove cce compiler from SCORE-P check (#958)
• Port DGEMM check to Tiger (#978)
• Port OPENACCCUDACPP to Tiger (#973)
• Port GPUDIRECTCUDA check to Tiger (#972)
• Port CPUTARGET check to Tiger (#984)
• Port SHMEM to Tiger (#983)
• Port FFTW check to Tiger (#982)
• Port KERNEL_LATENCY check to Tiger (#980)
• Port GPUBURN check to Tiger (#979)
• Port collectives halo exchange check to Tiger (#970)
• Port automatic arrays check to Tiger (#968)
• Port NETCDF check to Tiger (#967)
• Remove workaround for NetCDF C++ test with PGI (#987)
• Tag checks that use external resources (#964)
• Add specific tag to checks that test Cray PE (#969)
• Update NVML with source from cudatoolkit (#965)
• Port GPUDirect check to Tiger (#966)
• Update Paraview test (#962)
• Update IPM check with new Cray path to PAPI (PAT_BUILD_PAPI_LIBDIR) (#957)
• Remove tags from nsimd checks (#956)
• Update reference for AutomaticArrays test on Dom (#955)
• Update Gromacs reference value (#950)
• Fix cuda checks on Kesch (#943)
• Adapt module versions to PE19.09 (#945)
• Enable IOR test on Daint (#937)
• Update reference values of G2G [1] check (#939)
• Make the MpiP check generic (#938)
• Add performance report to the Intel Advisor test (#916)
• Enable performance reporting in the flexible alltoall OSU benchmark (#908)
• Add CPU memory latency test (#874)
• Fixing OpenACCFortran checks on Kesch (#928)
• Update GitHub repository in collectives MCH check (#929)
• Fixing I/O checks on Kesch (#925)
• Adapt MCH checks to CrayPE 17.06 on Kesch (RH7.5) (#913)
• Add performance report to Empirical Roofline Tool test (#917)
• Increase iterations for async kernel latency test (#910)
• Add libVC simd test (#899)
• Add nsimd simd test (#900)

Other

• Add config file for additional system (#960)
• Fix tags of modified regression tests to run in ci (#914)
• Run benchmark tests in CI with predefined number of tasks (#912)
• Fix literal include lines in performance logging documentation (#1009)
• Fix syntax of container tutorial example (#1008)
• Document the container platforms feature (#985)
• Update documentation links in README (#989)

ReFrame 2.19

ReFrame 2.19 Release Notes

New features and enhancements

• Print debug info when preload environment fails to load (#906)
• Enable modifications to ReFrame's environment per system basis (#887)
• Add automatically a default * reference for performance checks (#866)
• Remove support for the very old test syntax using _get_checks() (#876)
• Fall back to a generic system configuration if system auto-detection fails (#867)
• Allow self loops in module mappings (#855)
• Implement topological sort of the test cases (#815)
• Limit concurrent build jobs to 1 by default in build systems (#810)

Bug fixes

• Fix wrong error message emitted when a user module is loaded with the -m option (#886)
• Do not ignore sourcepath if build system is explicitly set (#875)
• Change to working directory before merging job array files (#869)
• Make Slurm WLM backends aware of job arrays (#840)
• Fix how Slurm states are mapped internally by the Slurm backend (#845)
• Fix ReFrame hanging when Slurm job gets OOM killed (#835)
• Fix timing in the allocation speed test (#824)
• Fix bug in performance report output (#818)

Regression tests

• Update tensorflow/horovod tests for latest modules (#904)
• Update reference for CP2K cpu check small (#902)
• Update Tensorflow checks for PE19.06 (#896)
• Update IPM tests for PE19.06 (#894)
• Update Intel VTune test (#891)
• Fix CPU sum output label for the MCH OpenACC CUDA MPI CPPSTD test (#889)
• inspector (#890)
• Add workaround for netcdf test with PrgEnv-pgi on Dom for CLE7 (#870)
• Remove RRTMGP from production (#877)
• Update STREAM benchmark test (#850)
• Log maximum GPU temperature in the GPU burn test (#872)
• Update OpenCL test for PrgEnv-pgi (#865)
• Update JacobiNoTool perf numbers (#864)
• Update Boost test for PE19.06 (#863)
• Remove stale comment in IOR check (#857)
• Adapt Haswell FMA check for CCE 9.0.x (#851)
• Adapt OpenMP flags for CCE 9.0.x in Jacobi tools tests (#849)
• Adapt OpenMP flags for CCE 9.0.x in HelloWorld tests (#847)
• Update IOR check (#838)
• Use ldflags to link cublas in CudaSimpleMPICheck (#846)
• Enable dynamic linking tests for Trilinos (#842)
• Add GridTools mini app benchmark (#760)
• Fix DefaultRequestGPUSetsGRES for updated slurm output (#837)
• Fix MemoryOverconsumptionCheck message check (#836)
• Set num_cpus_per_task to the TensorFlow Horovod test (#831)
• Make MCH halo cell exchange generic (#826)
• Make HPCG basic test generic (#830)
• Make DGEMM test generic (#827)
• Make GPU shared memory bandwidth test generic (#829)
• Make GPU burn test generic (#828)
• Update Intel tools version on Dom (#825)
• Add API calls in Intel Advisor Roofline check (#812)
• Make the Jacobi check generic (#819)
• Make HaloCellExchangeTest generic (#817)
• Add Intel Vtune roofline check (#813)
• Add dom:mc as valid system to IntelRooflineAdvisorTest (#814)

Other

• Add setuptools as requirement to reframe (#878)
• Update system config for Kesch (#861)
• Re-enable PGI on GPU/MC partitions on Dom (#862)
• Ignore irrelevant @ comments when triggering the CI (#859)
• Run unit tests for Daint in the CI partition (#858)
• Define machines to run the CI with Jenkins params (#852)
• Reenable PGI on Dom (#841)
• Add missing slides to the documentation page (#879)

ReFrame 2.18

ReFrame 2.18 Release Notes

New features and enhancements

• Print performance variable name in case of performance errors (#791)
• Print task allocation type (standard or flexible) for tests in detailed listing (#786)
• Do not use absolute paths in generated batch scripts (#779)
• Add option to print performance report (#733)
• Remove deprecated --logdir command-line option and the corresponding attribute in the logging configuration (#741)
• Enable environment variable substitution in shell code emitted by build systems (#744)

Bug fixes

• Fix crash when using syslog logging handler with the -v option (#808)
• Add version checks for TMod and TMod4 (#794)
• Fix exception message formatting in reservation node extraction (#793)
• Fix filename order in build error message (#790)
• Do not crash when invalid regular expressions are passed to the test selection options (#757)
• Force units in gperftools report (#759)
• Fix gdb4hpc failure with C++ (#747)
• Fix failure in Intel Advisor Roofline check (#745)
• Fix duplicate module commands emitted while executing test cases (#740)
• Fix typo in Likwid bandwidth benchmark (#735)
• Fix crash in flexible task allocation due to refactoring of SystemPartition (#729)
• Remove unnecessary module from the "Hello, World" tests (#726)
• Fix string error in header file of CUDA memtest (#718)
• Fix deprecation warning in check filter functions unit tests (#723)

Regression tests

• Revise MAGMA library benchmark (#783)
• Add simple FFTW benchmark (#776)
• Make OSU tests check generic (#801)
• Make the STREAM test generic (#799)
• Make alloc_speed check generic (#800)
• Make CUDA kernel latency tests generic (#796)
• Hardcode version in Intel Vtune test to prevent failure when default changes (#785)
• Add Roofline check of the GPP code using NVIDIA nvprof (#778)
• Add ert roofline check (gpu) (#775)
• Add ert roofline check (#773)
• Add sde roofline check (#774)
• Fix ddt+cuda/10.0 check (#772)
• Add CUDA shared memory bandwidth benchmark (#770)
• Adapt TensorFlow+Horovod test to CrayGNU-19.03 (#771)
• Update Tensorflow check to r1.12.0 (#768)
• Add flexible HPCG GPU benchmark test (#762)
• Update Score-P check for PE/19.03 (#767)
• Update IPM check for PE/19.03 (#766)
• Adapt Paraview GPU check to CLE7 (#758)
• Update magma tests (#754)
• Update Jacobi test for CLE7 (#727)
• Fix ldflags for nvprof, cuda_gdb tests (#751)
• Update boost python test to use the latest module (#748)
• Update libsci_acc_symlink tests for PE19.03 (#734)
• Update Nvidia35Resolve test for the new PE (#732)
• Update perftools check for CLE7 (#731)
• Rename for loop variable to fix warnings in kernel latency test (#750)
• Add test for gdb4hpc (#603)
• Enable small scale application tests also on Daint (#738)
• Sort cores list in affinity test before comparing (#739)
• Add Intel Advisor Roofline test (#721)
• Add mpiP parallel test (#720)
• Add gperftools parallel test (#719)

Other

• Require full tag names in genrelnotes.py script (#807)
• Expand Ault configuration (#782)
• Add Ault TDS to the config file (#777)
• Update deployment script (#755)
• Add PyPI badge to README (#752)
• Disable running modified tests on Daint (#749)
• Prepare ReFrame for deployment to PyPI (#743)
• Document the --performance-report command line option (#761)
• Fix reference to exclusive_access in tutorial (#742)
• Ongoing work for supporting containers (#787)
• Ongoing work for supporting test dependencies (#769, #797)

ReFrame 2.17

ReFrame 2.17 Release Notes

New features and enhancements

• Add support for selecting tests using regular expressions (#664)
• Add logging handler for sending records to syslog (#675)
• Remove support for passing options to the RegressionTest.compile() method (#697)
• Support command substitution in variable expansion (#683)
• Drop support for setting flags in environment (#678)
• Extend Job API for retrieving the list of nodes assigned to the regression test (#660)
• Add support for directories in keep_files attribute (#673)

Bug fixes

• Define the set of run environments correctly when -p option is used with regular expressions (#716)
• Fix Slurm backend failures when trying node lists with inexistent nodes (#713)
• Fix unit test failure with EBUSY when running on NFS filesystem (#715)
• Fix error message in case of performance errors (#704)
• Fix deprecation warnings with Python 3.7 (#711)
• Correct the units of the kernel latency test (#703)
• Fix flexible task allocation bug ignoring Slurm partition defined in access (#693)
• Use the default Slurm partition with --flex-alloc-tasks if none is specified (#676)
• Fix crash in failure report when test is killed due to timeout (#680)
• Fix bug in 'keep_files' unit tests (#672)

Regression tests

• Add OSU Allreduce test (#686)
• Add Intel Vtune test (#714)
• Add description to the collective halo tests (#690)
• Add Intel Inspector test (#710)
• Add Fulen as a supported system in CSCS config (#684)
• Add MPI halo cell exchange benchmark (#666)
• Enlarge buffer sizes for H2D and D2D multi GPU tests (#689)
• Extend DGEMM test to build with OpenBLAS (#691)
• Add test for COSMO post-processing with Fieldextra (#685)
• Change resources dir for GROMACS test (#695)
• Fix tags of flexible tests + other improvements (#681)
• Add test for gperftools (#670)
• Add CPU affinity test (#657)
• Add test for NERSC IPM (#631)
• Update maintenance tests (#679)
• Extend DGEMM test for using MKL (#662)

Other

• Reenable Dom in the Jenkinsfile (#709)
• Disable Dom from running in the ci (#696)
• Add documentation for regular expression support in test selection options (#717)
• Move documentation to readthedocs.org (#705)
• Remove link to prior to 2.6.1 documentation (#687)

ReFrame 2.16

ReFrame 2.16 Release Notes

New features and enhancements

• Add support for setting a minimum number of required tasks in tests with flexible task allocation (#652)
• Add command line option for increasing the verbosity level of output (#650)
• Add command line options for showing the system and environments configuration (#633)
• Print error information when a modules system operation fails (#628)
• Add support for units in performance references (#576)
• Add a new count_uniq() sanity function (#584)

Bug fixes

• Improve handling of bad tests to avoid frontend crashing (#623)
• Fix bug in variable names in Slurm backend (#639)
• Do not crash if Slurm node does not contain partition information (#621)
• Check if 'UnavailableNodes' are down to cancel job (#599)
• Fix deadlock in the flexible task allocation when the node list output is too large (#614)

Regression tests

• Port remaining tests to the new syntax (#656)
• Add Likwid benchmarks for cache, memory and cross-socket bandwidths (#647)
• Add new MPI_Init_thread() check (#620)
• Add benchmark measuring CUDA kernel launch latency (#646)
• Add HPX 1D stencil tests (#617)
• Add tests for SPEC-ACCEL benchmarks (#618)
• Add test for distributed training with TensorFlow and Horovod (#637)
• Add flexible HPCG test (#604)
• Add HPX hello world test (#616)
• Add allocation speed check (#597)
• Fail if test is not compiled with OpenACC (#615)
• Convert GpuBandwidthCheck to a flexible diagnostic check (#586)
• Testing the environment of the login nodes when accessed remotely over SSH (#583)
• Promote DGEMM to a multi-node check (#530)
• Enable NCO and CDO for the C2SM-RCM environments (#624)
• Make Alltoallv tests non-strict (#622)
• Add specific tag for MCH tests (#594)
• Remove ifdef USE_MPI from the MCH OpenACC/CUDA/MPI/C++ test (#585)
• Set exclusive_access for Kesch tests (#591)
• Adapt MCH regression tests to C2SM-RCM environments (#513)
• Adapt CP2K test (#596)

Other

• Make .pep8speaks.yml consistent with coding style (#649)
• Add option to disable CSCS CI tests (#648)
• Enable pep8speaks for code style checking in the repository (#626)
• Use integer comparison for ReFrame versions in deployment script (#600)
• Replace python 3.7-dev with 3.7 in .travis.yml (#588)
• Document verbosity option (#665)
• Add link for the automatic invites to Slack (#661)
• Update README with contact info (#642)
• Move SC18 presentation & demo to docs (#606)