GROMACS 2021.2: `make check` fails at `NbLibListedForcesTests`

ostueker · May 11, 2021, 8:39pm

GROMACS version: 2021.2 & 2021.1
GROMACS modification: No

Dear GROMACS team,

I have been installing GROMACS for the past 5 years or so for Compute Canada’s HPC systems.

I’m currently trying to build GROMACS 2021.2 with the following toolchain:

gcc/9.3.0
openmpi/4.0.3
imkl/2020.1.217

The building works well, but during make check some of the NbLibListedForcesTests are failing.

The commands that are used during the build are:

tar xzf /home/stuekero/.local/easybuild/sources/g/GROMACS/gromacs-2021.2.tar.gz 
cmake \
 -DCMAKE_INSTALL_PREFIX=/home/stuekero/.local/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/gromacs/2021.2 \
 -DCMAKE_BUILD_TYPE=Release \
 -DCMAKE_C_COMPILER='mpicc' \
 -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fopenmp' \
 -DCMAKE_CXX_COMPILER='mpicxx' \
 -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fopenmp' \
 -DCMAKE_Fortran_COMPILER='mpifort' \
 -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fopenmp' \
 -DCMAKE_VERBOSE_MAKEFILE=ON \
 -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  \
 -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  \
 -DENABLE_XHOST=OFF  \
 -DCMAKE_SKIP_INSTALL_RPATH=ON \
 -DGMX_MPI=OFF \
 -DGMX_THREAD_MPI=ON \
 -DGMX_DOUBLE=OFF  \
 -DGMX_GPU=OFF  \
 -DGMXAPI=ON  \
 -DGMX_PREFER_STATIC_LIBS=OFF  \
 -DGMX_EXTERNAL_BLAS=ON \
 -DGMX_EXTERNAL_LAPACK=ON  \
 -DGMX_X11=OFF  \
 -DGMX_SIMD=AVX2_256  \
 -DGMX_OPENMP=ON  \
 -DGMX_FFT_LIBRARY=mkl \
 -DMKL_INCLUDE_DIR="/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/include"   \
 -DMKL_LIBRARIES="-Wl,--start-group;/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/lib/intel64/libmkl_gf_lp64.a;/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/lib/intel64/libmkl_sequential.a;/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/lib/intel64/libmkl_core.a;-Wl,--end-group -lpthread -lm -ldl"   \
 -DBUILD_SHARED_LIBS=ON  /tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/ 
make -j 8   
make check -j 8

The long list of CMake options is because we are using the easybuild to build our software stack.
This is the first of a total of 4 builds within the same module:

-DGMX_MPI=OFF -DGMX_THREAD_MPI=ON -DGMX_DOUBLE=OFF
-DGMX_MPI=ON -DGMX_THREAD_MPI=OFF -DGMX_DOUBLE=OFF
-DGMX_MPI=OFF -DGMX_THREAD_MPI=ON -DGMX_DOUBLE=ON
-DGMX_MPI=ON -DGMX_THREAD_MPI=OFF -DGMX_DOUBLE=ON
Builds with AVX512 and/or GPU (Cuda) will follow.

The output from the failed test is quoted at the bottom.

I get the same failed test (producing the same results) with these combinations:

GROMACS 2021.2: gcc/9.3.0 openmpi/4.0.3 imkl/2020.1.217
GROMACS 2021.2: gcc/9.3.0 openmpi/4.0.3 fftw/3.3.8
GROMACS 2021.1: gcc/9.3.0 openmpi/4.0.3 imkl/2020.1.217
GROMACS 2021.1: gcc/9.3.0 openmpi/4.0.3 fftw/3.3.8
using serial build (make check -j 1) does not make any difference either.

The installation and tests work just fine for GROMACS 2020.4 and 2020.6
using the same gcc/9.3.0 openmpi/4.0.3 imkl/2020.1.217 modules.

My questions:

Is this a known issue with the tests and/or compiler/library versions?
Is there an easy way to skip this specific test NbLibListedForcesTests?

Oliver

Appendix:
The output from the failed test:

/cvmfs/soft.computecanada.ca/gentoo/2020/usr/bin/ctest --output-on-failure -E physicalvalidationtests
Test project /tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/easybuild_obj
      Start  1: NbLibListedForcesTests
 1/67 Test  #1: NbLibListedForcesTests ................***Failed    0.29 sec
[==========] Running 23 tests from 9 test cases.
[----------] Global test environment set-up.
[----------] 5 tests from NBlibTest
[ RUN      ] NBlibTest.BondTypesOperatorEqualWorks
[       OK ] NBlibTest.BondTypesOperatorEqualWorks (0 ms)
[ RUN      ] NBlibTest.BondTypesLessThanWorks
[       OK ] NBlibTest.BondTypesLessThanWorks (0 ms)
[ RUN      ] NBlibTest.CanSplitListedWork
[       OK ] NBlibTest.CanSplitListedWork (0 ms)
[ RUN      ] NBlibTest.ListedForceBuffer
[       OK ] NBlibTest.ListedForceBuffer (0 ms)
[ RUN      ] NBlibTest.ListedForceCalculatorCanConstruct
[       OK ] NBlibTest.ListedForceCalculatorCanConstruct (0 ms)
[----------] 5 tests from NBlibTest (0 ms total)

[----------] 1 test from Kernels
[ RUN      ] Kernels.HarmonicScalarKernelCanCompute
[       OK ] Kernels.HarmonicScalarKernelCanCompute (0 ms)
[----------] 1 test from Kernels (0 ms total)

[----------] 5 tests from ListedExampleData
[ RUN      ] ListedExampleData.ComputeHarmonicBondForces
[       OK ] ListedExampleData.ComputeHarmonicBondForces (0 ms)
[ RUN      ] ListedExampleData.ComputeHarmonicBondEnergies
[       OK ] ListedExampleData.ComputeHarmonicBondEnergies (0 ms)
[ RUN      ] ListedExampleData.ComputeHarmonicAngleForces
[       OK ] ListedExampleData.ComputeHarmonicAngleForces (0 ms)
[ RUN      ] ListedExampleData.CanReduceForces
[       OK ] ListedExampleData.CanReduceForces (0 ms)
[ RUN      ] ListedExampleData.CanReduceEnergies
[       OK ] ListedExampleData.CanReduceEnergies (0 ms)
[----------] 5 tests from ListedExampleData (0 ms total)

[----------] 1 test from LinearChainDataFixture
[ RUN      ] LinearChainDataFixture.Multithreading
[       OK ] LinearChainDataFixture.Multithreading (0 ms)
[----------] 1 test from LinearChainDataFixture (0 ms total)

[----------] 1 test from ListedShims
[ RUN      ] ListedShims.ParameterConversion
[       OK ] ListedShims.ParameterConversion (0 ms)
[----------] 1 test from ListedShims (0 ms total)

[----------] 1 test from ListedTransformations
[ RUN      ] ListedTransformations.SortInteractionIndices
[       OK ] ListedTransformations.SortInteractionIndices (1 ms)
[----------] 1 test from ListedTransformations (1 ms total)

[----------] 3 tests from FourCenter/ProperDihedralTest
[ RUN      ] FourCenter/ProperDihedralTest.CheckListed/0
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[1]/Z
    Actual: 2.9845130443572998
 Reference: -14.707831382751465
Difference: 17.6923 (2175423882 single-prec. ULPs, rel. 1.2), signs differ
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[2]/Z
    Actual: -5.9854903221130371
 Reference: 29.496799468994141
Difference: 35.4823 (2192278165 single-prec. ULPs, rel. 1.2), signs differ
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[3]/Z
    Actual: 3.0009772777557373
 Reference: -14.788968086242676
Difference: 17.7899 (2175578016 single-prec. ULPs, rel. 1.2), signs differ
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
[  FAILED  ] FourCenter/ProperDihedralTest.CheckListed/0, where GetParam() = ({ 12-byte object <7F-92 EA-BF 00-00 70-41 02-00 00-00> }, { 12-byte object <00-00 00-00 00-00 00-00 00-00 00-00>, 12-byte object <00-00 00-00 00-00 00-00 CD-CC 4C-3E>, 12-byte object <0A-D7 A3-3B 00-00 00-00 CD-CC CC-3D>, 12-byte object <6F-12 83-BA CD-CC CC-3D 00-00 00-00> }) (265 ms)
[ RUN      ] FourCenter/ProperDihedralTest.CheckListed/1
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[1]/Z
    Actual: -2.2539417743682861
 Reference: 0
Difference: 2.25394 (1074806933 single-prec. ULPs, rel. Inf), signs differ
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[2]/Z
    Actual: 5.1225948333740234
 Reference: 0
Difference: 5.12259 (1084484684 single-prec. ULPs, rel. Inf)
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[3]/Z
    Actual: -2.8686530590057373
 Reference: 0
Difference: 2.86865 (1077385219 single-prec. ULPs, rel. Inf), signs differ
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
[  FAILED  ] FourCenter/ProperDihedralTest.CheckListed/1, where GetParam() = ({ 12-byte object <7F-92 EA-BF 00-00 70-41 02-00 00-00> }, { 12-byte object <00-00 00-3F 00-00 00-00 00-00 00-00>, 12-byte object <00-00 00-3F 00-00 00-00 9A-99 19-3E>, 12-byte object <00-00 00-3F 29-5C 8F-3D AE-47 61-3E>, 12-byte object <00-00 00-3F EC-51 38-3E AE-47 61-3E> }) (0 ms)
[ RUN      ] FourCenter/ProperDihedralTest.CheckListed/2
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[1]/Z
    Actual: -386.67364501953125
 Reference: -884.99822998046875
Difference: 498.325 (10217897 single-prec. ULPs, rel. 0.563)
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[2]/Z
    Actual: 136.66950988769531
 Reference: 299.41543579101562
Difference: 162.746 (9243080 single-prec. ULPs, rel. 0.544)
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
/tmp/stuekero/avx2/GROMACS/2021.2/gomkl-2020a/gromacs-2021.2/src/testutils/refdata.cpp:934: Failure
   In item: /forces/[3]/Z
    Actual: -3.8129832744598389
 Reference: 331.76568603515625
Difference: 335.579 (2216290797 single-prec. ULPs, rel. 1.01), signs differ
 Tolerance: abs. 1e-12, rel. 1e-12, 1000 ULPs
[  FAILED  ] FourCenter/ProperDihedralTest.CheckListed/2, where GetParam() = ({ 12-byte object <7F-92 EA-BF 00-00 70-41 02-00 00-00> }, { 12-byte object <1E-16 EA-BD B0-03 E7-BC 00-00 00-00>, 12-byte object <00-00 00-00 33-C4 31-3D 00-00 00-00>, 12-byte object <21-B0 F2-3D 65-19 62-BC 00-00 00-00>, 12-byte object <77-BE 9F-BC 2C-65 19-3E 00-00 00-00> }) (0 ms)
[----------] 3 tests from FourCenter/ProperDihedralTest (265 ms total)

[----------] 3 tests from TwoCenter/HarmonicBondTest
[ RUN      ] TwoCenter/HarmonicBondTest.CheckListed/0
[       OK ] TwoCenter/HarmonicBondTest.CheckListed/0 (0 ms)
[ RUN      ] TwoCenter/HarmonicBondTest.CheckListed/1
[       OK ] TwoCenter/HarmonicBondTest.CheckListed/1 (0 ms)
[ RUN      ] TwoCenter/HarmonicBondTest.CheckListed/2
[       OK ] TwoCenter/HarmonicBondTest.CheckListed/2 (0 ms)
[----------] 3 tests from TwoCenter/HarmonicBondTest (0 ms total)

[----------] 3 tests from ThreeCenter/HarmonicAngleTest
[ RUN      ] ThreeCenter/HarmonicAngleTest.CheckListed/0
[       OK ] ThreeCenter/HarmonicAngleTest.CheckListed/0 (0 ms)
[ RUN      ] ThreeCenter/HarmonicAngleTest.CheckListed/1
[       OK ] ThreeCenter/HarmonicAngleTest.CheckListed/1 (1 ms)
[ RUN      ] ThreeCenter/HarmonicAngleTest.CheckListed/2
[       OK ] ThreeCenter/HarmonicAngleTest.CheckListed/2 (0 ms)
[----------] 3 tests from ThreeCenter/HarmonicAngleTest (1 ms total)

[----------] Global test environment tear-down
[==========] 23 tests from 9 test cases ran. (268 ms total)
[  PASSED  ] 20 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] FourCenter/ProperDihedralTest.CheckListed/0, where GetParam() = ({ 12-byte object <7F-92 EA-BF 00-00 70-41 02-00 00-00> }, { 12-byte object <00-00 00-00 00-00 00-00 00-00 00-00>, 12-byte object <00-00 00-00 00-00 00-00 CD-CC 4C-3E>, 12-byte object <0A-D7 A3-3B 00-00 00-00 CD-CC CC-3D>, 12-byte object <6F-12 83-BA CD-CC CC-3D 00-00 00-00> })
[  FAILED  ] FourCenter/ProperDihedralTest.CheckListed/1, where GetParam() = ({ 12-byte object <7F-92 EA-BF 00-00 70-41 02-00 00-00> }, { 12-byte object <00-00 00-3F 00-00 00-00 00-00 00-00>, 12-byte object <00-00 00-3F 00-00 00-00 9A-99 19-3E>, 12-byte object <00-00 00-3F 29-5C 8F-3D AE-47 61-3E>, 12-byte object <00-00 00-3F EC-51 38-3E AE-47 61-3E> })
[  FAILED  ] FourCenter/ProperDihedralTest.CheckListed/2, where GetParam() = ({ 12-byte object <7F-92 EA-BF 00-00 70-41 02-00 00-00> }, { 12-byte object <1E-16 EA-BD B0-03 E7-BC 00-00 00-00>, 12-byte object <00-00 00-00 33-C4 31-3D 00-00 00-00>, 12-byte object <21-B0 F2-3D 65-19 62-BC 00-00 00-00>, 12-byte object <77-BE 9F-BC 2C-65 19-3E 00-00 00-00> })

 3 FAILED TESTS

[...]

99% tests passed, 1 tests failed out of 67

Label Time Summary:
GTest              = 107.36 sec*proc (65 tests)
IntegrationTest    =  15.75 sec*proc (18 tests)
MpiTest            =  77.16 sec*proc (8 tests)
SlowTest           =  90.29 sec*proc (8 tests)
UnitTest           =   1.32 sec*proc (39 tests)

Total Test time (real) = 107.43 sec

The following tests FAILED:
	  1 - NbLibListedForcesTests (Failed)

pbauer · May 12, 2021, 6:43am

Hello,

we added the new listed forces calculation with NB-LIB in the 2021 release (that is not yet used outside of the tests). Pinging @Joe as the main NB-LIB contributor.

If the rest of the tests passed then the installation of GROMACS itself is fine, but we will need to fix the issue in NB-LIB.

Cheers

Paul

ostueker · May 12, 2021, 12:32pm

Thanks, Paul.

Is there an easy way to just skip the NbLibListedForcesTests e.g. by setting an environment variable or supplying an argument to make check ?

As part of our build-process we always run make check (if available for the package) and “EasyBuild” will just stop the installation if that fails. This is actually very good, because I as the installer can be confident that the new version will work as expected for our users (without having to run hours of simulations myself).
I know I could go into the source code and dig until I find the failing tests and write a patch that disables just those, but:
(a) this will take several hours of my time because I’m not that familiar with ctest and
(b) as the patch is applied before running cmake, the GROMACS version that will be installed on the clusters will be branded as MODIFIED for no good reason.

So being able to selectively skip certain tests would be really helpful once I compile GROMACS that has been patched with PLUMED, as there I have noticed MdrunTest.WritesHelp with the 2020.4 release as PLUMED adds new command line options to gmx mdrun which will cause the output to mismatch the expected output. My current workaround is to skip make check entirely for GROMACS-PLUMED.

Oliver