# 2022 Regression Test Timeouts **URL:** \ **Category:** User discussions\ **Tags:** mdrun, installation-error\ **Created:** [February 23, 2022, 3:48am UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731 "2022-02-23T03:48:05Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![mwd](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mwd](https://gromacs.bioexcel.eu/u/mwd)\ **Post date:** [February 23, 2022, 3:48am UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/1 "2022-02-23T03:48:05Z")
GROMACS version: 2022 GROMACS modification: No I’m troubleshooting repeated regression test failures using the new 2022 release. This is a CPU-only Zen2 system with 128 cores. The failing tests are… ```auto 65:MdrunIOTests 69:MdrunNonIntegratorTests 83:MdrunFEPTests ``` This is my compiling procedure: ```auto cmake -DCMAKE_INSTALL_PREFIX="$PREFIX" \ -DGMX_FFT_LIBRARY=fftw3 -DGMX_BUILD_OWN_FFTW=ON -DGMX_HWLOC=ON -DGMX_MPI=ON \ -DGMX_SIMD=AVX2_128 -DGMX_BUILD_OWN_FFTW_URL="$DISTDIR/fftw-3.3.8.tar.gz" \ -DREGRESSIONTEST_PATH="$WORKDIR/regressiontests-2022" \ -DGMX_EXTERNAL_BLAS=ON -DGMX_EXTERNAL_LAPACK=ON .. make -j128 make VERBOSE=1 check ``` I’ve also tried `AVX2_256` without any significant change. No other significant processes are competing for resources on this machine. I’ve included my LastTest.log. Can anyone explain these timeouts? Thanks in advance! [LastTest.log](https://gromacs.bioexcel.eu/uploads/short-url/8HYP1F3ySmi0HZ8cRm2xvwkuIoW.log) (4.0 MB) UPDATE: I can also confirm that compiling Gromacs without external MPI support produces a build which completes the regression tests successfully. ---
**Author:** ![pbauer](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pbauer/32/13_2.png) [@pbauer](https://gromacs.bioexcel.eu/u/pbauer)\ **Post date:** [February 23, 2022, 11:07am UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/2 "2022-02-23T11:07:34Z")
Hello, this is likely an issue with the tests trying to run on all of the 128 tests, and being extremely slowed down by this. Please try running with a lower number of OMP threads. Cheers Paul ---
**Author:** ![mwd](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mwd](https://gromacs.bioexcel.eu/u/mwd)\ **Post date:** [February 23, 2022, 8:40pm UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/3 "2022-02-23T20:40:09Z")
I gave this a shot: ```auto export OMP_NUM_THREADS=16 make VERBOSE=1 check ``` but it caused a ton of failures with this repeated message: ```auto Environment variable OMP_NUM_THREADS (16) and the number of threads requested on the command line (2) have different values. Either omit one, or set them both to the same value. ``` When I used our job scheduler to allocate 16 only cores, I got the same failures as before: ```auto The following tests FAILED: 65 - MdrunIOTests (Timeout) 69 - MdrunNonIntegratorTests (Timeout) 83 - MdrunFEPTests (Timeout) ``` Is 16 threads still too many? ---
**Author:** ![mwd](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mwd](https://gromacs.bioexcel.eu/u/mwd)\ **Post date:** [February 28, 2022, 8:29pm UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/4 "2022-02-28T20:29:38Z")
This is still an issue, impeding the deployment of an updated Gromacs build on our system. ---
**Author:** ![pbauer](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pbauer/32/13_2.png) [@pbauer](https://gromacs.bioexcel.eu/u/pbauer)\ **Post date:** [March 1, 2022, 9:18am UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/5 "2022-03-01T09:18:36Z")
I tried reproducing this locally (but only on 16 cores), and the tests don’t time out when building the same way as you, but there are a few NB-LIB test failures instead. Going through the log it looks like the task setting is all done correctly, tests are running on the number of ranks that they should, they just seem to take a long time to do so. One issue I found in the attached log is this, but you already said that this shouldn’t be an issue. ```auto Highest SIMD level supported by all nodes in run: AVX2_256 SIMD instructions selected at compile time: AVX2_128 Compiled SIMD newer than supported; program might crash ``` ---
**Author:** ![mwd](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mwd](https://gromacs.bioexcel.eu/u/mwd)\ **Post date:** [March 1, 2022, 3:13pm UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/6 "2022-03-01T15:13:21Z")
I did the same, using a Ryzen 2700X processor. My results were similar to yours, and did not match the results on the HPC system. Standard GROMACS runs provide a detailed explanation of where time is being spent. Is such an explanation available for the test cases? ---
**Author:** ![pbauer](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pbauer/32/13_2.png) [@pbauer](https://gromacs.bioexcel.eu/u/pbauer)\ **Post date:** [March 2, 2022, 8:18am UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/7 "2022-03-02T08:18:23Z")
The unit tests don’t write this out by default, sorry. ---
**Author:** ![swong](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@swong](https://gromacs.bioexcel.eu/u/swong)\ **Post date:** [June 19, 2023, 11:05pm UTC](https://gromacs.bioexcel.eu/t/2022-regression-test-timeouts/3731/8 "2023-06-19T23:05:48Z")
I did the following and it got helped: export OMP\_NUM\_THREADS=1,2,4,6,8