Error installing gromacs-2022 (C++ One Definition Rule)

User discussions
1

GROMACS version: 2022
GROMACS modification: No
Hi all, I’m trying to install gromacs-2022 (2022.3 didn’t work, waited for 2022.4 and got the same problem) but “make” fails with the following error:

[ 92%] Linking CXX shared library ../../lib/libgromacs_mpi.so
/home/user/Downloads/gromacs-2022.4/src/gromacs/mdlib/lincs.cpp:90:8: warning: type ‘struct AtomPair’ violates the C++ One Definition Rule [-Wodr]
   90 | struct AtomPair
      |        ^
/home/user/Downloads/gromacs-2022.4/src/gromacs/mdlib/lincs_gpu.h:61: note: a different type is defined in another translation unit
   61 | struct AtomPair
      | 
/home/user/Downloads/gromacs-2022.4/src/gromacs/mdlib/lincs.cpp:96:9: note: the first difference of corresponding definitions is field ‘index1’
   96 |     int index1;
      |         ^
/home/user/Downloads/gromacs-2022.4/src/gromacs/mdlib/lincs_gpu.h:64: note: a field with different name is defined in another translation unit
   64 |     int i;
      | 
/tmp/cclqhlb4.s: Assembler messages:
/tmp/cclqhlb4.s:603: Error: symbol `fatbinData' is already defined
...
lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [src/gromacs/CMakeFiles/libgromacs.dir/build.make:13618: lib/libgromacs_mpi.so.7.0.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:4722: src/gromacs/CMakeFiles/libgromacs.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

I’ve configured with CUDA 11.8, gcc-10 on Ubuntu 22.04. Curiously gromacs-2021.6 was installed without problems (well, at first I tried with gcc-11 and installation failed, after changing to gcc-10 it worked). I’ve also tried with gcc-9, with and without MPI, with and without external blas/lapack. Also, if I set -DGMX_BUILD_OWN_FFTW=ON I see the warning below:

src/external/build-fftw/fftwBuild-prefix/src/fftwBuild/tools/fftw-wisdom.c:303:33: warning: ‘%s’ directive output may be truncated writing up to 1535 bytes into a region of size 64 [-Wformat-truncation=]
  303 |     snprintf(ps, sizeof(ps), "%s%s", types[j], canonical_sizes[i]);
      |                                 ^~

I´d appreciate if you could point me a solution or explanation of what is happening.

2

Hi Diniz!

I could not reproduce the issue (Ubuntu 22.04 with CUDA 11.8; GCC 10.4.0 and other dependencies from standard repos). Could you please share how you called CMake and the full version of the relevant software? Are you, by any chance, enabling LTO (link-time optimization) in your build?

The ODR warning you’re observing seems valid, but I don’t think it’s related to fatbinData is already defined.

Could you please check whether the problem still persists in Files · aa-fix-lincs-odr-violation · GROMACS / GROMACS · GitLab? There’s a link to download the source code archive near the big blue “Clone” button.

3

Hi Andrey,
thank you for your reply and for providing the patched code for testing. I’ve tried it with gcc versions 10 and 11 and although the warnings regarding C++ One Definition Rule disappeared, the problem with LTO persists:

[ 92%] Linking CXX shared library ../../lib/libgromacs_mpi.so
/tmp/ccTY3IHD.s: Assembler messages:
/tmp/ccTY3IHD.s:603: Error: symbol `fatbinData' is already defined
...
/tmp/ccTY3IHD.s:2106455: Error: symbol `fatbinData' is already defined
lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [src/gromacs/CMakeFiles/libgromacs.dir/build.make:13186: lib/libgromacs_mpi.so.7.0.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:4722: src/gromacs/CMakeFiles/libgromacs.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

here’s the command line used:

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs-2022.4aa -DGMX_GPU=CUDA -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DGMX_FFT_LIBRARY=fftw3 -DGMX_HWLOC=ON -DGMX_USE_LMFIT=internal -DGMX_MPI=on -DMPIEXEC=mpirun -DMPIEXEC_NUMPROC_FLAG=-n -DMPIEXEC_PREFLAGS= -DMPIEXEC_POSTFLAGS= -DLAPACK_LIBRARIES=/usr/local/lib/lapack/liblapack.so

(previously, not with the patched code, I’ve also tried building fftw and turning external blas and lapack off, and also without MPI, which didn’t help)
The machine I’m using has Ubuntu 22.04.1 without third party software enabled, then cuda-11.8.89 was installed from .deb file, followed by fftw-3.3.10, lapack-3.10.1, BLAS-3.10.0 and mpich-4.0.
I’ve also installed plumed-2.8.0 and I noticed in config.log from fftw and plumed the following related to LTO (but I didn’t take any action for that):

--with-b--with-build-config=bootstrap-lto-lean
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapperuild-config=bootstrap-lto-lean

and in MPICH I found this:

g++  -g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -O2

gcc versions installed are 11.3.0 (original), 10.4.0 and 9.5.0, and these were set (to gcc-10 or gcc-9, together with g++) with update-alternatives before running cmake, followed by make -j 12.
I’m starting to think I would have to reinstall fftw, mpich and perhaps cuda using gcc-10, but that wouldn’t explain why gromacs-2021.6 worked and 2022 doesn’t…

4

So, the same error happens without MPICH, right? Then, I think, rebuilding MPICH won’t be helpful.

Can you try adding -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF to your CMake call?

If that does not help, could you please run VERBOSE=1 make and share the output around the errors (or just upload a full file here or to pastebin).

5

Hi Andrey,
after using -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF I got the folowing message:

[ 10%] Linking CXX shared library ../../lib/libgromacs_mpi.so
lto1: fatal error: bytecode stream in file ‘CMakeFiles/libgromacs.dir/utility/alignedallocator.cpp.o’ generated with LTO version 11.3 instead of the expected 9.4
compilation terminated.
lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [src/gromacs/CMakeFiles/libgromacs.dir/build.make:29435: lib/libgromacs_mpi.so.7.0.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:4722: src/gromacs/CMakeFiles/libgromacs.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

this happened with both gcc-10 and gcc-9 set. In the beginning cmake identifies the C and CXX compilers as set with update-alternatives, is it possible that it is using another version afterwards? Or do these LTO versions have nothing to do with compiler versions?

6

Could you please check what happens without MPI? It could be that MPICH’s mpicxx wrapper enforces the use of a different compiler version. I’d suggest trying a very basic configure command cmake .. -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10 -DGMX_GPU=CUDA in a clean build tree, and then adding other options (clearing the build tree between attempts). It is a bit tedious, but, unfortunately, such issues, caused by a combination of different packages, are hard to diagnose.

7

Hi Andrey,
I tried it without MPI and got the same error, so I decided to also try without CUDA and it actually worked! Even with MPI ON, leaving CUDA out also made the One Definition Rule warnings (together with fatbinData errors) go away. Well, as I would like the GPU support, I guess I’ll have to reinstall CUDA while keeping gcc-10 set. I still don’t understand why gromacs-2021.6 was installed correctly on this same machine. Also, I’ll probably have to reinstall gromacs-2021 if I change cuda, right?

8

LTO is not a separate compiler version, just a step in a normal compilation process: usually, each file with the source code is compiled independently, and then all resulting assembly code is just bundled together; LTO is an additional optimization step during this “bundling”. lto-wrapper is a part of GCC, which is called internally if certain compiler options (e.g., -flto) are set. For some reason, these options are always enabled on your machine, which is not typical.

Thanks for checking. That is expected. fatbinData is an internal CUDA thing (how the GPU kernel is stored in the compiled binary), and the ODR is violated by one of the GPU-specific headers. So, it does not mean that the LTO is now disabled, unfortunately.

At the risk of breaking things completely, could you try reinstalling it? Perhaps there was some configuration change to your machine between the installation of 2021.6 and 2022.3?

One more thing to check: could you run env | grep lto to see if perhaps there are some environment variables influencing the default compiler flags?

There is certain compatibility between CUDA versions so that the old version might work. If there are any problems, they would be quite obvious: gmx not starting at all or unable to detect the GPU.

9

Hi Andrey,
thanks for the explanation about LTO. In this case, that message I got before mentioned “LTO 9.4 was expected”, so does it mean it was somehow bound to gcc-9 ?
I checked the environment variables with env and there is nothing with LTO. However, after running

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs-2022.4 -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10 -DGMX_MPI=OFF -DGMX_GPU=CUDA

I see this in CMakeCache.txt (notice DGMX_MPI is OFF):

MPI_CXX_COMPILE_OPTIONS:STRING=-flto=auto;-ffat-lto-objects;-flto=auto
MPI_C_COMPILE_OPTIONS:STRING=-flto=auto;-ffat-lto-objects;-flto=auto

the same is seen without CUDA.
The CMakeCache.txt file in gromacs-2021.6 shows this:

//Test HAS_FLTO
HAS_FLTO:INTERNAL=1
PYBIND11_LTO_CXX_FLAGS:INTERNAL=-flto;-fno-fat-lto-objects
PYBIND11_LTO_LINKER_FLAGS:INTERNAL=-flto

Could it be that if I edit the CMakeCache.txt in gromacs-2022.4 and remove those lines it would work, even if I want to keep MPI? (I mean, is it ok to do that?)

10

Ok, I think we found it. Could you check that running mpicxx -showme:compile produces the same set of flags?

GROMACS does some basic MPI detection even if MPI is not used. I don’t think we considered a situation like yours, where MPI flags can be problematic, so we always apply them.

I’d suggest running cmake .. -DMPI_CXX_COMPILE_OPTIONS="" after the initial cmake call.

Manually editing CMakeCache is a poor practice, but should work.

That should not be a problem. These flags are detected, but are not used as far as I can tell.

11

I tried to check the flags but mpicxx actually called g++ and didn’t accept that option:

mpicxx -showme:compile
g++: error: unrecognized command-line option ‘-showme:compile’

However, running

cmake .. -DMPI_CXX_COMPILE_OPTIONS=""

right after the initial cmake did work, and I could install gromacs-2022.4 without any further changes. Thanks a lot for your time and support!

12

Sorry, that command was for OpenMPI. You’re using MPICH, so that should be mpicxx -compile-info.

Great that it now works!

If you want to build with MPI, you can use the same trick: MPI_CXX_COMPILE_OPTIONS only has LTO-related things, so they can be removed without harming the MPI functionality.

It would probably be “cleaner” to rebuild MPICH so it does not enforce LTO on its targets. I did not have such a problem myself, but others had (e.g., https://bugzilla.redhat.com/show_bug.cgi?id=1821728). Presumably, it would help if you recompiled MPICH without LTO; then, it will behave nicely.

In the mean-time, I opened a bug report, so hopefully, we will avoid such issues in future versions.

13

Here’s the output from mpicxx -compile-info:

g++ -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -I/usr/include/x86_64-linux-gnu/mpich -L/usr/lib/x86_64-linux-gnu -lmpichcxx -lmpich

actually these flags weren’t set by me, I just used “apt-get install mpich”. I could also install gromacs with mpi using the same trick, but I’ll try to have a look on rebuilding mpich without those options, or perhaps using OpenMP instead.
Thanks again!

14

Did you mean OpenMPI? GROMACS uses OpenMP too, but, in normal circumstances, it is always enabled, regardless of whether you use “library” MPI (like OpenMPI or MPICH), “threadMPI” (our internal MPI, supporting only single node runs), or no MPI at all.

Indeed, installing OpenMPI can probably be easier, unless you know you need MPICH. As far as I can tell, OpenMPI does not have this habit of using any extra unnecessary flags.

15

Sorry, I meant to write OpenMPI indeed :-)