Hi,
I have prepared a Gromacs simulation on the Bioexcel Building Blocks website (Homepage - BioExcel Building Blocks Workflows) for a protein complex. According to the help page (Help - BioExcel Building Blocks Workflows), to run the simulation on a supercomputer the tpr file generated needs to be transferred to the HPC along with a script to run the job. The example script provided is for SLURM:
#!/bin/bash
######################################
# GROMACS MD Execution Script #
# Usage: sbatch FILE_NAME.sh #
######################################
#
#########################################################################
# To use with Slurm Queue Manager #
# #SBATCH --ntasks : Number of processes to start. #
# #SBATCH --cpus-per-task : Jobs CPU n = cpus_per_task * total_tasks. #
# #SBATCH --ntasks-per-node : Number of tasks x node. #
# #SBATCH --nodes : Number of nodes. #
# #SBATCH -t : Job time limit. #
# #SBATCH --qos : Queue name. #
# #SBATCH -J : Job name. #
# #SBATCH -e : Job standard error file. #
# #SBATCH -o : Job standard output file. #
# #SBATCH -D : Working directory. #
#########################################################################
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=48
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=1
#SBATCH -t 0-2:00:00
#SBATCH --qos=debug
#SBATCH -J JOB_NAME
#SBATCH -o job.out
#SBATCH -e job.err
#SBATCH -D .
module load gromacs
# GROMACS production MD
gmx_mpi -nobackup -nocopyright mdrun -s gppmdsim.tpr -o output_prod.trr -c output_prod.gro -e output_prod.edr -g output_prod.log -x output_prod.xtc
However, the HPC that I am working with uses Sun Grid Engine (SGE) rather than SLURM. Therefore, I have tried creating a script for SGE, but I am experiencing some problems:
Script:
# SGE job submission system
#$ -S /bin/bash
##wall time for calculations
#$ -l h_rt=0:10:0
# For 1 GPU
#$ -l gpu=1
##memory required
#$ -l mem=4G
##name of calculation
#$ -N Gromacs_2XQW_test2
##number of cores needed for calculation. Tip: use multiples of 16 as there are 16 cores per node,
##this makes it easier for the queue system to allocate you job to the resources.
#$ -pe mpi 16
##this tells the computer where to run the calculation, cwd stands for current working directory
##meaning that the code will be run in the directory that the submisson script is submitted
#$ -cwd
##these are modules and dependencies that cp2k uses to run calculations
module unload -f compilers mpi
module load compilers/intel/2018/update3
module load mpi/intel/2018/update3/intel
module load libmatheval
module load flex
module load plumed/2.5.2/intel-2018
module load gromacs/2019.3/plumed/intel-2018
# GROMACS production MD
gerun gmx_mpi -nobackup -nocopyright mdrun -s gppmdsim.tpr -o output_prod.trr -c output_prod.gro -e output_prod.edr -g output_prod.log -x output_prod.xtc
Error with Script:
GERun:
GERun: Note: Lines like this one prefixed with "GERun:" are for debugging
GERun: purposes only and you do not need to report them to rc-support
GERun: unless your job fails for other reasons.
GERun:
GERun:
GERun: Grid Engine parallel launcher abstraction layer version iv (public)
GERun: Dr Owain Kenway, RCAS, RITS, ISD, UCL, 7th of February, 2018
GERun:
GERun: For licensing terms, see LICENSE.txt
GERun:
GERun: Using environment: intel
GERun: Running on 16 slots:
GERun: 16 MPI tasks
GERun: 1 threads per task
GERun: TMPDIR=/tmpdir/job/718657.undefined
GERun:
GERun: Contents of machinefile:
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun: node-e00a-016
GERun:
GERun: GErun command being run:
GERun: mpirun --rsh=ssh -machinefile /tmpdir/job/718657.undefined/machines.unique -np 16 -rr gmx_mpi -nobackup -nocopyright mdrun -s gppmdsim.tpr -o output_prod.trr -c output_prod.gro -e
:-) GROMACS - gmx mdrun, 2019.3 (-:
Executable: /shared/ucl/apps/gromacs/2019.3/plumed/intel-2018/bin/gmx_mpi
Data prefix: /shared/ucl/apps/gromacs/2019.3/plumed/intel-2018
Working dir: /lustre/home/ucbtcd1/2xqw_original_10ns_test2
Command line:
gmx_mpi -nobackup -nocopyright mdrun -s gppmdsim.tpr -o output_prod.trr -c output_prod.gro -e
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 0 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 1 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 2 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 3 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 4 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 5 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 6 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 7 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 8 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 9 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 10 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 11 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 12 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 13 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 14 (out of 16)
Error in user input:
Invalid command-line options
-------------------------------------------------------
Program: gmx mdrun, version 2019.3
Source file: src/gromacs/commandline/cmdlineparser.cpp (line 276)
Function: void gmx::CommandLineParser::parse(int *, char **)
MPI rank: 15 (out of 16)
Error in user input:
Invalid command-line options
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 8
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 12
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 14
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 9
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 13
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 15
In command-line option -s
File 'gppmdsim.tpr' does not exist or is not accessible.
The file could not be opened.
Reason: No such file or directory
(call to fopen() returned error code 2)
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 11
/var/opt/sge/node-e00a-016/job_scripts/718657: line 37: output_prod.edr: command not found
/var/opt/sge/node-e00a-016/job_scripts/718657: line 38: output_prod.log: command not found
Therefore, I am not sure what is wrong with the script or how I can resolve this issue. I have checked the documentation for the HPC that I use, and they provide a few different examples for loading the modules of gromacs:
Version 1:
# Example for gromacs/2021.2/gnu-7.3.0
module load beta-modules
module unload -f compilers mpi gcc-libs
module load gcc-libs/7.3.0
module load compilers/gnu/7.3.0
module load mpi/openmpi/3.1.4/gnu-7.3.0
module load python3
module load gromacs/2021.2/gnu-7.3.0
# Run GROMACS - replace with mdrun command line suitable for your job!
gerun mdrun_mpi -v -stepout 10000
Version 2:
# Example for gromacs/2019.3/intel-2018
module unload -f compilers mpi
module load compilers/intel/2018/update3
module load mpi/intel/2018/update3/intel
module load gromacs/2019.3/intel-2018
# Run GROMACS - replace with mdrun command line suitable for your job!
gerun mdrun_mpi -v -stepout 10000
Version 3:
# Plumed example for gromacs/2019.3/plumed/intel-2018
module unload -f compilers mpi
module load compilers/intel/2018/update3
module load mpi/intel/2018/update3/intel
module load libmatheval
module load flex
module load plumed/2.5.2/intel-2018
module load gromacs/2019.3/plumed/intel-2018
# Run GROMACS - replace with mdrun command line suitable for your job!
gerun gmx_mpi -v -stepout 10000
In my case, I used Version 3 for loading the modules because this is the only example which shows gmx_mpi. The other two use mdrun_mpi. Since the example SLURM script from Biobb uses gmx_mpi and mdrun, I figured this would be the best option, but I am not sure. I believe my issue arises from the gerun line, so any suggestions on how to fix this would be highly appreciated.