Gromacs (GROningen MAchine for Chemical Simulations, http://www.gromacs.org/About_Gromacs) is a well-known package for molecular dynamics simulations. It is available under the LGPL license, and therefore can be freely used.
Details about the software, its capabilities and performance can be found at the official website ( http://www.gromacs.org).
The program can be run on a variety of machines, from desktops to large clusters, is parallelized using MPI and OpenMP, and also supports GPGPUs.
Different versions of gromacs are installed as modules. To get a list of the available gromacs module just type module avail gromacs. The output will be something like
---------------------- /apps/dcc/etc/ModulesSL73/modulefiles --------------------- gromacs/2018.2 gromacs/2018.5-plumed-2.5 gromacs/2019.4-gpu gromacs/2018.2-gpu gromacs/2018.8-chap-0.9.1 gromacs/2019.4-gpu-plumed-2.6 gromacs/2018.5 gromacs/2018.8-gpu-chap-0.9.1 gromacs/2020-gpu gromacs/2018.5-gpu gromacs/2019.4 gromacs/2020.0
Notice that there are different modules for cpu and GPU. Load the desired version before using it. The version loaded is automatically the one suitable for the cpu-model of the machine.
Gromacs is run from the command line, but you need to specify the right command. Given that gromacs can be run with many different options, and that it can use a combination of MPI, multithreading and also GPU, it can be difficult to figure out the optimal configuration.
The optimal computational configuration (“Is it better to use MPI or OpenMP threads? Should GPUs be used? How many MPI processes, how many OpenMP threads? How many GPus?”) depends on the specific model and the specific simulation settings.
In the following we provide a few templates for cpu and GPU jobscripts as reference.
Do not simply assume that using more resources improves the performance.
Gromacs tries to help by providing suggestions in the log file.
It is important that you make some performance tests with different number of cores, threads, threadMPI, GPUs before choosing a specific configuration.
Single node cpu-only job script
The following script can be used to run a gromacs job on cpu-only, on a single node.
### General options
### -- select the queue --
#BSUB -q steno ### -- set the job Name -- #BSUB -J Gromacs_cpu_job ### -- ask for number of cores (default: 1) -- #BSUB -n 6 ### -- specify that the cores MUST BE on a single host! It's a SMP job! -- #BSUB -R "span[hosts=1]" ### -- set walltime limit: hh:mm -- #BSUB -W 16:00 ### -- specify that we need 2GB of memory per core/slot -- #BSUB -R "rusage[mem=2GB]" ### -- set the email address -- # please uncomment the following line and put in your e-mail address, # if you want to receive e-mail notifications on a non-default address ##BSUB -u your_email_address ### -- send notification at start -- #BSUB -B ### -- send notification at completion-- #BSUB -N ### -- Specify the output and error file. %J is the job-id -- ### -- -o and -e mean append, -oo and -eo mean overwrite -- #BSUB -o Output_%J.out #BSUB -e Error_%J.err #Load a suitable cpu-only gromacs module module load gromacs/2018.5 # MPI only gromacs run export OMP_NUM_THREADS=1 gmx mdrun -deffnm md_0_1 -ntmpi $LSB_DJOB_NUMPROC
LSB_DJOB_NUMPROC is an environment variable that is set automatically to the number of cores requested in the line #BSUB -n 6, 6 in this case. The line
gmx mdrun -deffnm md_0_1 -ntmpi $LSB_DJOB_NUMPROC
will start gromacs using $LSB_DJOB_NUMPROC thread-MPI processes (flag -ntmpi), one per each of the cores requested. No OpenMP multithreading is used, because the number of OpenMP threads is set to 1 in the line
export OMP_NUM_THREADS=1
To run gromacs with 1 MPI process only, and $LSB_DJOB_NUMPROC OpenMP threads, replace the last two lines with the following:
# OpenMP only gromacs run export OMP_NUM_THREADS=$LSB_DJOB_NUMPROC gmx mdrun -deffnm md_0_1 -ntmpi 1
In the first line the number of OpenMP threads is set to the total number of cores requested, and the second line start gromacs with a single threadMPI process -ntmpi 1. Finally, another option is to use both treadMPI processes (flag -ntmpi) and OpenMP threads (flag -ntomp):
# MPI and OpenMP gromacs run export OMP_NUM_THREADS=N gmx mdrun -deffnm md_0_1 -ntmpi M -ntomp N
where you need to replace M and N with suitable numbers. The previous run will start M threadMPI processes, and each of them will start N OpenMP threads. Therefore, to get optimal performance the following equality must be satisfied: M*N = $LSB_DJOB_NUMPROC
Single node GPU job script
The following template can be used to run gromacs on the cluster on a GPU node.
### General options ### -- select the queue -- #BSUB -q stenogpu ### -- set the job Name -- #BSUB -J Gromacs_gpu_job ### -- ask for number of cores (default: 1) -- #BSUB -n 6 ### -- specify that the cores MUST BE on a single host! It's a SMP job! -- #BSUB -R "span[hosts=1]" ### -- Select the resources: 1 gpu in exclusive process mode -- #BSUB -gpu "num=1:mode=exclusive_process" ### -- set walltime limit: hh:mm -- #BSUB -W 16:00 ### -- specify that we need 2GB of memory per core/slot -- #BSUB -R "rusage[mem=2GB]" ### -- set the email address -- # please uncomment the following line and put in your e-mail address, # if you want to receive e-mail notifications on a non-default address ##BSUB -u your_email_address ### -- send notification at start -- #BSUB -B ### -- send notification at completion-- #BSUB -N ### -- Specify the output and error file. %J is the job-id -- ### -- -o and -e mean append, -oo and -eo mean overwrite -- #BSUB -o Output_%J.out #BSUB -e Error_%J.err #Load a suitable GPU gromacs module module load gromacs/2018.5-gpu # set NGPUS to the number of GPUs requested in the job NGPUS=1 # set OMPTHREADS to the number of OpenMP threads desired OMPTHREADS=6 #set the OMP_NUM_THREADS environment variable export OMP_NUM_THREADS=$OMPTHREADS # like this it works #start gromacs, 1 threadMPI per GPU gmx mdrun -deffnm md_0_1 -ntmpi $NGPUS -ntomp $OMPTHREADS
In the script, the line
#BSUB -gpu "num=1:mode=exclusive_process"
requests one GPU card in exclusive mode. If you want to use more, you must changed this number. This number is assigned for convenience to an environment variable
NGPUS=1
to be used to set the number of threadMPI. Gromacs needs at least one threadMPI per GPU card. If you change the number of GPUs, change also this value in the script. In the script above, 6 cores, were requested, 1 GPU, and 1 single threadMPI is started. So 6 OpenMP threads can be started. This is what the last line does
gmx mdrun -deffnm md_0_1 -ntmpi $NGPUS -ntomp $OMPTHREADS
NOTES:
- In order to use the resources efficiently, the product of the number of threaMPI processes (flag
-ntmpi) and of the OpenMP threads (flag-ntomp) should be equal to the total number of cores requested (#BSUB -n X, saved in the environment variableLSB_DJOB_NUMPROC). - Please specify explicitly the
-ntmpiand-tompoptions. If those are missing, gromacs assigns default values that are either the total number of physical cores of the machine, or 1, depending on the queue settings. Both defaults are most likely wrong. - Adding extra Force Fields: users can add additional force fields by copying the corresponding
<force-field-name>.ffdirectories in their working directory, i.e. the directory from where they start gromacs. Then, when gromacs starts, it will look for*.ffdirectories first in the working directory, and then in the system ones, and will pick up all the*.ffforce fields that it finds in both the places.