MPI Jobs under LSF

We assume here, that you know how to program MPI, and that you have a working MPI application.

Choose the right MPI version: MPI libraries and compilers must fit together – thus we provide different versions for different compilers. You can check the available versions with the module command:

module avail mpi

The most common ones are mpi/gcc (OpenMPI for GCC), mpi/studio (OpenMPI for Sun/Oracle Studio) and mpi/intel (OpenMPI for Intel Compilers). The default version is mpi/gcc.

Let us say, you have a MPI program compiled with gcc against openmpi, my_mpi_prog – you want to execute that program on 8 cores and you don’t care about the distribution over compute nodes:

#!/bin/sh
# embedded options to bsub - start with #BSUB
### -- set the job Name --
#BSUB -J MPIJob
### –- specify queue --
#BSUB -q hpc
### -- ask for number of cores (default: 1) --
#BSUB -n 8
### -- set walltime limit: hh:mm --
#BSUB -W 10:00 
### -- specify that we need 4GB of memory per core/slot -- 
#BSUB -R "rusage[mem=4GB]"
### -- set the email address --
# please uncomment the following line and put in your e-mail address,
# if you want to receive e-mail notifications on a non-default address
##BSUB -u your_email_address
### -- send notification at start --
#BSUB -B
### -- send notification at completion--
#BSUB -N
### -- Specify the output and error file. %J is the job-id -- 
### -- -o and -e mean append, -oo and -eo mean overwrite -- 
#BSUB -o Output_%J.out
#BSUB -e Output_%J.err
# here follow the commands you want to execute
# 

# load the necessary modules
# NOTE: this is just an example, check with the available modules
module load mpi/4.1.4-gcc-12.2.0-binutils-2.39
### This uses the LSB_DJOB_NUMPROC to assign all the cores reserved
### This is a very basic syntax. For more complex examples, see the documentation
mpirun -np $LSB_DJOB_NUMPROC ./my_MPI_program

If you want to have all 8 cores of one node (our HPC nodes have eight cores each), add the line

#BSUB -R "span[hosts=1]"

If you want to run the same program on 8 cores, on 2 nodes, this line should be replaced by

#BSUB -R "span[ptile=4]"
  • The environment variable LSB_DJOB_NUMPROC is automatically set to the total number of cores reserved by LSF.
  • When using openMPI, you can omit the -np $LSB_DJOB_NUMPROC command, because the program gets the information about the total number of cores automatically. If you use different MPI library, it may be necessary to specify explicitly the numberof MPI processes on the command line in this way.