MPI and OpenMP can be combined to write programs that exploit the possibility given by multi-core SMP machines, and the possibility of using the Message Passing Interface based communications between nodes on the cluster.
Special care must be taken to write the job script, for this kind of problem.
Typically, one would want to run a different MPI process on each node, and then a certain number of openMP threads on each node.
An example script is the following:
#!/bin/sh ### General options ### –- specify queue -- #BSUB -q hpc ### -- set the job Name -- #BSUB -J Hybrid_MPI_openMP ### -- ask for total number of cores (wherever) -- #BSUB -n 8 ### -- specify that must be assigned 4 per node -- #BSUB -R "span[ptile=4]" ### -- set walltime limit: hh:mm -- #BSUB -W 72:00 ### -- specify that we need 4GB of memory per core/slot -- #BSUB -R "rusage[mem=4GB]" ### -- set the email address -- # please uncomment the following line and put in your e-mail address, # if you want to receive e-mail notifications on a non-default address ##BSUB -u your_email_address ### -- send notification at start -- #BSUB -B ### -- send notification at completion -- #BSUB -N ### -- Specify the output and error file. %J is the job-id -- ### -- -o and -e mean append, -oo and -eo mean overwrite -- #BSUB -o Output_%J.out #BSUB -e Output_%J.err # -- load compiler (if needed) and the OpenMPI module -- # NOTE: this is just an example: check that you load the right modules module load mpi/4.1.4-gcc-12.2.0-binutils-2.39 # -- OpenMP environment variables -- Num=$(echo $LSB_SUB_RES_REQ | sed -e 's/span\[ptile=/ /g' -e 's/\]/ /g') OMP_NUM_THREADS=$Num export OMP_NUM_THREADS # here follow the commands you want to execute myapplication.x < input.in > output.out # -- program invocation here -- mpirun -npernode 1 your_hybrid_program
This program asks for a total of 8 cores, three on each of two nodes. On each of these nodes it will run 1 MPI process, that will in turn use 4 OpenMP threads. We want the threads to run on two different nodes, and not all on a single node (this may not be important in general, but in same case it can be crucial). Then we want that only 1 MPI process runs on each node.
This is realized in this way.
- First, reserve the cores:
#BSUB -n 8
- Then asks to have three cores per node:
#BSUB -R "span[ptile=4]"
- Set the number of threads to be used in each of the OpenMP blocks:
OMP_NUM_THREADS=$Num;
export OMP_NUM_THREADS
Note: the number of threads can also be set through OpenMP runtime library routines, and it can be different for the different MPI processes. In this case be sure to have enough cores reserved on each node.
- Load the OpenMPI library
module load mpi/4.1.4-gcc-12.2.0-binutils-2.39
- Run the program through the MPI wrapper, specifying that only 1 MPI process per node should be started:
mpirun -npernode 1 your_hybrid_program
Notes:
- This is tested with OpenMPI, it could not work with other MPI implementations.
- The number of openMP threads is set using
$num, that is based value specified inspan[ptile=3]. This can be not ideal if you want to run more that one MPI process per node, for example.- It is possible to run more that one MPI process per node, but you have to modify the script accordingly.
- If you need to specify other OpenMP special environment variable, follow the same syntax as for
OMP_NUM_THREADS- You can of course add other MPI options.