General Info
OpenFOAM is an open source Computational Fluid Dynamics software Package used for simulations in many different fields in engineering. It has a modular structure, and consists of many different solvers and utilities for pre- and post-processing. For more information and documentation please refer to the OpenFOAM website.
Being a free and open-source code, OpenFOAM can be modified and re-compiled by the user. However, if you’re going to run it on the DTU HPC cluster we recommend to use one of the versions already installed on our system. The package is installed as a module, so you have to load it before using it.
Test/Interactive run
If you want to test it with a short interactive run, log in to the system and open an interactive session with
linuxsh
Note: if you open a ssh session from the command line, remember that you are on the front-end node. You have to first switch to an interactive node to be able to load any module.
Load the necessary modules. For this example the 5.0 version is used:
module load OpenFoam/5.0/gcc-6.3.0-openmpi
Note: When loading the module, the usual openFOAM environment variables are set, together with the alias usually set at install time. For example, you can move to the tutorials directory by typing tut. However, always copy the relevant files to a folder in your own directory.
OpenFOAM job scripts
If you came to the HPC to use OpenFOAM you are probably interested in running it in batch mode, to take advantage of the cluster computational resources. OpenFOAM is compiled against OpenMPI and can therefore run in parallel both on a single node and across multiple nodes. Remember that you have to prepare your model (“case” in OpenFOAM language) so that it is correctly set-up to run in parallel, otherwise it will not work as you would like to. We show here two simple job script examples, one for a serial job, and one for a parallel one.
Serial job
We use the same files used for the previous interactive example. According to the general instructions for writing a job script the first part of the script contains the information for the resource manager, and the second part the user’s commands:
#!/bin/sh ### General options ### -- set the job Name -- #BSUB -J openfoam_Serial ### -- ask for number of cores (default: 1) -- #BSUB -n 1 ### -- set walltime limit: hh:mm -- #BSUB -W 16:00 ### -- specify that we need 2GB of memory per core/slot -- #BSUB -R "rusage[mem=2GB]" ### -- specify that we want the job to get killed if it exceeds 3 GB per core/slot -- #BSUB -M 3GB ### -- set the email address -- # please uncomment the following line and put in your e-mail address, # if you want to receive e-mail notifications on a non-default address ##BSUB -u your_email_address ### -- send notification at start -- #BSUB -B ### -- send notification at completion-- #BSUB -N ### -- Specify the output and error file. %J is the job-id -- ### -- -o and -e mean append, -oo and -eo mean overwrite -- #BSUB -o Output_%J.out #BSUB -e Error_%J.err # -- load the OpenFOAM module -- module load OpenFoam/5.0/gcc-6.3.0-openmpi # call OpenFOAM interFOAM -parallel > Output.log
Save the script with the name you like, for example OpenFOAM_serial.sh and submit it:
bsub < OpenFOAM_serial.sh
In this script we ask for only one core on one single node. An explanation of the #BSUB options can be found here.
Parallel job
Here is a simple script for parallel OpenFOAM execution.
#!/bin/sh ### General options ### -- set the job Name -- #BSUB -J openfoam_Parallel ### -- ask for number of cores (default: 1) -- #BSUB -n 8 ### -- specify that the cores must be on the same host -- #BSUB -R "span[hosts=1]" ### -- set walltime limit: hh:mm -- #BSUB -W 16:00 ### -- specify that we need 2GB of memory per core/slot -- #BSUB -R "rusage[mem=2GB]" ### -- specify that we want the job to get killed if it exceeds 3 GB per core/slot -- #BSUB -M 3GB ### -- set the email address -- # please uncomment the following line and put in your e-mail address, # if you want to receive e-mail notifications on a non-default address ##BSUB -u your_email_address ### -- send notification at start -- #BSUB -B ### -- send notification at completion-- #BSUB -N ### -- Specify the output and error file. %J is the job-id -- ### -- -o and -e mean append, -oo and -eo mean overwrite -- #BSUB -o Output_%J.out #BSUB -e Error_%J.err # -- load the OpenFOAM module -- module load OpenFoam/5.0/gcc-6.3.0-openmpi # -- program invocation here -- mpirun interFOAM -parallel > Output.log
In this script, 8 cores are reserved on one single node:
#BSUB -n 8 #BSUB -R "span[hosts=1]"
Notice that there is no need to specify the number of cores as an argument of mpirun because the system OpenMPI installation gets this information directly from the resource manager.
To reserve 16 cores on 2 nodes, 8 on each one,
#BSUB -n 16 #BSUB -R "span[ptile=8]"
In this case the scheduler could will allocate all 8 processes to a node and the next 8 to a second node.
Note: On the cluster there are nodes with different amount of cores. Use the command nodestat to see the current distribution of nodes/cores. It is recommended to completely fill-up a node, before distributing across multiple nodes.
Running “BIG” jobs
OpenFOAM produces a lot of small files during the computation, and with a peculiar hierarchy of directories. During a single simulations, many thousands small files are created, read and written, and this puts a lot of pressure on the filesystem, potentially causing substantial performance degradation, and affecting the whole cluster. Following some advices can help in mitigating these side-effects, and at the same time give better performance.
1. Run on one of the scratch filesystems
We recommend the OpenFOAM users to run their BIG jobs directly on one of the scratch filesystem. Potential advantages are that these filesystems are connected to the nodes via a faster interconnect than $HOME, there is a large quota limit, so even large simulations can be run. Scratch filesystems are not backed-up, however, and their performance degrades rapidly when there is not enough free space left.
For these reasons, you are asked to:
- write an email to support@cc.dtu.dk and ask for a personal directory under one of our scratch filesystems (it will be called after your username, e.g. /work3/username/);
- read carefully the instructions in /work3/readme.txt;
- run the simulation in your /work3/username/;
- there is no back-up, so keep a copy of your jobscript- and input-files in your home directory;
- this is a shared filesystem with a large quota: BUT please clean up the files and directories you don’t need any longer as soon as possible;
- after the simulation:
- reconstruct the model, and then remove all the
processor##directories;
- copy the important results back to $HOME (if you do it manually, then please use the dedicated transfer.gbar.dtu.dk machine);
- remove the original files from the scratch filesystem.
- reconstruct the model, and then remove all the
2. Change I/O Settings in your project
The amount of read/write/inspect that a simulation does is controlled by the settings in the controlDict database (file system/controlDict in the main directory of your case, see the OpenFOAM guide).
1. Reduce the checkpoint frequency:
keywords writeControl and writeInterval
Set the variables that control the frequency of writing to file: writeControl and writeInterval keywords.
2. Reduce the number of checkoint files saved during the job executions:
keyword purgeWrite
You can control the number of checkpoints kept during the computations, i.e. you can have high frequency of checkpoint but keep only the most recent, the older ones are deleted. Set the keyword purgeWrite. purgeWrite 0 means that all the checkpoints are kept. purgeWrite n means that only n checkpoints are kept, cyclically overwriting the older ones during the simulation. For steady-state solutions, results from previous iterations can be continuously overwritten by specifying purgeWrite 1. Notice: the default value is 0.
3. Avoid very small time step:
keyword deltaT
Using unnecessarily small time steps affects the performance of your simulations, because at each time step the data need to be synchronized, and a log file entry is written. This book-keeping requires additional time, and puts additional load to the filesystem. It’s beneficial therefore to aim for the largest possible value of the time step, which can be done for example by increasing the number of cells per core.
4. Do not allow on-the-fly modifications:
keyword runTimeModifiable no
By default, OpenFOAM allow the user to modify the problem parameters on the fly. This is a bad idea for many reasons, and this setting should only be used in testing/debugging mode. It has a very bad effect on performance, since forces OpenFOAM to inquire about the properties (stats) of the input files (all the dictionaries) at every time step, putting additionally unnecessary load on the filesystem, and slowing down the computations. Therefore please explicitly set the keyword runTimeModifiable to no.
3. Estimating the disk space required
It can be a good idea to estimate the amount of data your job will produce:
- Create a testjob with a couple of timesteps;
- then use a
du -h
on you subdirectory to find out the size of a single timestep for a single processor and then estimate the amount of data that the full run will produce. If it’s more than 500GB for the complete run then think about your setup again, maybe reducing the frequency of the saved data 🙂
4. Reconstructing the model: hint
After running OpenFOAM in parallel you have to reconstruct your model. You can do it via a small batch script, like:
#!/bin/sh
### General options
### -- set the job Name --
#BSUB -J openfoam_Reconstruct
### -- ask for number of cores (default: 1) --
#BSUB -n 1
### -- set walltime limit: hh:mm --
#BSUB -W 8:00
### -- specify that we need 2GB of memory per core/slot --
#BSUB -R "rusage[mem=2GB]"
### -- specify that we want the job to get killed if it exceeds 3 GB per core/slot --
#BSUB -M 3GB
### -- set the email address --
# please uncomment the following line and put in your e-mail address,
# if you want to receive e-mail notifications on a non-default address
##BSUB -u your_email_address
### -- send notification at start --
#BSUB -B
### -- send notification at completion--
#BSUB -N
### -- Specify the output and error file. %J is the job-id --
### -- -o and -e mean append, -oo and -eo mean overwrite --
#BSUB -o Output_%J.out
#BSUB -e Error_%J.err
# -- load the OpenFOAM module --
module load OpenFoam/5.0/gcc-6.3.0-openmpi
# call OpenFOAM
# -- if needed, cd to the directory that contains your processor# directory --
#cd ~/Path_of_your_Case
# -- run model reconstruction --
reconstructPar
ret=$?
# -- remove processor-directories if reconstruction was successful
if [ $ret -eq 0 ] ; then
for i in processor* ; do rm -rf $i & done
wait
fi
exit $ret
Then run this job files as a normal job.
Notice the lines
if $ret -eq 0 ; then
for i in processor*; do rm -rf $i &; done
wait
fi
that check the returning code of the reconstructPar command to be sure that the reconstruction was successful before start deleting the temporary processor## directories.
Older Versions
You may be interested in using older versions of openFOAM for benchmarking, or for compatibility issues related to the defective backward-compatibility of the new releases. For this reason, the older versions of openFOAM are kept available to the cluster users, as independent modules. After the login, and after moving to one of the app nodes (linuxsh) type
module list
to check which openFOAM modules are available, and load the one that you need. Be aware that you should probably modify the way your model is specified, in order to make it run with different openFOAM versions.