Using GPUs under LSF10


To use our LSF10-setup you have to login to login2.gbar.dtu.dk or login2.hpc.dtu.dk.

We have right now 14 nodes with GPUs in our LSF10-setup.
The walltime is limited to 24 hours per job at the moment.

4 (old) nodes with 2 x Tesla M2050 – queuename: gpum2050
2 nodes with 4 x TitanX (Pascal) – queuename: gputitanxpascal
6 nodes with 2 x Tesla V100 16 GB (owned by DTU Compute&elektro) – queuename: gpuv100
4 nodes with 2 x Tesla V100 32 GB (owned by DTU Compute&DTU Environment&DTU MEK) – queuename gpuv100
3 nodes with 4 x Tesla V100 32 GB with NVlink (owned by DTU Compute) – queuename gpuv100
1 node with 4 x Tesla K80 – queuename: gpuk80
1 node with 4 x Tesla K40 – queuename: gpuk40
MATLAB should be able to utilize the Nvidia M2050, for Tensorflow & other modern software you need to use the TitanX-queue or one of the other GPU-queues.

1 interactive V100-node reachable via voltash
1 interactive V100-node with NVlink reachable via sxm2sh.

Here is an example jobscript:

 

#!/bin/sh
### General options
### –- specify queue --
#BSUB -q gpuv100
### -- set the job Name --
#BSUB -J testjob
### -- ask for number of cores (default: 1) --
#BSUB -n 1
### -- Select the resources: 1 gpu in exclusive process mode --
#BSUB -gpu "num=1:mode=exclusive_process"
### -- set walltime limit: hh:mm --  maximum 24 hours for GPU-queues right now
#BSUB -W 1:00
# request 5GB of system-memory
#BSUB -R "rusage[mem=5GB]"
### -- set the email address --
# please uncomment the following line and put in your e-mail address,
# if you want to receive e-mail notifications on a non-default address
##BSUB -u your_email_address
### -- send notification at start --
#BSUB -B
### -- send notification at completion--
#BSUB -N
### -- Specify the output and error file. %J is the job-id --
### -- -o and -e mean append, -oo and -eo mean overwrite --
#BSUB -o gpu-%J.out
#BSUB -e gpu_%J.err
# -- end of LSF options --

nvidia-smi
# Load the cuda module
module load cuda/9.1

/appl/cuda/9.1/samples/bin/x86_64/linux/release/deviceQuery

Then submit with

bsub < jobscript.sh

For requesting a GPU with 32GB of memory, then please add a

#BSUB -R "select[gpu32gb]"

For requesting GPUs with NVLINK, then please add a

#BSUB -R "select[sxm2]"

to your jobscript.