Span string

A span string specifies the locality of a parallel job. If span is omitted, LSF allocates the required processors for the job from the available set of processors.

Syntax

The span string supports the following syntax:

span[hosts=1]

Indicates that all the processors allocated to this job must be on the same host.

span[block=value]

For parallel jobs, LSF will allocate slots to the job based on block size. LSF tries to pack as many blocks on one host as possible, then goes to next one. Each host is only checked once.

span[ptile=value]

Indicates the number of processors on each host that should be allocated to the job, where value is one of the following:

Default ptile value, specified by n processors. In the following example, the job requests 4 processors on each available host, regardless of how many processors the host has:
```
span[ptile=4]
```
Predefined ptile value, specified by ’!’. The following example uses the predefined maximum job slot limit lsb.hosts (MXJ per host type/model) as its value:
```
span[ptile='!']
```
Tip:
If the host type/model does not define MXJ, the default predefined ptile value is 1.

Restriction:
Under bash 3.0, the exclamation mark (!) is not interpreted correctly by the shell. To use predefined ptile value (ptile='!'), use the +H option to disable '!' style history substitution in bash (sh +H).
Predefined ptile value with optional multiple ptile values, per host type or host model:
- For host type, you must specify same[type] in the resource requirement. In the following example, the job requests 8 processors on a host of type HP, and 2 processors on a host of type LINUX, and the predefined maximum job slot limit in lsb.hosts (MXJ) for other host types:
```
span[ptile='!',HP:8,LINUX:2] same[type]
```
- For host model, you must specify same[model] in the resource requirement. In the following example, the job requests 4 processors on hosts of model PC1133, and 2 processors on hosts of model PC233, and the predefined maximum job slot limit in lsb.hosts (MXJ) for other host models:
```
span[ptile='!',PC1133:4,PC233:2] same[model]
```

span[hosts=-1]

Disables span setting in the queue. LSF allocates the required processors for the job from the available set of processors.

Resizable jobs

For resource requirements with span[hosts=1], a resize request is limited to slots on the first-execution host of the job. This behavior eliminates the ambiguities that arise when the span expression is modified from the time that the job was originally dispatched.

For span[ptile=n], the job will be allocated exactly n slots on some number of hosts, and a number between 1 and n slots (inclusive) on one host. This is true even if a range of slots is requested. For example, for the following job submission:

bsub -n "1,20" -R "span[ptile=2]" sleep 10000

This special span behavior does not only apply to resize requests. It applies to resizable jobs only when the original allocation is made, and in making additional resize allocations.

If every host has only a single slot available, the job is allocated one slot.

Resize requests with partially filled hosts are handled so that LSF does not choose any slots on hosts already occupied by the job. For example, it is common to use the ptile feature with span[ptile=1] to schedule exclusive jobs.

For a resizable job (auto-resizable or otherwise) with a range of slots requested and span[ptile=n], whenever the job is allocated slots, it will receive either of the following:

The maximum number of slots requested, comprising n slots on each of a number of hosts, and between 0 and n-1 (inclusive) slots on one host
n slots on each of a number of hosts, summing to some value less than the maximum

For example, if a job requests between 1 and 14 additional slots, and span[ptile=4] is part of the job resource requirement string, when additional slots are allocated to the job, the job receives either of the following:

14 slots, with 2 slots on one host and 4 slots on each of 3 hosts
4, 8 or 12 slots, such that 4 slots are allocated per host of the allocation

Note:

Resizable jobs cannot have compound resource requirements.

Example

When running a parallel exclusive job, it is often desirable to specify span[ptile=1] so that the job is allocated at most one slot on each host. For an autoresizable job, new slots are allocated on hosts not already used by the job. The following job submission specifies this resource request:

bsub -x -ar -app <appplicaion_file> -n "1,100" -R "span[ptile=1]" myjob

When additional slots are allocated to a running job, the slots will be on new hosts, not already occupied by the job.

Block Scheduling

For applications that are not especially sensitive to network latency, or where you prefer to get throughput, you can allocate slots for a parallel job with a specific block size. The applications specified by the job may be running as threaded processes on groups of n cores, but using MPI applications or other socket connections between blocks. LSF will allocate slots to the job based on block size. LSF tries to pack as many blocks on one host as possible, then goes to next one. Each host is only checked once. It does not matter which host contains the slot blocks. The job can start as soon as any previous job is complete.

In the illustration below, for example, each color represents a different job. There are four 16 way jobs:

For bsub -n 16 and block=4, only 4 x 4 slot blocks are necessary. It does not matter which host contains the slot blocks. The job can start as soon as any previous job is complete.

This packing policy is supported by the keyword block (“span[block=value]”) in the span section of the resource requirement string. “span[block=value]” can also be configured in the RES_REQ parameter in lsb.queues and lsb.applications.

When a block size is specified for a job, LSF allocates only a multiple of the block size for the job. For example, for jobs with block size = 4:

bsub -n 2,13: 4, 8 or 12 slots are allocated to the job (in blocks of size 4).
bsub -n 5: The job is rejected.
bsub -n 9,10: The job is rejected.
bsub -n 2,3: The job is rejected.
bsub -n 12: The job is accept, and allocates 3 blocks of size 4.
bsub -n 2: The job is rejected.
bsub -n 3: The job is rejected.

The minimum value in -n min,max is silently changed to a multiple of the block. For example:

bsub -n 2,8 -R span[block=4] sleep 1d

is changed to:

bsub -n 4,8 -R span[block=4] sleep 1d

LSF tries to pack as many blocks in to one host as possible, then goes to the next host. For example, assume host1 has 8 slots, and host2 has 8 slots, and host3 also has 8 slots, where 2 slots of each host are consumed by other jobs. For a job with -n 9 "span[block=3]”, the allocation will be:

host1: 6 slots
host2: 3 slots

The following is an example of how you can display hosts with their static and dynamic resource information, specify a block size and resource requirements for a job, and see the output:

bhosts
HOST_NAME  STATUS   JL/U   MAX  NJOBS   RUN  SSUSP  USUSP    RSV 
hostA      ok        -     8      0     0      0      0      0
hostB      ok        -     8      0     0      0      0      0
hostC      ok        -     8      0     0      0      0      0
hostD      unavail   -     1      0     0      0      0      0
hostE      ok        -     4      0     0      0      0      0
hostF      ok        -     4      0     0      0      0      0
 
bsub -n 24 -R "order[slots] span[block=4]" sleep 1d
Job <418> is submitted to default queue <normal>.
 
bjobs
JOBID  USER  STAT  QUEUE  FROM_HOST  EXEC_HOST  JOB_NAME  SUBMIT_TIME
418    user1 RUN   normal hostE      8*hostC    sleep 1d  Sep  4 21:36
                                     8*hostB    sleep 1d  Sep  4 21:36
                                     8*hostA    sleep 1d  Sep  4 21:36
 
bhosts
HOST_NAME  STATUS    JL/U   MAX  NJOBS   RUN  SSUSP  USUSP   RSV
hostA      closed     -     8      8     8      0      0     0
hostB      closed     -     8      8     8      0      0     0
hostC      closed     -     8      8     8      0      0     0
hostD      unavail    -     1      0     0      0      0     0
hostE      ok         -     4      0     0      0      0     0
hostF      ok         -     4      0     0      0      0     0

The following are some additional examples of how you can use “span[block=value]” when submitting a job with resource requirements:

To specify a predefined block value, per host type or host model, using !:

bsub -n "2,10" –R "span[block='!'] same[type]" myjob
To specify a predefined block value with optional multiple block values, per host type or host model:

bsub -n "2,10" –R “span[block='!',HP:8,SGI:8,LINUX:2] same[type]" myjob

If the host type/model does not define MXJ, the default predefined block value is 1.

“span[block=value]” can be displayed by bjobs -l, bhist -l, bqueues -l, bapp -l and bacct -l.

When using the block scheduling feature, note the following:

For Queue Host Limit (HOSTLIMIT_PER_JOB), mbatchd will not reject a job with block=x because the exact number of allocated hosts can only be obtained during scheduling.
“span[block=value]” and “span[ptile=value]” cannot be specified at the same time. “span[block=value]” and “span[host=value]” also cannot be specified at the same time because span cannot accept more than one criteria and multiple -R does not support multiple span definitions.
In MultiCluster forwarding model, job with block=x can not be forwarded to a remote cluster which has a version prior to 9.1.2. In MultiCluster leasing model, job with block=x can not be allocated to hosts leased from a remote cluster with a version prior to 9.1.2.