Configure APS across multiple queues

Procedure

Use QUEUE_GROUP in an absolute priority queue in lsb.queues to configure APS across multiple queues.

When APS is enabled in the queue with APS_PRIORITY, the FAIRSHARE_QUEUES parameter is ignored. The QUEUE_GROUP parameter replaces FAIRSHARE_QUEUES, which is obsolete in LSF 7.0.

For example, you want to schedule jobs from the normal queue and the short queue, factoring the job priority (weight 1) and queue priority (weight 10) in the APS value:

Begin Queue
QUEUE_NAME   = normal
PRIORITY     = 30
NICE         = 20
APS_PRIORITY = WEIGHT [[JPRIORITY, 1] [QPRIORITY, 10]]
QUEUE_GROUP = short
DESCRIPTION  = For normal low priority jobs, running only if hosts are lightly loaded.
End Queue
...
Begin Queue   
QUEUE_NAME   = short
PRIORITY     = 20
NICE         = 20
End Queue

The APS value for jobs from the normal queue and the short queue are: calculated as:

APS_PRIORITY = 1 * (1 * job_priority + 10 * queue_priority)

The first 1 is the weight of the WORK factor; the second 1 is the weight of the job priority subfactor; the 10 is the weight of queue priority subfactor.

If you want the job priority to increase based on the pending time, you must configure JOB_PRIORITY_OVER_TIME parameter in the lsb.params.

Example

Extending the example, you now want to add user-based fairshare with a weight of 100 to the APS value in the normal queue:

Begin Queue
QUEUE_NAME   = normal
PRIORITY     = 30
NICE         = 20
FAIRSHARE = USER_SHARES [[user1, 5000] [user2, 5000] [others, 1]]
APS_PRIORITY = WEIGHT [[JPRIORITY, 1] [QPRIORITY, 10] [FS, 100]]
QUEUE_GROUP = short
DESCRIPTION  = For normal low priority jobs, running only if hosts are lightly loaded.
End Queue

The APS value is now calculated as

APS_PRIORITY = 1 * (1 * job_priority + 10 * queue_priority) + 100 * user_priority

Finally, you now to add swap space to the APS value calculation. The APS configuration changes to:

APS_PRIORITY = WEIGHT [[JPRIORITY, 1] [QPRIORITY, 10] [FS, 100] [SWAP, -10]]

And the APS value is now calculated as

APS_PRIORITY = 1 * (1 * job_priority + 10 * queue_priority) 
+ 100 * user_priority + 1 * (-10 * SWAP)

View pending job order by the APS value

Procedure

Run bjobs -aps to see APS information for pending jobs in the order of absolute scheduling priority.

The order that the pending jobs are displayed is the order in which the jobs are considered for dispatch.

The APS value is calculated based on the current scheduling cycle, so jobs are not guaranteed to be dispatched in this order.

Pending jobs are ordered by APS value. Jobs with system APS values are listed first, from highest to lowest APS value. Jobs with calculated APS values are listed next ordered from high to low value. Finally, jobs not in an APS queue are listed. Jobs with equal APS values are listed in order of submission time.

Results

If queues are configured with the same priority, bjobs -aps may not show jobs in the correct expected dispatch order. Jobs may be dispatched in the order the queues are configured in lsb.queues. You should avoid configuring queues with the same priority.

Example bjobs -aps output

The following example uses this configuration;

The APS only considers the job priority and queue priority for jobs from normal queue (priority 30) and short queue (priority 20)
- APS_PRIORITY = WEIGHT [[QPRIORITY, 10] [JPRIORITY, 1]]
- QUEUE_GROUP = short
Priority queue (40) and idle queue (15) do not use APS to order jobs
JOB_PRIORITY_OVER_TIME=5/10 in lsb.params
MAX_USER_PRIORITY=100 in lsb.params

bjobs -aps was run at 14:41:

bjobs -aps
JOBID     USER     STAT    QUEUE    FROM_HOST    JOB_NAME     SUBMIT_TIME    APS
  15     User2     PEND    priority    HostB       myjob     Dec 21 14:30     -
  22     User1     PEND    Short       HostA       myjob     Dec 21 14:30     (60)
   2     User1     PEND    Short       HostA       myjob     Dec 21 11:00     360
  12     User2     PEND    normal      HostB       myjob     Dec 21 14:30     355
   4     User1     PEND    Short       HostA       myjob     Dec 21 14:00     270
   5     User1     PEND    Idle        HostA       myjob     Dec 21 14:01      -

For job 2, APS = 10 * 20 + 1 * (50 + 220 * 5 /10) = 360For job 12, APS = 10 *30 + 1 * (50 + 10 * 5/10) = 355For job 4, APS = 10 * 20 + 1 * (50 + 40 * 5 /10) = 270

View APS configuration for a queue

Procedure

Run bqueues -l to see the current APS information for a queue:

bqueues -l normal
 
QUEUE: normal
  -- No description provided.  This is the default queue.
 
PARAMETERS/STATISTICS
PRIO NICE STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN SSUSP USUSP  RSV
500   20  Open:Active       -    -    -    -     0     0     0     0     0    0
 
SCHEDULING PARAMETERS
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -
 
SCHEDULING POLICIES:  FAIRSHARE  APS_PRIORITY
APS_PRIORITY:
                   WEIGHT FACTORS    LIMIT FACTORS    GRACE PERIOD
  FAIRSHARE            10000.00                -              -
  RESOURCE            101010.00                -          1010h
    PROCESSORS           -10.01                -              -
    MEMORY              1000.00         20010.00             3h
    SWAP               10111.00                -              -
  WORK                     1.00                -              -
    JOB PRIORITY     -999999.00         10000.00          4131s
    QUEUE PRIORITY     10000.00            10.00              -
 
USER_SHARES:  [user1, 10]
 
SHARE_INFO_FOR: normal/
 USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME
user1       10       3.333      0        0         0.0        0
 
USERS: all
HOSTS:  all
REQUEUE_EXIT_VALUES:  10