LSF calculates a dynamic user priority for individual users or for a group, depending on how the shares are assigned. The priority is dynamic because it changes as soon as any variable in formula changes. By default, a user’s dynamic priority gradually decreases after a job starts, and the dynamic priority immediately increases when the job finishes.
By default, LSF calculates the dynamic priority for each user based on:
The number of shares assigned to the user
The resources used by jobs belonging to the user:
Number of job slots reserved and in use
Run time of running jobs
Cumulative actual CPU time (not normalized), adjusted so that recently used CPU time is weighted more heavily than CPU time used in the distant past
If you enable additional functionality, the formula can also involve additional resources used by jobs belonging to the user:
Decayed run time of running jobs
Historical run time of finished jobs
Committed run time, specified at job submission with the -W option of bsub, or in the queue with the RUNLIMIT parameter in lsb.queues
Memory usage adjustment made by the fairshare plugin (libfairshareadjust.*).
LSF measures resource usage differently, depending on the type of fairshare:
For user-based fairshare:
For queue-level fairshare, LSF measures the resource consumption of all the user’s jobs in the queue. This means a user’s dynamic priority can be different in every queue.
For host partition fairshare, LSF measures resource consumption for all the user’s jobs that run on hosts in the host partition. This means a user’s dynamic priority is the same in every queue that uses hosts in the same partition.
For queue-based fairshare, LSF measures the resource consumption of all jobs in each queue.
By default, LSF calculates dynamic priority according to the following formula:
dynamic priority = number_shares / (cpu_time * CPU_TIME_FACTOR + run_time * RUN_TIME_FACTOR + (1 + job_slots) * RUN_JOB_FACTOR + fairshare_adjustment*FAIRSHARE_ADJUSTMENT_FACTOR)
The maximum value of dynamic user priority is 100 times the number of user shares (if the denominator in the calculation is less than 0.01, LSF rounds up to 0.01).
For cpu_time, run_time, and job_slots, LSF uses the total resource consumption of all the jobs in the queue or host partition that belong to the user or group.
The number of shares assigned to the user.
The cumulative CPU time used by the user (measured in hours). LSF calculates the cumulative CPU time using the actual (not normalized) CPU time and a decay factor such that 1 hour of recently-used CPU time decays to 0.1 hours after an interval of time specified by HIST_HOURS in lsb.params (5 hours by default).
The total run time of running jobs (measured in hours).
The number of job slots reserved and in use.
The adjustment calculated by the fairshare adjustment plugin (libfairshareadjust.*).
You can give additional weight to the various factors in the priority calculation by setting the following parameters for the queue in lsb.queues or for the cluster in lsb.params. When the queue value is not defined, the cluster-wide value from lsb.params is used.
CPU_TIME_FACTOR
RUN_TIME_FACTOR
RUN_JOB_FACTOR
FAIRSHARE_ADJUSTMENT_FACTOR
HIST_HOURS
If you modify the parameters used in the dynamic priority formula, it affects every fairshare policy in the cluster.
The CPU time weighting factor.
Default: 0.7
The run time weighting factor.
Default: 0.7
The job slots weighting factor.
Default: 3
The fairshare plugin (libfairshareadjust.*) weighting factor.
Default: 0
Interval for collecting resource consumption history
Default: 5
In some cases the dynamic priority equation may require adjustments beyond the run time, cpu time, and job slot dependencies provided by default. The fairshare adjustment plugin is open source and can be customized once you identify specific requirements for dynamic priority.
All information used by the default priority equation (except the user shares) is passed to the fairshare plugin. In addition, the fairshare plugin is provided with current memory use over the entire cluster and the average memory that is allocated to a slot in the cluster.
If you modify the parameters used in the dynamic priority formula, it affects every fairshare policy in the cluster. The fairshare adjustment plugin (libfairshareadjust.*) is not queue-specific. Parameter settings passed to the fairshare adjustment plugin are those defined in lsb.params.
Jobs assigned to a single slot on a host can consume host memory to the point that other slots on the hosts are left unusable. The default dynamic priority calculation considers job slots used, but doesn’t account for unused job slots effectively blocked by another job.
The fairshare adjustment plugin example code provided by LSF is found in the examples directory of your installation, and implements a memory-based dynamic priority adjustment as follows:
fairshare adjustment= (1+slots)*((used_memory /used_slots)/(slot_memory*THRESHOLD))