LSF documentation in the IBM Knowledge Center

The IBM Knowledge Center is the home for IBM product documentation. Find Platform LSF documentation in the IBM Knowledge Center on the IBM Web site: www.ibm.com/support/knowledgecenter/SSETD4/.

Search all the content in IBM Knowledge Center for subjects that interest you, or search within a product, or restrict your search to one version of a product. Sign in with your IBM ID to take full advantage of the personalization features available in IBM Knowledge Center. Create and print custom collections of documents you use regularly, and communicate with colleagues and IBM by adding comments to topics.

Documentation available through the IBM Knowledge Center may be updated and regenerated following the original release of Platform LSF 9.1.3.

LSF documentation packages

You can download, extract and install these packages to any server on your system to have a local version of the full LSF documentation set. Navigate to the location where you extracted the files and open index.html in any browser. Easy access to each document in PDF and HTML format is provided, as well as full search capabilities within the full documentation set or within a specific document type.

If you have installed IBM Platform Application Center (PAC), you can access and search the LSF documentation through the Help link in the user interface.

LSF documentation in PDF format

Platform LSF documentation is also available in PDF format On the IBM Publications Center: www.ibm.com/e-business/linkweb/publications/servlet/pbi.wss.

LSF documentation in PDF format is also available for Version 9.1.2 and earlier releases on the IBM Support Portal: http://www.ibm.com/support/customercare/sas/f/plcomp/platformlsf.html.

Connect. Learn. Share. Collaborate and network with the IBM Platform Computing experts at the IBM Technical Computing community. Access the Technical Computing community on IBM Service Management Connect at http://www.ibm.com/developerworks/servicemanagement/tc/. Join today!

• IBM AIX 6.x and 7.x on IBM Power 6/7/8
• Linux Kernel 2.6 and 3.x on IBM Power 6/7/8
• Linux x64 Kernel 2.6 and 3.x on x86_64
• HP UX B.11.31 (64-bit) on HP 9000 Servers (PA-RISC)
• HP UX B.11.31 (IA64) on HP Integrity Servers (Itanium2)
• Solaris 10 and 11 on Sparc
• Solaris 10 and 11 on x86-64
• Cray XE6, XT6, XC-30, Linux Kernel 2.6, glibc 2.3 on x86_64
• ARMv8 Kernel 3.12 glibc 2.17
• ARMv7 Kernel 3.6 glibc 2.15 (LSF slave host only)
• Apple Mac OS 10.x (LSF slave host only)
• Windows 2003 SP1/2, 2008 x86, 7 x86, 8 x86, and 8.1 x86 on x86/x86_64 (32-bit)
• Windows 2003 SP1/2, 2003 CCE SP1/SP2, 2008 x64, 7 x64, 8.1 x64, 2008 R2 x64, HPC server 2008, 2012 x64, and 2012 R2 x64 on x86_64 (64-bit)

For detailed LSF system support information, refer to the Compatibility Table at:

www.ibm.com/systems/technicalcomputing/platformcomputing/products/lsf/

To achieve the highest degree of performance and scalability, use a powerful master host.

There is no minimum CPU requirement. For the platforms on which LSF is supported, any host with sufficient physical memory can run LSF as master host. Swap space is normally configured as twice the physical memory. LSF daemons use about 40 MB of memory when no jobs are running. Active jobs consume most of the memory LSF requires.

Platform LSF 7.x, 8.0.x, 8.3, and 9.1.x, servers are compatible with Platform LSF 9.1.3 master hosts. All LSF 7.x, 8.0.x, 8.3, and 9.1.x features are supported by Platform LSF 9.1.3 master hosts.

IBM Platform RTM

Customers can use IBM Platform RTM (Platform RTM) 8.3 or 9.1.x to collect data from Platform LSF 9.1.3 clusters. When adding the cluster, select 'Poller for LSF 8' or 'Poller for LSF 9.1'.

IBM Platform License Scheduler

IBM Platform License Scheduler (License Scheduler) 8.3 and 9.1.x are compatible with Platform LSF 9.1.3.

IBM Platform Analytics

IBM Platform Analytics (Analytics) 8.3 and 9.1.x are compatible with Platform LSF 9.1.3 after the following manual configuration:

IBM Platform Application Center

IBM Platform Application Center (PAC) 8.3 and higher versions are compatible with Platform LSF 9.1.x after the following manual configuration.

If you are using PAC 8.3 with LSF 9.1.x, $PAC_TOP/perf/lsf/8.3 must be renamed to $PAC_TOP/perf/lsf/9.1

For example:

mv /opt/pac/perf/lsf/8.3 /opt/pac/perf/lsf/9.1

To take full advantage of new Platform LSF 9.1.3 features, recompile your existing Platform LSF applications with Platform LSF 9.1.3.

Applications need to be rebuilt if they use APIs that have changed in Platform LSF 9.1.3.

New and changed Platform LSF APIs

For detailed information about APIs changed or created for LSF 9.1.3, refer to the IBM Platform LSF 9.1.3 API Reference.

Third party APIs

Packages are available at www.github.com.

For more information on using third party APIs with LSF 9.1.3 see the Technical Computing community on IBM Service Management Connect at www.ibm.com/developerworks/servicemanagement/tc/plsf/index.html.

To migrate an existing LSF 7 Windows cluster to Platform LSF 9.1.3 on Windows, follow the steps in Migrating IBM Platform LSF Version 7 to IBM Platform LSF Version 9.1.3 on Windows.

Changes to task and slot concept

To keep up with the increasing density of hosts (cores/threads per node) and the growth in threaded applications (for example, a job may request 4 slots, then run 4 threads per slot, so in reality it is using more than 4 cores) there is greater disparity between what a user requests and what needs to be allocated to satisfy the request. This is particularly true in HPC environments where exclusive allocation of nodes is more prevalent.

For consistency, the “slot” concept in LSF has been superseded by “task”. In the first example above, a job running 4 processes each with 4 threads is 16 tasks, and with one task per core, it requires 16 cores to run.

A new parameter, LSB_ENABLE_HPC_ALLOCATION in lsf.conf is introduced. For new installations, this parameter will be enabled automatically (set to Y). For upgrades, it will be set to N and must be enabled manually.

When set to Y|y, this parameter changes the concept of the required number of slots for a job to the required number of tasks for a job. The specified numbers of tasks (using bsub), will be the number of tasks to launch on execution hosts. The allocated slots will change to all slots on the allocated execution hosts for an exclusive job in order to reflect the actual slot allocation.

Restrict job size requested by parallel jobs

Specifying a list of allowed job sizes (number of tasks) in queues or application profiles enables LSF to check the requested job sizes when submitting, modifying, or switching jobs.

Certain applications may yield better performance with specific job sizes (for example, the power of two, so that the job sizes are x^2). The JOB_SIZE_LIST parameter in lsb.queues or lsb.applications defines a discrete list of allowed job sizes for the specified queues or application profiles. LSF will reject jobs requesting job sizes that are not in this list, or jobs requesting multiple job sizes.

The first job size in the JOB_SIZE_LIST is the default job size, which is assigned to jobs that do not explicitly request a job size. The rest of the list can be defined in any order:

JOB_SIZE_LIST=default_size [size ...]

For example, the following defines a job size list for the queue1 queue:

Begin Queue 
QUEUE_NAME = queue1 
... 
JOB_SIZE_LIST=4 2 8 16 
... 
End Queue

This job size list allows 2, 4, 8, and 16 tasks. If you submit a parallel job requesting 10 tasks in this queue (bsub -q queue1 -n 10 ...), that job is rejected because the job size of 10 is not explicitly allowed in the list. The default job size is 4 tasks, and job submissions that do not request a job size are automatically assigned a job size of 4.

When using resource requirements to specify job size, the request must specify a single fixed job size and not multiple values or a range of values:

• When using compound resource requirements with -n (that is, -n with the -R option), ensure that the compound resource requirement matches the -n value, which must match a value in the job size list.
• When using compound resource requirements without -n, the compound resource requirement must imply a fixed job size number, and the implied total job size must match a value in the job size list.
• When using alternative resource requirements, each of the alternatives must request a fixed job size number, and all alternative values must match the values in the job size list.

When defined in both a queue (lsb.queues) and an application profile (lsb.applications), the job size request must satisfy both requirements. In addition, JOB_SIZE_LIST overrides any TASKLIMIT (TASKLIMIT replaces PROCLIMIT in LSF 9.1.3) parameters defined at the same level.

Terminate Orphan Jobs

Often, complex workflows are required with job dependencies for proper job sequencing as well as job failure handling. For a given job, called the parent job, there can be child jobs which depend on its state before they can start. If one or more conditions are not satisfied, a child job remains pending. However, if the parent job is in a state such that the event on which the child depends will never occur, the child becomes an orphan job. For example, if a child job has a DONE dependency on the parent job but the parent ends abnormally, the child will never run as a result of the parent’s completion and it becomes an orphan job.

Keeping orphan jobs in the system can cause performance degradation. The pending orphan jobs consume unnecessary system resources and add unnecessary loads to the daemons which can impact their ability to do useful work.

Submitting a job with a user-specified host file

When submitting a job, you can point the job to a file that specifies hosts and number of slots for job processing.

For example, some applications (typically when benchmarking) run best with a very specific geometry. For repeatability (again, typically when benchmarking) you may want it to always run it on the same hosts, using the same number of slots.

The user-specified host file specifies a host and number of slots to use per task, resulting in a rank file.

The -hostfile option allows a user to submit a job, specifying the path of the user-specified host file:

bsub -hostfile "spec_host_file"

Any user can create a user-specified host file. It must be accessible by the user from the submission host. It lists one host per line. The format is as follows:

# This is a user-specified host file
<host_name1>   [<# slots>]
<host_name2>   [<# slots>]
<host_name1>   [<# slots>]
<host_name2>   [<# slots>]
<host_name3>   [<# slots>]
<host_name4>   [<# slots>]

The resulting rank file is made available to other applications (such as MPI).

The LSB_DJOB_RANKFILE environment variable is generated from the user-specified host file. If a job is not submitted with a user-specified host file then LSB_DJOB_RANKFILE points to the same file as LSB_DJOB_HOSTFILE.

The esub parameter LSB_SUB4_HOST_FILE reads and modifies the value of the -hostfile option.

Use bsub -hostfile (or bmod -hostfile for a pending job) to enter the location of a user-specified host file containing a list of hosts and slots on those hosts. The job will dispatch on the specified allocation once those resources become available.

Use bmod -hostfilen to remove the hostfile option from a job.

bjobs -l and bhist -l show the host allocation for a given job.

Use -hostfile together with -l or -UF, to view the user-specified host file content as well.

The following are restrictions on the usage of the -hostfile option:

Smart memory limit enforcement

The new parameter LSB_MEMLIMIT_ENF_CONTROL in lsf.conf further refines the behavior of enforcing a job memory limit for a host. In the case that one or more jobs reach a specified memory limit for the host (both the host memory and swap utilization has reached a configurable threshold) at execution time, the worst offending job on the host will be killed. A job is selected as the worst offending job on that host if it has the most overuse of memory (actual memory rusage minus memory limit of the job).

You also have the choice of killing all jobs exceeding the thresholds (not just the worst).

For a description of usage and restrictions on this parameter, see LSB_MEMLIMIT_ENF_CONTROL.

Host-based memory and swap limit enforcement by Linux cgroup

LSF can now impose strict job-level host-based memory and swap limits on systems that support Linux cgroups. When LSB_RESOURCE_ENFORCE="memory" is set, memory and swap limits are calculated and enforced as a multiple of the number of tasks running on the execution host when memory and swap limits are specified for the job (at the job-level with -M and -v, or in lsb.queues or lsb.applications with MEMLIMIT and SWAPLIMIT).

The new bsub -hl option enables job-level (irrespective of the number of tasks) host-based memory and swap limit enforcement regardless of the number of tasks running on the execution host. LSB_RESOURCE_ENFORCE="memory" must be specified in lsf.conf for host-level memory and swap limit enforcement with the -hl option to take effect. If no memory or swap limit is specified for the job (the merged limit for the job, queue, and application profile, if specified), or LSB_RESOURCE_ENFORCE="memory" is not specified, a host-based memory limit is not set for the job. The -hl option only applies to memory and swap limits; it does not apply to any other resource usage limits.

See Administering IBM Platform LSF for more information about memory and swap resource usage limits, and memory enforcement based on Linux cgroup memory subsystem.

Change the default behavior of a job when it reaches the pre-execution retry threshold

When a job’s pre-execution fails, the job will be requeued and tried again. When the pre-exec has failed a defined number of times (LOCAL_MAX_PREEXEC_RETRY in lsb.params, lsb.queues, or lsb.applications) LSF suspends the job and places it in the PSUSP state. If this is a common occurrence, a large number of PSUSP jobs can quickly fill the system, leading to both usability issues and system degradation.

In this release, a pre-execution retry threshold is introduced so that a job exits once the pre-execution has failed a specified number of times. You can setLOCAL_MAX_PREEXEC_RETRY_ACTION cluster-wide in lsb.params, at the queue level in lsb.queues, or at the application level in lsb.applications. The default behavior specified in lsb.applications overrides lsb.queues, and lsb.queues overrides the lsb.params configuration.

Set LOCAL_MAX_PREEXEC_RETRY_ACTION=EXIT to have the job exit and to have LSF sets its status to EXIT. The job exits with the same exit code as the last pre-execution fail exit code.

MultiCluster considers TASKLIMIT on remote clusters before forwarding jobs

In the MultiCluster job forwarding model, the local cluster now considers the application profile or receive queue's TASKLIMIT setting on remote clusters before forwarding jobs. This reduces the number of forwarded jobs that stay pending before returning to the submission cluster due to the remote cluster's TASKLIMIT settings being unable to satisfy the job's task requirements. By considering the TASKLIMIT settings in the remote clusters, jobs are no longer forwarded to remote clusters that cannot run these jobs due to task requirements.

If the receive queue's TASKLIMIT definition in the remote cluster cannot satisfy the job's task requirements, the job is not forwarded to that remote queue. Likewise, if the application profile's TASKLIMIT definition in the remote cluster cannot satisfy the job's task requirements, the job is not forwarded to that cluster.

Enhancements to advance reservation

Control the propagation of job submission environment variables

When using bsub and tssub to submit jobs, you can use the -env option to control the propagation of job submission environment variables to the execution hosts:

Specify a comma-separated list of environment variables. Controls the propagation of the specified job submission environment variables to the execution hosts.

• Specify none to submit jobs that do not propagate any environment variables.
• Specify the variable name without a value to propagate the environment variable using its existing specified value.
• Specify the variable name with a value to propagate the environment variable with the specified value to overwrite the existing specified value. The specified value may either be a new value or quote the value of an existing environment variable. Job packs do not allow you to quote the value of an existing environment variable.

  For example,

  • In UNIX, fullpath=/tmp/:$filename appends /tmp/ to the beginning of the filename environment variable and assigns this new value to the fullpath environment variable. Use a colon (:) to separate multiple environment variables.
  • In Windows, fullpath=\Temp\;%filename% appends \Temp\ to the beginning of the filename environment variable and assigns this new value to the fullpath environment variable. Use a semicolon (;) to separate multiple environment variables.
• Specify all at the beginning of the list to propagate all existing submission environment variables to the execution hosts. You may also assign values to specific environment variables.

  For example, -env "all, var1=value1, var2=value2" submits jobs with all the environment variables, but with the specified values for the var1 and var2 environment variables.

• When using the all keyword, add ~ to the beginning of the variable name to prevent the environment variable from being propagated to the execution hosts.

The environment variable names cannot be "none" or "all".

The environment variable names cannot contain the following symbols: comma (,), "~", "=", double quotation mark (") and single quotation mark (').

The variable value can contain a comma (,) and "~", but if it contains a comma, you must enclose the variable value in single quotation marks.

An esub can change the -env environment variables by writing them to the file specified by the LSB_SUB_MODIFY_FILE environment variable. If the LSB_SUB_MODIFY_ENVFILE environment variable is also specified and the file specified by this environment variable contains the same environment variables, the environment variables in LSF_SUB_MODIFY_FILE take effect.

When -env is not specified with bsub, the default value is -env "all" (that is, all environment variables are submitted with the default values).

The entire argument for the -env option may contain a maximum of 4094 characters for UNIX and Linux, or up to 255 characters for Windows.

If -env conflicts with -L, the value of -L takes effect.

The following environment variables are not propagated to execution hosts because they are only used in the submission host and are not used in the execution hosts:

• HOME, LS_JOBPID, LSB_ACCT_MAP, LSB_EXIT_PRE_ABORT, LSB_EXIT_REQUEUE, LSB_EVENT_ATTRIB, LSB_HOSTS, LSB_INTERACTIVE, LSB_INTERACTIVE_SSH, LSB_INTERACTIVE_TTY, LSB_JOBFILENAME, LSB_JOBGROUP, LSB_JOBID, LSB_JOBNAME, LSB_JOB_STARTER, LSB_QUEUE, LSB_RESTART, LSB_TRAPSIGS, LSB_XJOB_SSH, LSF_VERSION, PWD, USER, VIRTUAL_HOSTNAME, and all variables with starting with LSB_SUB_
• Environment variables about non-interactive jobs. For example: TERM, TERMCAP
• Windows-specific environment variables. For example: COMPUTERNAME, COMSPEC, NTRESKIT, OS2LIBPATH, PROCESSOR_ARCHITECTURE, PROCESSOR_IDENTIFIER, PROCESSOR_LEVEL, PROCESSOR_REVISION, SYSTEMDRIVE, SYSTEMROOT, TEMP, TMP

The following environment variables do not take effect on the execution hosts: LSB_DEFAULTPROJECT, LSB_DEFAULT_JOBGROUP, LSB_TSJOB_ENVNAME, LSB_TSJOB_PASSWD, LSF_DISPLAY_ALL_TSC, LSF_JOB_SECURITY_LABEL, LSB_DEFAULT_USERGROUP, LSB_DEFAULT_RESREQ, LSB_DEFAULTQUEUE, BSUB_CHK_RESREQ, LSB_UNIXGROUP, LSB_JOB_CWD

View job file names with job ID and array index values

When submitting jobs with specified input, output, and error file names (using bsub -i, -is, -o, -oo, -e, and -eo options), you can use the special characters %J and %I in the name of the files. %J is replaced by the job ID. %I is replaced by the index of the job in the array, if the job is a member of an array, or by 0 (zero) if the job is not a member of an array. When viewing job information, bjobs -o, -l, or -UF now replaces %J with the job ID and %I with the array index when displaying job file names. Previously, bjobs -o, -l, or -UF displayed these file names with %J and %I without resolving the job ID and array index values.

Documentation and online help enhancements to bjobs and bsub

The documentation and online help for bjobs and bsub are now reorganized and expanded. The bjobs and bsub command options are grouped into categories, which describes the general goal or function of the command option.

In the IBM Platform LSF Command Reference documentation, the bjobs and bsub sections now list the categories, followed by the options, listed in alphabetical order. Each option lists the categories to which it belongs and includes a detailed synopsis of the command. Any conflicts that the option has with other options are also listed (that is, options that cannot be used together).

The online help in the command line for bjobs and bsub is organized by categories and allows you to view help topics for specific options in addition to viewing the entire man page for the command. To view the online help, run bjobs or bsub with the -h (or -help) option. This provides a brief description of the command and lists the categories and options that belong to the command. To view a brief description of all options, run -h all (or -help all). To view more details on the command, run -h description (or -help description). To view more information on the categories and options (in increasing detail), run -h (or -help) with the name of the category or the option:

bjobs -h[elp] [all] [description] [category_name ...] [-option_name ...]

bsub -h[elp] [all] [description] [category_name ...] [-option_name ...]

If you list multiple categories and options, the online help displays each entry in the order in which you specified the categories and options.

For example,

• To view a brief description of the bjobs command, run bjobs -h. The description includes a list of categories (with a brief description of each category) and the options belonging to each category.
• To view more details of the bjobs command, run bjobs -h description.
• To a brief description of all bjobs options, run bjobs -h all.
• To view a description of the bjobs filter category, run bjobs -h filter. The description includes a list of options with a brief description of each option.
• To view a detailed description of the bjobs -app option, run bjobs -h -app. The description includes the categories to which the option belongs, a detailed synopsis/usage of the option, and any conflicts the option has with other options.

LSF support for Gold v.2.2

Gold v2.2 (or newer) is supported on Linux and UNIX. Complete the steps in LSF_INSTALLDIR/9.1/misc/examples/gold/readme.txt to install and configure the Gold integration in LSF

bacct

• A new termination reason, TERM_ORPHAN_SYSTEM, shows that an orphan job was automatically terminated by LSF.
• When an allocation shrinks, bacct shows

  Release allocation on <num_hosts> Hosts/Processors <host_list> by user or
  administrator <user_name>
  Resize notification accepted;

• Output includes number of slots and host names that a job has been allocated to, based on number of tasks in the job:

  bacct -l -aff 6
  
  Accounting information about jobs that are:
    - submitted by all users.
    - accounted on all projects.
    - completed normally or exited
    - executed on all hosts.
    - submitted to all queues.
    - accounted on all service classes.
  ------------------------------------------------------------------------------
  
  Job <6>, User <user1>, Project <default>, Status <DONE>, Queue <normal>, Comma
                       nd <myjob>
  Thu Feb 14 14:13:46: Submitted from host <hostA>, CWD <$HOME>;
  Thu Feb 14 14:15:07: Dispatched <num_tasks> Task(s) on Host(s) <host_list>,
                       Allocated <num_slots> Slot(s) on Host(s) <host_list>;
                       Effective RES_REQ <select[type == local] order[r15s:pg]
                       rusage[mem=100.00] span[hosts=1] affinity[core(1,same=
                       socket,exclusive=(socket,injob))*1:cpubind=socket:membind
                       =localonly:distribute=pack] >;
  Thu Feb 14 14:16:47: Completed <done>.
  
    

bapp

• Added -alloc option. Shows counters for slots in RUN, SSUSP, and USUSP. The slot allocation will be different depending on whether the job is an exclusive job or not.
• Changes to output. The following fields have changed from a slot-based to a task-based concept: NJOBS, PEND, RUN, SSUSP, USUSP, and RSV.

bhist

• If a job was submitted or modified with a -hostfile option, to point to a user-specified host file, bhist -l will show the show the user-specified host file path. bhist -l -hostfile will also show the user-specified host file contents.
• For JOB_RESIZE_NOTIFY_START event, bhist displays:

  Added <num_tasks> tasks on host <host_list>, <num_slots> additional slots
  allocated on <host_list>

• For JOB_RESIZE_RELEASE event, bhist displays

  Release allocation on <num_hosts> Hosts/Processors <host_list> by user or
  administrator <user_name>
  Resize notification accepted;

• Output includes number of tasks in the job submitted and number of slots and host names that a job has been allocated to, based on number of tasks in the job:

  bhist -l 749
  Job <749>, User <user1>;, Project <default>, Command <my_pe_job>
  
  Mon Jun  4 04:36:12: Submitted from host <hostB>, to Queue <priority>,
                       CWD <$HOME>, 2 Task(s), Requested
                       Network <type=sn_all:protocol=mpi:mode=US:usage=
                       shared:instance=1> 
  
  Mon Jun  4 04:36:15: Dispatched <num_tasks> Task(s) on Host(s) <host_list>,
                       Allocated <num_slots> Slot(s) on Host(s) <host_list>;
                       Effective RES_REQ <select[type == local] rusage[nt1=1.00] >,
                       PE Network ID <1111111>  <2222222> used <1> window(s)
  Mon Jun  4 04:36:17: Starting (Pid 21006);

bhosts

bjdepinfo

• This command can now show if a job dependency condition was not satisfied.

bjobs

• If a job was submitted or modified with a -hostfile option, bjobs -l or bjobs -UF will show the user-specified host file path. bjobs -l -hostfile or bjobs -UF -hostfile will show the user-specified host file contents.
• The bjobs -o option lets you specify the following new bjob fields:
  • immediate_orphan_term indicates that an orphan job was terminated immediately and automatically.
  • host_file displays the user-specified host file path used for a job.
  • user_group (alias ugroup) indicates the user group to which the jobs are associated (submitted with bsub -G for the specified user group).

• Behavior change for bjobs -l: Predicted start time for PEND reserve job will not be shown with bjobs -l. LSF does not calculate predicted start time for PEND reserve job if no back fill queue is configured in the system. In that case, resource reservation for PEND jobs works as normal, and no predicted start time is calculated.

• Behavior change for bjobs -help: Displays the description of the specified category, command option, or sub-option to stdout and exits. You can now abbreviate the -help option to -h.

  Run bjobs -h (or bjobs -help) without a command option or category name to display the bjobs command description.

• The output of bjobs -l has changed:
  • Number of Processors Requested has changed to task concept, displaying number of Tasks.
  • For started jobs, slot allocation is shown after the number of Tasks started on the hosts.
  For example:

  bjobs -l 6
  
  Job <6>, User <user1>, Project <default>, Status <RUN>, Queue <normal>, Comman
                       d <myjob1>
  Thu Feb 14 14:13:46: Submitted from host <hostA>, CWD <$HOME>, 6 Tasks;
  Thu Feb 14 14:15:07: Started 6 Task(s) on Host(s) <hostA> <hostA> <hostA> <hostA>
                       <hostA> <hostA>, Allocated 6 Slots on Hosts <hostA>
                       <hostA> <hostA> <hostA> <hostA> <hostA>, Execution Home 
                       </home/user1>, Execution CWD </home/user1>;
  

bmod

• The -hostfile option allows a user to modify a PEND job with a user-specified host file. A user-specified host file contains specific hosts and slots that a user wants to use for a job:

  bmod -hostfile "host_alloc_file" <job_id>

  To remove a user-specified host file specified for a PEND job, use the -hostfilen option:

  bmod -hostfilen <job_id>

• The -hl option enables per-job host-based memory and swap limit enforcement on hosts that support Linux cgroups. The -hln option disables host-based memory and swap limit enforcement. -hl and -hln only apply to pending jobs. LSB_RESOURCE_ENFORCE="memory" must be specified in lsf.conf for host-based memory and swap limit enforcement with the -hl option to take effect. If no memory or swap limit is specified for the job (the merged limit for the job, queue, and application profile, if specified), or LSB_RESOURCE_ENFORCE="memory" is not specified, a host-based memory limit is not set for the job.
• The -ti option of -w enables immediate automatic orphan job termination at the job level. The -tin option cancels the -ti option of a submitted dependent job, in which case the cluster-level configuration takes precedence.
• The bmod -n command has changed from -n num_processors to -n num_tasks.

bqueues

• Added -alloc option. Shows counters for slots in RUN, SSUSP, and USUSP. The slot allocation will be different depending on whether the job is an exclusive job or not.
• Changes to output. The following fields have changed from a slot-based to a task-based concept: NJOBS, PEND, RUN, SUSP, SSUSP, USUSP, and RSV.

brestart

• The -ti option allows users to indicate that a job is eligible for automatic and immediate termination by the system as soon as the job is found to be an orphan, without waiting for the grace period to expire.

brsvadd

• The new-unit [slot | host] option specifies whether an advance reservation is for a number of slots or hosts. If -unit is not specified for brsvadd, the advance reservation request will use the slot unit by default.

  The following options are required for brsvadd, whether using the slot or host unit:

  • The number of slots or hosts to reserve, using the -n option.
  • The list of candidate hosts, using -m, -R, or both.
  • Users or user groups that have permission to use the advance reservation, using -u.
  • A time period for the reservation, using either -t or -b and -e together.
• The option -n has been changed to specify either job_slots or number_hosts. The number of either job slots or hosts (specified by -unit) to reserve. For a slot-based advance reservation (brsvadd -unit slot), -n specifies the total number of job slots to reserve. For host-based advance reservation brsvadd -unit host, -n specifies the total number of hosts to reserve.
• The option-m has been changed so that the number of slots specified by -n <job_slots> or hosts specified by -n <number_hosts> must be less than or equal to the actual number of hosts specified by -m.
• The -u option has been expanded to include multiple users and user groups, in combination, if desired.
• The -g option will be obsolete after LSF 9.1.3.

brsvmod

• The -u "user_name... | user_group ..." option has been changed so that it replaces the list of users or groups who are able to submit jobs to a reservation.

• The adduser subcommand has been added to add users and user groups to an advance reservation:

  adduser -u "user_name ... | user_group ..."] reservation_ID

• The rmuser subcommand has been added to remove users and user groups from an advance reservation.

  rmuser -u "user_name ... | user_group ..."] reservation_ID

• The option -n has been changed to specify number_unit. This option now changes the number of either job slots or hosts to reserve (based on the unit specified by brsvadd -unit slot | host. number_unit must be less than or equal to the actual number of slots for the hosts selected by -m or -R for the reservation.
• The option-m has been changed to modify the list of hosts for which job slots or number of hosts specified with -n are reserved.
• The -g option will be obsolete after LSF 9.1.3.

brsvs

• The output of brsvs is expanded to show:
  • whether the advance reservation was created with the user (-u) or group (-g) option, as shown under the TYPE heading.
  • the list of users or user groups specified for the advance reservation, under the USER heading.

  • (with the -l option), the Resource Unit (Slot or Host) specified for an advance reservation.

bslots

• Behavior change for bslots: LSF does not calculate predicted start times for PEND reserve jobs if no backfill queue is configured in the systemc. In that case, the resource reservation for PEND jobs works as normal, but no predicted start time is calculated, and bslots does not show the backfill window.

bsub

• The -ti suboption of -w enables automatic orphan job termination at the job level. If configured, the cluster-level orphan job termination grace period is ignored and the job is terminated as soon as it is found to be an orphan. This option is independent of the cluster-level configuration. If the LSF administrator did not enable ORPHAN_JOB_TERM_GRACE_PERIOD at the cluster level, you can still use automatic orphan job termination on a per-job basis.
• The -hostfile option allows a user to submit a job with a user-specified host file. A user-specified host file contains specific hosts and slots that a user wants to use for a job. The user-specified host file specifies the order in which to launch tasks, ranking the slots specified in the file. This command specifies the path of the user-specified host file:

  bsub -hostfile "host_alloc_file" ./a.out

• The -hl option enables job-level host-based memory and swap limit enforcement on systems that support Linux cgroups. When -hl is specified, a memory limit specified at the job level by -M or by MEMLIMIT in lsb.queues or lsb.applications is enforced by the Linux cgroup subsystem on a per-job basis on each host. Similarly, a swap limit specified at the job level by -v or by SWAPLIMIT in lsb.queues or lsb.applications is enforced by the Linux cgroup subsystem on a per-job basis on each host. Host-based memory and swap limits are enforced regardless of the number of tasks running on the execution host. The -hl option only applies to memory and swap limits; it does not apply to any other resource usage limits.

  LSB_RESOURCE_ENFORCE="memory" must be specified in lsf.conf for host-based memory and swap limit enforcement with the -hl option to take effect. If no memory or swap limit is specified for the job (the merged limit for the job, queue, and application profile, if specified), or LSB_RESOURCE_ENFORCE="memory" is not specified, a host-based memory limit is not set for the job.

  When LSB_RESOURCE_ENFORCE="memory" is configured in lsf.conf, and memory and swap limits are specified for the job, but -hl is not specified, memory and swap limits are calculated and enforced as a multiple of the number of tasks running on the execution host.

• The -env option allows you to control the propagation of the specified job submission environment variables to the execution hosts. Specify a comma-separated list of environment variables to propagate to the execution hosts, or add ~ to the beginning of the variable name to block the environment variable from being propagated. The none keyword prevents all environment variables from being propagated, while the all keyword allows all environment variables to be propagated with their default values.
• The bsub -n command has changed from -n min_processors[,max_processors] to -n min_tasks[,max_tasks]

  Submits a parallel job and specifies the number of tasks in the job. The number of tasks is used to allocate a number of slots for the job. Usually, the number of slots assigned to a job will equal the number of tasks specified. For example, one task will be allocated with one slot. (Some slots/processors may be on the same multiprocessor host).

• Behavior change for bsub -help: Displays the description of the specified category, command option, or sub-option to stdout and exits. You can now abbreviate the -help option to -h.

  Run bsub -h (or bsobs -help) without a command option or category name to display the bsub command description.

busers

lsmake

• The option --no-block-shell-mode has been added to allow lsmake to build customized Android 4.3 code. This option allows lsmake to perform child "shell" tasks without blocking mode. Without this parameter, blocking mode is used, making the build for Android 4.3 take a long time.

tssub

• The -env option allows you to control the propagation of the specified job submission environment variables to the execution hosts. Specify a comma-separated list of environment variables to propagate to the execution hosts, or add ~ to the beginning of the variable name to block the environment variable from being propagated. The none keyword prevents all environment variables from being propagated, while the all keyword allows all environment variables to be propagated with their default values.

lsb.applications

lsb.params

lsb.serviceclasses

lsb.queues

lsb.resources

• The PER_SLOT value in the ReservationUsage section has been changed to PER_TASK with the change to a task concept for job resource allocation.

lsf.conf

Environment variables

The following job event fields are added or changed for LSF 9.1.3.

lsb.acct

lsb.events