About IBM Platform Session Scheduler

While traditional LSF job submission, scheduling, and dispatch methods such as job arrays or job chunking are well suited to a mix of long and short running jobs, or jobs with dependencies on each other, Session Scheduler is ideal for large volumes of independent jobs with short run times.

As clusters grow and the volume of workload increases, the need to delegate scheduling decisions increases. Session Scheduler improves throughput and performance of the LSF scheduler by enabling multiple tasks to be submitted as a single LSF job.

Session Scheduler implements a hierarchal, personal scheduling paradigm that provides very low-latency execution. With very low latency per job, Session Scheduler is ideal for executing very short jobs, whether they are a list of tasks, or job arrays with parametric execution.

The Session Scheduler provides users with the ability to run large collections of short duration tasks within the allocation of an LSF job using a job-level task scheduler that allocate resources for the job once, and reuses the allocated resources for each task.

Each Session Scheduler is dynamically scheduled in a similar manner to a parallel job. Each instance of the ssched command then manages its own workload within its assigned allocation. Work is submitted as a task array or a task definition file.

Session Scheduler satisfies the following goals for running a large volume of short jobs:
  • Minimize the latency when scheduling short jobs

  • Improve overall cluster utilization and system performance

  • Allocate resources according to LSF policies

  • Support existing LSF pre-execution, post-execution programs, job starters, resources limits, etc.

  • Handle thousands of users and more than 50000 short jobs per user

Session Scheduler system requirements

Supported operating systems
Session Scheduler is delivered in the following distribution:
  • lsf9.1.3_ssched_lnx26-libc23-x64.tar.Z

Required libraries

Note: These libraries may not be installed by default by all Linux distributions.

On Linux 2.6 (x86_64), the following external libraries are required:
  • libstdc++.so.5

  • libpthread-2.3.4.so or later

Compatible Linux distributions
Certified compatible distributions include:
  • Red Hat Enterprise Linux AS 3 or later

  • SUSE Linux Enterprise Server 10

Platform LSF

Session Scheduler is included with Platform LSF Advanced Edition and is available as an add-on for other editions of Platform LSF:

  • If you are using Platform LSF Advanced Edition, download the Session Scheduler distribution package from the same download page as the Platform LSF Advanced Edition distribution packages.
  • If you are using other editions of Platform LSF, purchase Session Scheduler as a separate add-on, then download the distribution package from the Session Scheduler download page.

Session Scheduler terminology

Job

A traditional LSF job that is individually scheduled and dispatched to sbatchd by mbatchd and mbschd

Task

Similar to a job, a unit of workload that describes an executable and its environment that runs on an execution node. Tasks are managed and dispatched by the Session Scheduler.

Job Session

An LSF job that is individually scheduled by mbatchd, but is not dispatched as an LSF job. Instead, a running Session Scheduler job session represents an allocation of nodes for running large collections of tasks

Scheduler

The component that accepts and dispatches tasks within the nodes allocated for a job session.

Session Scheduler architecture

Session Scheduler jobs are submitted, scheduled, and dispatched like normal LSF jobs.

When the Session Scheduler begins running, it starts a Session Scheduler execution agent on each host in its allocation.

The Session Scheduler then reads in the task definition file, which contains a list of tasks to run. Tasks are sent to an execution agent and run. When a task finishes, the next task in the list is dispatched to the available host. This continues until all tasks have been run.

Tasks submitted through Session Scheduler bypass the LSF mbatchd and mbschd. The LSF mbatchd is unaware of individual tasks.

Session Scheduler components

Session Scheduler comprises the following components.

Session Scheduler command (ssched)

The ssched command accepts and dispatches tasks within the nodes allocated for a job session. It reads the task definition file and sends tasks to the execution agents. ssched also logs errors, performs task accounting, and requeues tasks as necessary.

sservice and sschild

These components are the execution agents. They run on each remote host in the allocation. They set up the task execution environment, run the tasks, and enable task monitoring and resource usage collection.

Session Scheduler performance

Session Scheduler has been tested to support up to 50,000 tasks. Based on performance tests, the best maximum allocation size (specified by bsub -n) depends on the average runtime of the tasks. Here are some typical results:

Average Runtime (seconds)

Recommended maximum allocation size (slots)

0

12

5

64

15

256

30

512

Install Session Scheduler

About this task

There are two ways of installing Session Scheduler.

Procedure

  • Install Session Scheduler and LSF together
    1. Copy the Session Scheduler distribution file into the same location as the LSF distribution files.
    2. Edit the install.config file.
    3. Set LSF_TARDIR to the location where you put the Session Scheduler and LSF distribution files and save your changes.
    4. Run lsfinstall -f install.config to install LSF and Session Scheduler together.

      When asked if you want to install Session Scheduler, follow the prompts to install it.

  • Install Session Scheduler after LSF is already installed
    1. Edit the install.config file.
    2. Set LSF_TARDIR to the location where you put the Session Scheduler distribution file and save your changes.
    3. Run lsfinstall -f install.config to install Session Scheduler.

      When asked if you want to install Session Scheduler, follow the prompts to install it. You can also use the unattended install for Session Scheduler.

Results

The unattended install is supported for Session Scheduler.