Resource preemption steps

About this task

To make resource preemption useful, you may need to work through all of these steps.

Procedure

  1. Read.
    Before you set up resource preemption, you should understand the following:

    • Preemptive Scheduling

    • Resource Preemption

    • Resource Reservation

    • Customizing Resources

    • Customizing Job Controls

  2. Plan.
    When you plan how to set up resource preemption, consider:

    • Custom job controls: Find out what signals or actions you can use with your application to control the preemption resource when you suspend and resume jobs.

    • Existing cluster configuration: Your design might be based on preemptive queues or custom resources that are already configured in your cluster.

    • Requirements for resource preemption: Your design must be able to work. If a host-based resource such as memory is the preemption resource, you cannot set up only one queue for each host because preemption occurs when 2 jobs are competing for the same resource.

  3. Write the ELIM.
  4. Configure LSF.
    1. lsb.queues

      • Set PREEMPTION in at least one queue (to PREEMPTIVE in a high-priority queue, or to PREEMPTABLE in a low-priority queue).

      • Set JOB_CONTROLS (or TERMINATE_WHEN) in the low-priority queues. Optional. Set RES_REQ to automatically reserve the custom resource.

    2. lsf.shared

      Define the custom resource in the Resource section.

    3. lsb.params

      • Set PREEMPTABLE_RESOURCES and specify the custom resource.

      • Optional. Set PREEMPTION_WAIT_TIME to specify how many seconds to wait for dynamic resources to become available.

      • Optional. Set PREEMPT_JOBTYPE to enable preemption of exclusive and backfill jobs. Specify one or both of the keywords EXCLUSIVE and BACKFILL. By default, exclusive and backfill jobs are only preempted if the exclusive low priority job is running on a host that is different than the one used by the preemptive high priority job.

    4. lsf.cluster.cluster_name

      Define how the custom resource is shared in the ResourceMap section.

    5. lsf.task.cluster_name

      Optional. Configure the RemoteTasks section to automatically reserve the custom resource.

  5. Reconfigure LSF to make your changes take effect.
  6. Operate.

    • Use resource reservation to reserve the preemption resource (this might be configured to occur automatically). For dynamic resources, you must specify a duration as well as a quantity.

    • Distribute jobs that use the preemption resource in way that allows preemption to occur between queues (this should happen as a result of the cluster design).

  7. Track.

    Use bparams -l to view information about preemption configuration in your cluster.

Parent topic: Resource Preemption