Upgrading your IBM Platform LSF Cluster

Important: This document describes how to upgrade a cluster running LSF Version 6.x or earlier, and LSF Version 7 Update 2 or later. If you have LSF 7 or LSF 7 Update 1, and you do not have the IBM Platform Management Console installed, follow the steps in the document Migrating IBM Platform LSF Version 7 to IBM Platform LSF Version 9.1.3 on Windows. If you have LSF with the IBM Platform Management Console installed for a version earlier than LSF Version 7 Update 4, contact Product Support for additional information.

Complete the following steps to upgrade to LSF Version 9.1.3 on UNIX and Linux.

  • Download LSF distribution tar files
  • Prepare the cluster for upgrading
  • Use lsfinstall to upgrade LSF
  • Use hostsetup to set up LSF hosts
  • Restart your cluster

Download LSF distribution tar files

Procedure

  1. Log on to the LSF file server host as root.
  2. Select the appropriate LSF installer script package:
    • lsf9.1.3_lsfinstall_linux_x86_64.tar.Z for Linux x86_64 platforms requiring the Linux JRE. Requires approximately 120 MB.
    • lsf9.1.3_lsfinstall.tar.Z for all other platforms requiring the JRE. Requires approximately 1300 MB.
    • lsf9.1.3_no_jre_lsfinstall.tar.Z for all platforms not requiring the JRE. JRE version 1.4 or higher must already be installed on the system. Requires approximately 1 MB.
  3. Download and read Release Notes for IBM Platform LSF (Version 9.1.3) for detailed steps on downloading LSF distribution tar files. Release Notes also describe compatibility issues.
  4. Get the LSF installer script package that you selected and extract it.

    For example,

    • Linux x86_64 platforms: # zcat lsf9.1.3_lsfinstall_linux_x86_64.tar.Z | tar xvf -
    • Other platforms: # zcat lsf9.1.3_lsfinstall.tar.Z | tar xvf -
    • No JRE required: # zcat lsf9.1.3_no_jre_lsfinstall.tar.Z | tar xvf -
  5. Get the LSF distribution packages for all host types you need and put them in the same directory as the extracted LSF installer script.

    For example, for Linux 2.6 kernel glibc version 2.3, the distribution package is lsf9.1.3_linux2.6-glibc2.3-x86_64.tar.Z.

    Important: DO NOT extract the distribution tar files.

Prepare the cluster for upgrading

Procedure

  1. Deactivate all queues to make sure that no new jobs can be dispatched during the upgrade. After upgrading, remember to activate the queues again so pending jobs can be dispatched.
    • Deactivate all LSF queues: badmin qinact all
    • Reactivate all LSF queues after upgrading: badmin qact all
  2. If you have the IBM Platform Management Console (PMC) or IBM Platform Application Center (PAC) installed, shut it down.
    • If PMC or PAC is controlled by EGO, run:

      egosh service stop plc

      egosh service stop purger

      egosh service stop jobdt

      egosh service stop derbydb

      egosh service stop WEBGUI

    • If PMC or PAC is not controlled by EGO, run:

      perfadmin stop all

      pmcadmin stop

  3. Back up your existing LSF_CONFDIR, LSB_CONFDIR, and LSB_SHAREDIR according to the procedures at your site.

Use lsfinstall to upgrade LSF

Procedure

  1. Download LSF 9.1.3 install package.
  2. Go to the downloaded install package and extract it.
  3. Change to directory lsf9.1.3_lsfinstall/.
  4. Open lsf9.1.3_lsfinstall/install.config to set the installation variables you need.
  5. Follow the instructions in lsf_unix_install.pdf to run:

    ./lsfinstall -f install.config

    Important: You must run lsfinstall as root.

Results

lsfinstall backs up the following configuration files for your current installation in LSF_CONFDIR:
  • cshrc.lsf
  • lsf.cluster.cluster_name
  • lsf.conf
  • lsf.shared
  • profile.lsf

Use hostsetup to set up LSF hosts

Procedure

  1. Follow the steps in $LSF_ENVDIR/9.1.3/install/lsf_getting_started.html to set up your LSF hosts (hostsetup).
    1. Log on to each LSF server host as root. Start with the LSF master host.
    2. Run hostsetup on each LSF server host.

      For example:

      cd /usr/share/lsf/9.1.3/install

      ./hostsetup --top="/usr/share/lsf/"

  2. Set your LSF environment:

    For csh or tcsh: source LSF_TOP/conf/cshrc.lsf

    For sh, ksh, or bash: . LSF_TOP/conf/profile.lsf

Restart your cluster

Procedure

  1. Set the original LSF environment:

    For csh or tcsh: source LSF_TOP/conf/cshrc.lsf.<version> for example, source LSF_TOP/conf/cshrc.lsf.8.0

    For sh, ksh, or bash: . LSF_TOP/conf/profile.lsf.<version> for example, source LSF_TOP/conf/profile.lsf.8.0

  2. Use the following commands to shut down the original LSF daemons:

    badmin hshutdown all

    lsadmin resshutdown all

    lsadmin limshutdown all

  3. Set your LSF environment:

    For csh or tcsh: source LSF_TOP/conf/cshrc.lsf

    For sh, ksh, or bash: . LSF_TOP/conf/profile.lsf

  4. Use the following commands to start LSF using the newer daemons:

    lsadmin limstartup all

    lsadmin resstartup all

    badmin hstartup all

  5. Use the following command to reactivate all LSF queues after upgrading: badmin qact all
  6. Follow the steps in $LSF_TOP/9.1.3/lsf_quick_admin.html to verify that your upgraded cluster is operating correctly.

Upgrading LSF HPC to LSF 9.1.3

Before upgrading

  1. Back up your existing LSF_CONFDIR, LSB_CONFDIR, and LSB_SHAREDIR according to the procedures at your site.
  2. Deactivate all queues to make sure that no new jobs can be dispatched during the upgrade:

    badmin qinact all

    For SGI cpuset hosts, make sure all running jobs are done (all queues are drained of running jobs).

    Note: After upgrading, remember to activate the queues again so pending jobs can be dispatched: badmin qact all.

What happens automatically when you upgrade

Configuration file backup

lsfinstall backs up the following configuration files for your current installation in LSF_CONFDIR:
  • cshrc.lsf
  • lsf.cluster.cluster_name
  • lsf.conf
  • lsf.shared
  • profile.lsf

What the installer changes

The LSF 9.1.3 installer makes the following changes:

  • Comments out the PRODUCTS line in lsf.clusters.
    Note: If upgrading a pre-8.3 version cluster, uncomment this line.
  • Comments out LSF_LICENSE_FILE in lsf.conf.
    Note: If upgrading a pre-8.3 version cluster, uncomment this line.
  • Copies the lsf.entitlement file to the LSF_TOP/conf directory.

lsb.queues

  • Configures hpc_ibm queue for IBM POE jobs and the hpc_ibm_tv queue for debugging IBM POE jobs through Etnus TotalView.
  • Configures hpc_linux queue for LAM/MPI and MPICH-GM jobs and hpc_linux_tv queue for debugging LAM/MPI and MPICH-GM jobs through Etnus TotalView.

LSB_SUB_COMMANDNAME (lsf.conf)

If LSB_SUB_COMMANDNAME=N is already defined in lsf.conf, lsfinstall does not change this parameter; you must manually set it to LSB_SUB_COMMANDNAME=Y to enable the LSF_SUB_COMMANDLINE environment variable required by esub.

Reusing install.config from your existing installation

You can reuse the install.config file from your existing installation to specify your installation options. The install.config file containing the options you specified for your original installation is located in LSF_TOP/lsf_version/install/.

Note: If you are upgrading from a pre-8.3 version, this cannot be done without some modification to install.config.

If you change install.config to add new hosts in LSF_ADD_SERVERS and LSF_ADD_CLIENTS, or new LSF administrators in LSF_ADMINS, lsfinstall creates a new lsf.cluster.cluster_name file.

Run lsfinstall to upgrade

Make sure the following install.config variables are set for upgrade:

  • LSF_TARDIR specifies the location of distribution packages for upgrade. For example: LSF_TARDIR=/tmp

To run lsfinstall

  1. Log on to the file server host as root.
  2. Select the appropriate LSF installer script package:
    • lsf9.1.3_lsfinstall_linux_x86_64.tar.Z for Linux x86_64 platforms requiring the Linux JRE. Requires approximately 120 MB.
    • lsf9.1.3_lsfinstall.tar.Z for all other platforms requiring the JRE. Requires approximately 1300 MB.
    • lsf9.1.3_no_jre_lsfinstall.tar.Z for all platforms not requiring the JRE. JRE version 1.4 or higher must already be installed on the system. Requires approximately 1 MB.
  3. Get the LSF installer script package that you selected and extract it to the distribution directory where you downloaded the LSF product distribution tar files.

    For example,

    • Linux x86_64 platforms: # zcat lsf9.1.3_lsfinstall_linux_x86_64.tar.Z | tar xvf -
    • Other platforms: # zcat lsf9.1.3_lsfinstall.tar.Z | tar xvf -
    • No JRE required: # zcat lsf9.1.3_no_jre_lsfinstall.tar.Z | tar xvf -
  4. Change to the directory $LSF_ENVDIR/9.1.3/install/.
  5. Edit $LSF_ENVDIR/9.1.3/install/install.config or $LSF_ENVDIR/9.1.3/install/slave.config and set the installation variables you need.
  6. Run lsfinstall as root:

    # ./lsfinstall -f install.config

Run hostsetup

Running hostsetup is optional on Linux. You must run hostsetup on HP-UX hosts.

What hostsetup does

  • For HP-UX pset hosts, hostsetup adds the pset Boolean resource to the HOSTS section of lsf.cluster.cluster_name for each pset host.

--boot option

Use the --boot="y" option on hostsetup to configure system scripts to automatically start and stop LSF daemons at system startup or shutdown. You must run hostsetup as root to use this option to modify the system scripts. The default is --boot="n".

For complete hostsetup usage, enter hostsetup -h.

To run hostsetup

  1. Log on to each LSF server host as root. Start with the LSF master host.
  2. Run hostsetup on each LSF server host. For example:
    # cd /usr/share/lsf/9.1/install
    # ./hostsetup --top="/usr/share/lsf" --boot="y"

Setting non-privileged ports

For migration and compatibility for a single cluster:

  • If all hosts are upgraded, then they are working with non-privileged ports.
  • If all hosts are upgraded, but you want to use privileged ports for communication, then set LSF_NON_PRIVILEGED_PORTS=N and set a value for LSB_MAX_JOB_DISPATCH_PER_SESSION.
  • If the master host is upgraded, but some server hosts are still running old versions, then set LSF_NON_PRIVILEGED_PORTS=N, shutdown the cluster, then restart it to tell the old lim and sbatchd to use non-privileged ports for communication.
  • When upgrading a mixed cluster that was previously not supported in LSF version 9.1.3 to LSF 9.1.3, install all the new binaries, set the parameter NON_PRIVELIGE_PORTS to Y, then restart the cluster.

For Platform MultiCluster:

  • By default, the LSF 9.1.3 cluster uses privileged ports for communication from mbatchd to mbatchd. If clusters running older versions use non-privileged ports for communication, then LSF_MC_NON_PRIVILEGED_PORTS must be set to the same value for both clusters.

After upgrading

  1. Log on to the LSF master host as root.
  2. Set your environment:
    • For csh or tcsh:

      % source /LSF_TOP/conf/cshrc.lsf

    • For sh, ksh, or bash:

      # . /LSF_TOP/conf/profile.lsf

  3. Use the following commands to shut down the old LSF daemons:

    # badmin hshutdown all

    # lsadmin resshutdown all

    # lsadmin limshutdown all

  4. Use the following commands to start LSF using the upgraded daemons:

    # lsadmin limstartup all

    # lsadmin resstartup all

    # badmin hstartup all

  5. Test your cluster by running some basic LSF commands (for example, lsid, lshosts, bhosts).
  6. Use the following command to reactivate all LSF queues after upgrading:

    # badmin qact all

  7. Have users run one of the shell environment files to switch their environment to the new cluster.

After your cluster is up and running, users can start submitting jobs to it.