Setup

The following environment variables assume that the management server is named HostM and the license file is located in /pcc/software/license/pcmae_entitlement.dat:

export MASTERHOST=HostM
export LICENSEFILE=/pcc/software/license/pcmae_entitlement.dat
export CLUSTERNAME=PCMAE_DC
export CLUSTERADMIN=admin
export BASEPORT=15937

If you want to use management server failover, set the SHAREDDIR environment variable with the file path to the shared directory. For example:

export SHAREDDIR=/usr/share/platform

To configure manager server failover for Platform Cluster Manager, refer to Configuring master managment server failover in the Administering IBM Platform Cluster Manager Advanced Edition guide.

./pcmae_3.2.0.2_mgr_linux2.6-x86_64.bin

When the Platform Cluster Manager installer prompts you to install the provisioning engine on the following prompt, select no:

Do you want to install the provisioning engine on the same host as your management server?(yes/no)

By default, the Platform Cluster Manager installer uses the following parameter values:

• Username: isf
• Password: isf
• Port: 1521
• Service name: XE
• The installer will create /etc/init.d/ego.

• csh or tcsh: source /opt/platform/cshrc.platform

• sh, ksh, or bash: . /opt/platform/profile.platform

egosh ego start

1. Use yum to install the required virtualization files.

   yum groupinstall Virtualization*

2. Ensure that VT-x or AMD-V is enabled in the BIOS.
3. Ensure that the libvirtd service is enabled and running.
4. Stop and disable the libvirt-guests service.

   chkconfig libvirt-guests off

   service libvirt-guests stop

5. Configure a network bridge to give the virtual machines direct access to the network.

   For more details, refer to Bridged network with libvirt in the RHEL product documentation: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/sect-Virtualization_Host_Configuration_and_Guest_Installation_Guide-Network_Configuration-Network_Configuration-Bridged_networking_with_libvirt.html.

   Note: The RHEL KVM supports two methods of connecting virtual machines to the physical network: software bridge and MacVTap. Use the software bridge because MacVTap has performance issues with Windows guest operating systems.
   

The following environment variables assume that the management server is named HostM:

export MASTERHOST=HostM
export CLUSTERNAME=PCMAE_DC
export CLUSTERADMIN=admin
export BASEPORT=15937

./pcmae_3.2.0.2_agent_linux2.6-x86_64.bin

• csh or tcsh: source /opt/platform/cshrc.platform

• sh, ksh, or bash: . /opt/platform/profile.platform

egosh ego start

• C shell: source /opt/platform/cshrc.platform
• Bourne shell: . /opt/platform/profile.platform

egosh user logon -u Admin -x Admin

The default password (for -x) is Admin.

For example,

$ cat SAMPLE_INPUT_FILE 
172.27.101.184 myip101184.lsf.example.com 255.255.0.0 172.27.232.2
172.27.101.185 myip101185.lsf.example.com 255.255.0.0 172.27.232.2

vsh ips add -f SAMPLE_INPUT_FILE

You can view the list of available IP addresses in the Portal for Platform Cluster Manager by logging into the Portal as an administrator and navigating to IP Pool > Dynamic Cluster IP Pool in the Resources tab.

You can also view the list of available IP addresses from the command line using the following command:

vsh ips list

• For KVM hypervisors, refer to the RedHat Virtualization Host Configuration and Guest Installation Guide:

  https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/

  Note:

  • When selecting storage for the VM, select the Select managed or other existing storage option and create a new volume in the default storage pool. The volume must use the qcow2 format.

    Despite the RedHat Virtualization Host Configuration and Guest Installation Guide stating that you must install VMs on shared network storage to support live and offline migrations, you still need to create the VM on a local qcow2 disk. Dynamic Cluster copies the VM image onto shared storage to manage when the VM is converted to a template.

  • If you intend to migrate virtual machines with running jobs from one hypervisor host to another (live migration of VMs), select the Cache mode as none (in Advanced options) when creating a VM.

• For VMware hypervisors, use the VMware vSphere Client.

The installer is on your Platform Cluster Manager management server in the folder: /opt/platform/virtualization/4.1.

• If you are making a template for a Linux guest operating system, extract and run the VMTools installer package that you copied to your VM.
• If you are making a template for a Windows guest operating system:
  1. Log into the VM as the Windows OS Administrator account.
  2. For VMware hypervisors, add the C:\Program Files\VMware\VMware Tools directory to your system PATH environment variable.

     Click Start, right-click Computer, then select Advanced System Settings > Environment Variables. Select Path and click Edit.

  3. Extract the VMTools installer package that you copied to your VM.
  4. Open a command line window and run the cscript install.vbs script file to install VMTools.

• If you are making a template for a Linux guest operating system:

  1. Follow the steps in the LSF installation guide for Linux.

     Make sure to specify the following parameter in the slave.config configuration file:

     LSF_LOCAL_RESOURCES="[resource jobvm]"

  2. Run the hostsetup and chkconfig commands to configure the host to not boot LSF when the VM launches:

     # hostsetup --boot="n"
     # chkconfig lsf off

• If you are making a template for a Windows guest operating system:

  1. Follow the steps in the LSF installation guide for Windows.
  2. Edit the lsf.conf file and specify the LSF_USER_DOMAIN parameter in the lsf.conf file.
     • If all Windows accounts are local accounts, use "." as the domain:

       LSF_USER_DOMAIN="."

     • If the VM will join a Windows domain, specify the domain name:

       LSF_USER_DOMAIN="domain_name"

  3. Edit the lsf.conf file and specify the LSF_LOCAL_RESOURCES parameter:

     LSF_LOCAL_RESOURCES="[resource jobvm]"

  4. Create a Windows OS account with the same name as the LSF cluster administrator (for example, lsfadmin).
     Note: The same Windows user account in all templates must use the same password.

• For KVM hypervisors, Platform Cluster Manager manages storage repositories. Use the Platform Cluster Manager Portal to add a storage repository:

  Click the Resources tab, then navigate to Inventory > KVM > Storage List and click Add Storage.

• For VMware hypervisors, VMware vCenter manages storage repositories. In a Dynamic Cluster-enabled LSF cluster, all VMs could be started on any hypervisor host, so the VM datastore must make all VMs disks available to all hypervisors.

For more details on storage repositories, refer to Managing storage repositories in the Platform Cluster Manager Administration Guide.

For VMware, Platform Cluster Manager supports two types of templates: standard templates and snapshot templates. With a standard template, the full VM disk must be copied before a new VM can be powered on. Depending on the size of the disk and the performance of your storage infrastructure, a full copy of a VM disk can take minutes or hours to complete. A snapshot template uses copy on write to instantly clone the VM disk from the template. Local disk intensive applications will have better performance with a standard template.

• Create a KVM standard template:
  1. Log into the Platform Cluster Manager Portal as an administrator.
  2. Click the Resources tab, then navigate to Inventory > KVM in the navigation tree.
  3. Click the Machines tab in the main window.
  4. Select your VM from the list.
  5. Click Manage and select Convert to Template.
• Create a VMware standard template:
  1. Log into the Platform Cluster Manager Portal as an administrator.
  2. Click the Resources tab, then navigate to Inventory > VMware > vCenter_host_name in the navigation tree.
  3. Click the Machines tab in the main window.
  4. Select your VM from the list.
  5. Click Manage and select Convert to Template.
• Create a VMware snapshot template:
  1. In the VMware vSphere Client, create a snapshot of your VM.
  2. Log into the Platform Cluster Manager Portal as an administrator.
  3. Click the Resources tab, then navigate to Inventory > VMware in the navigation tree.
  4. Select your vCenter Server in the navigation tree.
  5. Click the VM Snapshots tab in the main window.
  6. Select your VM snapshot from the list.
  7. Click Set as template.

For more details, refer to Create a post-provisioning script.

• csh or tcsh: source /opt/lsf/conf/cshrc.platform
• sh, ksh, or bash: ./opt/lsf/conf/profile.platform

Use the same domain name as the one you specified in the LSF_USER_DOMAIN parameter.

For example, use ".\lsfadmin" for local accounts or "domain_name\lsfadmin" for a Windows domain.

For installation instructions, extract the package and refer to the README file in the package.

A template for dc_conf.lsf_cluster_name.xml is provided for easy setup.

1. Copy the template file TMPL.dc_conf.CLUSTER_NAME.xml from /opt/lsf/9.1/misc/conf_tmpl/ into $LSF_ENVDIR .
2. Change the <Templates> section in the file to match the templates that you created and make other required changes for your configuration.
3. If you set the SHAREDDIR environment variable when installing the Platform Cluster Manager management server (to enable management server failover), change the file path in the DC_CONNECT_STRING parameter from the default /opt/platform to the value that you set for the SHAREDDIR environment variable.

   For example, if you set SHAREDDIR to /usr/share/platform by running the following command:

   export SHAREDDIR=/usr/share/platform

   Navigate to the DC_CONNECT_STRING parameter and change the file path as follows:

           <Parameter name="DC_CONNECT_STRING">
               <Value>Admin::/usr/share/platform</Value>
           </Parameter>

   
4. Rename the file to dc_conf.lsf_cluster_name.xml.

For example, for KVM hosts:

# cat /opt/lsf/conf/dc_conf.lsf_cluster_name.xml
<?xml version="1.0" encoding="UTF-8"?>
<dc_conf xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <ParametersConf>
        <Parameter name="DC_VM_MEMSIZE_DEFINED">
            <memsize>8192</memsize>
            <memsize>4096</memsize>
            <memsize>2048</memsize>
            <memsize>1024</memsize>
        </Parameter>
        <Parameter name="DC_VM_RESOURCE_GROUPS">
            <Value>KVMRedHat_Hosts</Value>
        </Parameter>
        <Parameter name="DC_CONNECT_STRING">
            <Value>Admin::/opt/platform</Value>
        </Parameter>
    </ParametersConf>
    <Templates>
        <Template>
            <Name>RH_VM_TMPL</Name>
            <PCMAE_TemplateName>isf_rhel56vm_tmpl</PCMAE_TemplateName>
            <PostProvisioningScriptName>start-lsf.sh</PostProvisioningScriptName>
            <PostProvisioningScriptArguments/>
            <Description>RHEL 5.6 Virtual Machine</Description>
            <RES_ATTRIBUTES>
                <bigmem />
                <hostType>linux</hostType>
                <OSTYPE>linux2.6-glibc2.4</OSTYPE>
            </RES_ATTRIBUTES>
        </Template>
        <Template>
            <Name>RH_KVM</Name>
        </Template>
    </Templates>
    <ResourceGroupConf>
        <HypervisorResGrps>
            <ResourceGroup>
                <Name>KVMRedHat_Hosts</Name>
                <Template>RH_KVM</Template>
                <MembersAreAlsoPhysicalHosts>Yes</MembersAreAlsoPhysicalHosts>
            </ResourceGroup>
        </HypervisorResGrps>
    </ResourceGroupConf>
</dc_conf>

For VMware hosts, make the following changes to the previous example file:

1. Change the ParametersConf node:

   Change the DC_VM_RESOURCE_GROUPS parameter to match the name of your VMware resource group.

   To determine the name of the VMware resource group, use the Platform Cluster Manager Portal:

   1. Log into the Platform Cluster Manager Portal as an administrator.
   2. Click the Resources tab, then navigate to Inventory > VMware in the navigation tree.
   3. Click the Resource Group tab in the main window.
   4. Locate the name of your resource group in the Resource Group Name column of the table.
2. Change the ResourceGroupConf node:
   1. Change the name of the hypervisor resource group to the same resource group name used in the ParametersConf node.

      The hypervisor resource group is found in the following XML path: HypervisorResGrps\ResourceGroup\Name.

   2. Delete the HypervisorResGrps\ResourceGroup\Template node.
   3. Change the value of the MembersAreAlsoPhysicalHosts parameter to No.

If you set the SHAREDDIR environment variable when installing the Platform Cluster Manager managment server, change the file path in the

The DC_VM_MEMSIZE_DEFINED parameter specifies the possible virtual machine memory sizes that will be created. Jobs that specify resource requirements that fall between these values will run in VMs whose memory allocations are rounded up to the next highest value in this list.

The <ResourceGroupConf> section defines the resource groups.

See the Reference section of this guide for details about other dc_conf.lsf_cluster_name.xml parameters.

# cat /opt/lsf/conf/lsf.cluster.lsf_cluster_name
Begin Host
HOSTNAME  model    type        server r1m  mem  swp  RESOURCES    #Keywords
#apple    Sparc5S  SUNSOL       1     3.5  1    2   (sparc bsd)   
#Example
...
<Master>  !        !            1     3.5  ()   ()  (mg)
<Host1>   !        !            1     3.5  ()   ()  (dchost)
<Host2>   !        !            1     3.5  ()   ()  (dchost)
...
<HostN>   !        !            1     3.5  ()   ()  (dchost)
End Host

The default value in LSF is 5 minutes, but this is too short and can cause performance problems with Dynamic Cluster.

# cat /opt/lsf/conf/lsbatch/lsf_cluster_name/configdir/lsb.params
...
Begin Parameters
DEFAULT_QUEUE  = normal   #default job queue name
MBD_SLEEP_TIME = 20       #mbatchd scheduling interval (60 secs is default)
SBD_SLEEP_TIME = 15       #sbatchd scheduling interval (30 secs is default)
JOB_ACCEPT_INTERVAL = 1   #interval for any host to accept a job 
                          # (default is 1 (one-fold of MBD_SLEEP_TIME))
ENABLE_EVENT_STREAM = n   #disable streaming of lsbatch system events
PREEMPTION_WAIT_TIME=1800 #at least 30 minutes
End Parameters

If you do not want to use hyperthreading, disable hyperthreading in the BIOS of your hypervisor hosts and reboot the hosts.