Portable hardware locality

Portable Hardware Locality (hwloc) is an open source software package that is distributed under BSD license. It provides a portable abstraction (across OS, versions, architectures, etc.) of the hierarchical topology of modern architectures, including NUMA memory nodes, socket, shared caches, cores, and simultaneous multi-threading (SMT). It also gathers various system attributes such as cache and memory information as well as the locality of I/O device such as network interfaces. It primarily aims at helping applications with gathering information about computing hardware. Hwloc can support most platforms LSF supports.

Functionality

Hwloc is integrated into LSF to detect hardware information. It detects each host hardware topology when the LIM starts and the host topology information is changed. The master LIM detects the topology of the master host. The slave LIM detects the topology of the local host. It updates the topology information to the master host when it joins the cluster or sends topology information to the master LIM for host configuration. Host topology information is updated once the hardware topology changes. Hardware topology changes if any NUMA memory node, caches, socket, core, PU, etc., changes. Sometimes topology information changes even though the core number did not change.

The commands lim –T and lshosts –T display host topology information. lim –t displays the total number of NUMA nodes, total number of processors, total number of cores, and total number of threads.

Structure of topology

A NUMA node contains sockets. Each socket contains cores (processes) which contain threads. If there is no hwloc library, LSF uses the PCT logic. Some AMD CPUs have the opposite structure where socket nodes contain NUMA nodes. The hierarchies of the topology is similar to a tree. Therefore, the host topology information (NUMA memory nodes, caches, sockets, cores, pus, etc.) from hwloc is organized as a tree. Each tree node has its type. The type includes host, NUMA, socket, cache, core, and pu. Each tree node also includes its attributes.

In the following example, hostA (with two Intel Xeon E5-2670 CPUs) has 64 GB of memory and two NUMA nodes. Each NUMA node has one socket, eight cores, 16 PUs (two PUs per core), and 32 GB of memory. Both the NUMA nodes and the PUs are numbered in series that is provided by the system; LSF displays NUMA information based on the level it detects from the system. The output format displays as a tree, and the NUMA information displays as NUMA[ID: memory]. The PU displays as parent_node(ID ID ...), where parent_node may be host, NUMA, socket, or core. 

Host[64G] hostA 
NUMA[0: 32G]
 Socket
   core(0 16)
   core(1 17)
   core(2 18)
   core(3 19)
   core(4 20)
   core(5 21)
   core(6 22)
   core(7 23)
NUMA[1: 32G]
 Socket
   core(8 24)
   core(9 25)
   core(10 26)
   core(11 27)
   core(12 28)
   core(13 29)
   core(14 30)
   core(15 31)

In the previous example, NUMA[0: 32G] means that the NUMA ID is 0 and has 32 GB of memory. core(0 16) means that there are two PUs under the parent core node, and the ID of the two PUs are 0 and 16.

Some CPUs, especially old ones, may have incomplete hardware topology in terms of missing information for NUMA, socket, or core. Therefore, their topology is incomplete.

For example,

  • hostB (with one Intel Pentium 4 CPU) has 2G of memory, one socket, one core, and two PUs per core. Information on hostB may display as follows:
    Host[2G] hostB 
Socket
      core(0 1)
  • hostC (with one Intel Itanium CPU) has 4 GB of memory, and two PUs. Information on hostC may display as follows:
    Host[4G] (0 1) hostC

Some platforms or operating system versions will only report a subset of topology information.

For example, hostD has the same CPU as hostB, but hostD is running RedHat Linux 4, which does not supply core information. Therefore, information on hostD may display as follows:

Host[1009M] hostD
Socket (0 1)
Parent topic: LSF Resources