Optimize performance in large sites

As your site grows, you must tune your LSF cluster to support a large number of hosts and an increased workload.

This chapter discusses how to efficiently tune querying, scheduling, and event logging in a large cluster that scales to 6000 hosts and 500,000 pending jobs at any one time.

LSF performance enhancement features

LSF provides parameters for tuning your cluster, which you will learn about in this chapter. However, before you calculate the values to use for tuning your cluster, consider the following enhancements to the general performance of LSF daemons, job dispatching, and event replaying:

  • Both scheduling and querying are much faster

  • Switching and replaying the events log file, lsb.events, is much faster. The length of the events file no longer impacts performance

  • Restarting and reconfiguring your cluster is much faster

  • Job submission time is constant. It does not matter how many jobs are in the system. The submission time does not vary.

  • The scalability of load updates from the slaves to the master has increased

  • Load update intervals are scaled automatically

Parent topic: Achieving Performance and Scalability