About external authentication (eauth)

The external authentication feature uses an executable file called eauth. You can write an eauth executable that authenticates users, hosts, and daemons using a site-specific authentication method such as Kerberos or DCE Security Services client authentication. You can also specify an external encryption key (recommended) and the user account under which eauth runs.

Important:

LSF uses an internal encryption key by default. To increase security, configure an external encryption key by defining the parameter LSF_EAUTH_KEY in lsf.sudoers.

During LSF installation, a default eauth executable is installed in the directory specified by the parameter LSF_SERVERDIR (set by cshrc.lsf and profile.lsf). The default executable provides an example of how the eauth protocol works. You should write your own eauth executable to meet the security requirements of your cluster.

Figure 1. Default behavior (eauth executable provided with LSF)

The eauth executable uses corresponding processes eauth -c host_name (client) and eauth -s (server) to provide a secure data exchange between LSF daemons on client and server hosts. The variable host_name refers to the host on which eauth -s runs; that is, the host called by the command. For bsub, for example, the host_name is NULL, which means the authentication data works for any host in the cluster.

Figure 2. How eauth works

One eauth -s process can handle multiple authentication requests. If eauth -s terminates, the LSF daemon invokes another instance of eauth -s to handle new authentication requests.

The standard input stream to eauth -s is a text string with the following format:

uid gid user_name client_addr client_port user_auth_data_len eauth_client 
eauth_server aux_data_file aux_data_status user_auth_data

where

The variable …	Represents the …
`uid`	User ID of the client user
`gid`	Group ID of the client user
`user_name`	User name of the client user
`client_addr`	IP address of the client host
`client_port`	Port number from which the client request originates
`user_auth_data_len`	Length of the external authentication data passed from the client host
`eauth_client`	Daemon or user that invokes eauth -c
`eauth_server`	Daemon that invokes eauth -s
`aux_data_file`	Location of the temporary file that stores encrypted authentication data
`aux_data_status`	File in which eauth -s stores authentication status. When used with Kerberos authentication, eauth -s writes the source of authentication to this file if authentication fails. For example, if mbatchd to mbatchd authentication fails, eauth -s writes "mbatchd" to the file defined by `aux_data_status`. If user to mbatchd authentication fails, eauth -s writes "user" to the `aux_data_status` file.
`user_auth_data`	External authentication data passed from the client host

The variables required for the eauth executable depend on how you implement external authentication at your site. For eauth parsing, unused variables are marked by '''.

User credentials

When an LSF user submits a job or issues a command, the LSF daemon that receives the request verifies the identity of the user by checking the user credentials. External authentication provides the greatest security of all LSF authentication methods because the user credentials are obtained from an external source, such as a database, and then encrypted prior to transmission. For Windows hosts, external authentication is the only truly secure type of LSF authentication.

Host credentials

LSF first authenticates users and then checks host credentials. LSF accepts requests sent from all hosts configured as part of the LSF cluster, including floating clients and any hosts that are dynamically added to the cluster. LSF rejects requests sent from a non-LSF host. If your cluster requires additional host authentication, you can write an eauth executable that verifies both user and host credentials.

Daemon credentials

Daemon authentication provides a secure channel for passing credentials between hosts, mediated by the master host. The master host mediates authentication by means of the eauth executable, which ensures secure passing of credentials between submission hosts and execution hosts, even though the submission host does not know which execution host will be selected to run a job.

Daemon authentication applies to the following communications between LSF daemons:

mbatchd requests to sbatchd
sbatchd updates to mbatchd
PAM interactions with res
mbatchd to mbatchd (in a MultiCluster environment)

Kerberos authentication

Kerberos authentication is an extension of external daemon authentication, providing authentication of LSF users and daemons during client-server interactions. The eauth.krb executable is provided in the installation package under $LSF_SERVERDIR and uses Kerberos Version 5 APIs for interactions between mbatchd and sbatchd, and between pam and res. When you use Kerberos authentication for a cluster or MultiCluster, authentication data is encrypted along the entire path from job submission through to job completion.

You can also use Kerberos authentication for delegation of rights (forwarding credentials) when a job requires a Kerberos ticket during job execution. LSF ensures that a ticket-granting ticket (TGT) can be forwarded securely to the execution host. LSF also automatically renews Kerberos credentials.

Kerberos Integration

Kerberos integration for LSF includes the following features:

The dedicated binary krbrenewd renews TGTs for pending jobs and running jobs. It is enhanced to handle a large number of jobs without creating too much overhead for mbatchd and KDC.
Separate user TGT forwarding from daemon and user authentication with a parameter, LSB_KRB_TGT_FWD, to control TGT forwarding.
Kerberos solution package is preinstalled in the LSF install directory, relieving users from compiling from source code. krb5 function calls are dynamically linked.
Preliminary TGT forwarding support for parallel jobs, including shared directory support for parallel jobs. If all hosts at a customer site have a shared directory, you can configure this directory in lsf.conf via parameter LSB_KRB_TGT_DIR, and the TGT for each individual job will be stored here.
LSF Kerberos integration works in a NFSv4 environment.

Install LSF in a location that does not require a credential to access.

You must provide the following krb5 libraries since they do not ship with LSF:

libkrb5.so
libkrb5support.so
libk5crypto.so
libcom_err.so

Set LSB_KRB_LIB_PATH in lsf.conf to the path that contains these four libraries.

When using Kerberos integration, note the following:

If you turn on the account mapping feature of LSF, you must ensure that the execution user has read/write permission for the directory defined by LSB_KRB_TGT_DIR which holds the run time TGT
krb5 libs are required for TGT manipulation.
TGT renew limit should be configured so it is long enough for jobs to finish running. Long jobs which last several hours or even several days need their TGTs renewed in time to keep the job running. Ensure the job execution time does not exceed the TGT renew limit.
With blaunch, only one task res is invoked per host.
blaunch krb5 does not support auto-resizable jobs.
blaunch krb5 does not support pre LSF 9.1.2 remote execution server, and therefore the renew script will not work in pre 9.1.2 RES. Similarly, blaunch krb5 does not support pre LSF 9.1.2 sbatchd. Therefore, child sbatchds cannot be kerberized and the renew script does not work in pre 9.1.2 root sbatchd.
brequeue does not transfer new TGTs to mbatchd. If a job is re-queued by the brequeue command, the TGT job used is the one that is cached by mbatchd.
LSF does not check the contents or exit code of the erenew script. If erenew contains the wrong command, AFS tokens may not be renewed and LSF will not report any error in the log file. Therefore, users must ensure that the commands in erenew can renew AFS tokens successfully.
Some bsub arguments, such as bsub -Zs, bsub –is, etc., require the bsub command to do file manipulation. In this case, if the file involved resides in the AFS volume, users must ensure they acquire a proper AFS token before they run bsub.

Kerberos Support for NFSv4 and AFS

When using LSF on NFSv4 or Andrew File System (AFS), each process in a sequential job or a distributed parallel job needs to periodically renew its credentials. For this re-authentication to take place in a secure, user friendly environment, a TGT file is distributed to each execution host and the root sbatchd in each execution host renews the TGT.

Start of change If you use the AFS feature, you must provide libs libkopenafs.so or libkopenafs.so.1 which do not ship with LSF. You can use them from the openafs-authlibs-* package or build them directly from the AFS source. End of change

To support AFS, LSF provides an external renew hook mechanism which is called after TGT is renewed. Users can write their own renew logic through this renew hook. More specifically, users can use the demo script named erenew.krb5 in $LSF_SERVERDIR and rename it to erenew. Users can also create an executable named erenew in $LSF_SERVERDIR. This erenew script will be called immediately at job startup time to make sure the user’s job has a valid AFS token. LSF will also automatically call this binary after TGT is renewed. For example, AFS users can use this hook to run aklog for renewing AFS tokens.

Scope

Applicability	Details
Operating system	UNIX
Allows for	Authentication of LSF users, hosts, and daemons Authentication of any number of LSF users
Not required for	Authorization of users based on account permissions
Dependencies	UNIX user accounts must be valid on all hosts in the cluster, or the correct type of account mapping must be enabled: For a cluster with a non-uniform user name space, between-host account mapping must be enabled For a MultiCluster environment with a non-uniform user name space, cross-cluster user account mapping must be enabled User accounts must have the correct permissions to successfully run jobs. The owner of lsf.sudoers on Windows must be `Administrators`.