The Queueing System

The Queueing System | FAQ | Hardware Specification | Software Description | How to get an account

How to Use the Emerson Center Facilities

1. LoadLeveler Queueing System

The EC uses LoadLeveler (LL) to provide efficient usage of the hardware equipment and software to our subscribers. LL runs scripts (jobs) on your behalf on the appropriate machine in a cluster of machines. LL issues standard UNIX commands in batch mode, submits your jobs into existing queues, and assigns a number and name to your jobs. In brief, you are required to prepare your input file, include the name of that input file into a LL shell script (usually ksh) called xxx.cmd (fixed formatted, xxx could be any name) and submit it (xxx.cmd) to our computers by executing the “llsubmit xxx.cmd” command.

EC has three independent groups of queues, namely SP3+SGI, SP2, and RS/6000. While the number of jobs submitted to the RS/6000 queue is unlimited, there exist limits for the maximum number of jobs per subscriber group on the SP2 and SP3+SGI systems corresponding to the shares (or units) of subscribers.

The groups of queues can be distinguished from each other by LL parameters such as "class" and "Features", which are LL keywords that need to be included in the LL xxx.cmd script file.

A. IBM RS/6000 and P+ (AIX 5.) standalone service machines:

B. IBM SP2 Power (AIX 5.1) cluster

* X=1,2,...,6

C. IBM SP3 Power (AIX 5.3) cluster

* X=1,2,...,15

D. SGI Origin (IRIX 6.5) workstations

2. How to submit a job to LoadLeveler

Use this .cmd file as a template. Note that to submit to fire queues, substitute /home by /fire in the "initialdir" line below.

#
#!/bin/ksh
# @ initialdir = /home/<yourgroup>/<yourname>/...
# @ requirements = (Arch == "R6000") && (OpSys == "AIX53")
# @ class = pp2
# @ notify_user = <you@e-mail address>
# @ group = <yourgroup>
# @ error = error
# @ queue
#
INPF=<youin>
OUTF=<youout>
. /libs/scripts/g92 # gaussian
#

Please feel free to browse /libs/scripts for available codes.

Please include the direction for LoadLeveler in your $PATH environment variable. This can be done with placing the following additional lines at the end of your .profile (for ksh users):
PATH=/usr/lpp/LoadL/nfs/bin:$PATH
export PATH
or .cshrc (for csh users):
set path=(/usr/lpp/LoadL/nfs/bin $path)

In order to submit a job, "ssh euch4e.chem.emory.edu -l YOURNAME". Job submission is only allowed from this particular machine. The LL command for submitting a job is llsubmit:
euch4e# llsubmit xxx.cmd
Typical LL scriptfiles are located in the /usr/local/Samples directory of euch4e, and we highly recommend to start creating own script files by modifying these files appropriately.

3. Program specific LoadLeveler include scripts

Most installed software packages can be used by including ksh scripts provided by the center. These files can be found in the /libs/scripts directory on euch4e and are usually incorporated ino a LL script by the following three lines:
INP=<input_filename>
OUT=<output_filename>
. /libs/scripts/<script_filename>
(please note the leading dot in the third line). This syntax species the location of input, output files and the type of program to run. Most important scripts are:/libs/scripts/g98a9 (to run GAUSSIAN98 Revision A.9 on SP3 nodes)
/libs/scripts/sybyl (to run Sybyl Version 6.7 on SGI)
/libs/scripts/acesrun (to run ACESII 3.0 on RS/6000 and SP2 nodes)
/libs/scripts/acesp3 (to run ACESII 3.0 on SP3 nodes)
/libs/scripts/amber (to run AMBER Version 6.0 on SGI)
/libs/scripts/bmin (to run MacroModel_7.1 Batchmin on SGI)
/libs/scripts/cadpac (to run CADPAC Version 5.2 on IBM’s)
/libs/scripts/dalrun (to run DALTON Version 2.0 on IBM’s)
/libs/scripts/gmsrun (to run GAMESS on RS/6000 and SP2 nodes)
/libs/scripts/gmsrunsp3 (to run GAMESS on SP3 nodes)
Please note that there are different LL include scripts for the same software, depending on where the job will run. It is important not to mix platforms and include files up.
If a user has his own shell script, it is important to know that /work is the default location of temporary files. However, when using the SP3 nodes, the following part needs to be included in the script in order to determine the location of the work directory (either /work, /work1, /work2, or /work3):
LOADL_ID=`echo "${LOADL_JOB_NAME}" | sed 's/.chem.emory.edu//'`
WDIR=/var/loadl/workdirs
w1=`cat $WDIR/work`
w2=`cat $WDIR/work1`
w3=`cat $WDIR/work2`
w4=`cat $WDIR/work3`
if [ "$w1" = "$LOADL_ID" ]
then
export scr=/work
elif [ "$w2" = "$LOADL_ID" ]
then
export scr=/work1
elif [ "$w3" = "$LOADL_ID" ]
then
export scr=/work2
elif [ "$w4" = "$LOADL_ID" ]
then
export scr=/work3
else
exit 111
fi