OSCER
Quick & Dirty Introduction to Compiling and Running on schooner

Compiling

Compiling MPI Programs

If you're not sure what an MPI program is, then skip down to the next section.

Before compiling an MPI program, you need to set two environment variables:

MPIENV, which indicates the family of compilers that you're using (ENV is short for "environment");
MPIDEV, which indicates the kind of communication that you're using (DEV is short for "device").

A quick discussion of how to set environment variables is below.

MPIENV
This environment variable indicates which compiler family to use.
OSCER supports multiple compiler families on schooner: Intel and GNU.
Possible values you can set MPIENV to are:

intel or intel9
Either of these values indicates that you want to compile using the Intel family of compilers: ifort (formerly known as ifc) for Fortran 77/90/95, or icc for C/C++.
NOTE: Currently, both intel and intel9 refer to ifort and icc version 9.0.
gnu or gcc
Either of these values indicates that you want to compile using the GNU family of compilers: g77, gcc, g++, version 3.2.3.
NOTE: The GNU family does not yet include a Fortran 90/95 compiler; g95 and gfortran have been partially released, but are not yet installed on schooner.

MPIDEV
This environment variable indicates how to send MPI messages between processors. Possible values you can set MPIDEV to are:

ib
This is the default, and in general you should use this unless you have a very good reason to do otherwise. This means that MPI messages should be sent over the high performance Infiniband interconnect, using the native Infiniband software driver. This is the fastest way to communicate among multiple nodes.
p4
This means that MPI messages should be sent using the slower TCP/IP software driver that is used for standard Ethernet (e.g., for sending data on the Internet). This is a substantially slower way to communicate among multiple nodes, and should be used only if you have a very good reason.
shmem
This means that MPI messages should be sent inside RAM. This should only be used if your MPI executable is guaranteed to be run on no more than 2 processors, and always within a single individual node.

How to set these environment variables depends on which shell program you are using. If you're not sure which shell program you're using (or if you're not sure what a "shell" program is), then type this at the Unix prompt and then press the "Enter" key:

ps

The response will look something like this:

  PID TTY          TIME CMD
28826 pts/19   00:00:00 tcsh
30567 pts/19   00:00:00 ps

If your shell is tcsh then to set the environment variables, type this at the Unix prompt:

setenv MPIENV intel
setenv MPIDEV ib

If your shell is bash or ksh or zsh, then to set the environment variables, type this at the Unix prompt:

export MPIENV=intel
export MPIDEV=ib

If your shell is "sh" then to set the environment variables, type this at the Unix prompt:

MPIENV=intel
MPIDEV=ib
export MPIENV
export MPIDEV

To Compile an MPI Program:

Now, to compile your MPI program, you need only use only of the following compiler names, as follows:

If your code is Fortran 90/95, use mpif90.
If your code is Fortran 77, use mpif77.
If your code is C, use mpicc.
If your code is C++, use mpiCC.

Compiling Non-MPI Programs

To compile a non-MPI program, you need simply to invoke the name of the compiler, with appropriate options. The compiler names are listed above, in the section on MPIENV.

Recommended Compiler Options

Each family of compilers has its own unique compiler options; you can find complete lists both in their man pages and in their full manuals.

So, the following are merely the recommended options for reasonably vanilla source codes.

Intel:
-O -tpp2 -ftz
GNU:
-O

Running

Except for very small, very brief non-MPI runs on 1 or 2 CPUs within a single node, all runs should be performed using the LSF batch queue system.

To start, you'll want to get a copy of one of the following files:

~hneeman/example_1node.bsub
Use this file to run any job that uses only 1 node, either because it uses a single CPU, or because it uses 2 threads (e.g., POSIX threads or OpenMP).
~hneeman/example_mpi.bsub
Use this file to run any job that uses multiple nodes with MPI over Infiniband, using 2 CPUs per node.
~hneeman/example_mpi_ppn1.bsub
Use this file to run any job that uses multiple nodes with MPI over Infiniband, using 1 CPU per node. You should avoid this one unless you have a really good reason to use 1 CPU per node.
~hneeman/example_mpi_gige.bsub
Use this file to run any job that uses multiple nodes with MPI over Gigabit Ethernet, using 2 CPUs per node.
~hneeman/example_mpi_ppn1_gige.bsub
Use this file to run any job that uses multiple nodes with MPI over Gigabit Ethernet, using 1 CPU per node. You should avoid this one unless you have a really good reason to use 1 CPU per node.

Once you've identified which file to use, do this:

cp  ~hneeman/example_1node.bsub  whatever.bsub

where whatever.bsub is the name of the batch script file that you want to create -- typically, you'll replace whatever with the name of your executable or the experiment or something.

You should be able to modify this file to suit your needs. It contains detailed information about how to set up and run a job.

IMPORTANT NOTE!!!
You MUST use the absolute FULL PATH for your executable! DON'T use a relative path nor leave out the path!

Running a Non-MPI Program

To run a non-MPI program, do as above with example_1node.bsub.

For help, please contact Henry Neeman.