Quick & Dirty Introduction to
Compiling and Running on topdawg
For help, please contact:
If you aren't sure whether your program uses the
Message
Passing Interface (MPI),
then it probably doesn't.
But, here's how you can find out for sure:
-
Log in to
topdawg.
-
At the Unix prompt,
change directory
(
cd)
to the subdirectory where your source code lives.
-
At the Unix prompt,
type this command:
grep "mpi.h" *.[cChH]*
-
At the Unix prompt,
type this command:
grep -i "mpif.h" *.[fF]*
-
At the Unix prompt,
type this command:
grep -i "use mpi" *.[fF]*
If, in response to any of these grep commands,
you get anything back other than the Unix prompt
(and especially anything that has the word "include"
or the word "use" in it),
then very probably your code uses MPI.
If not, then very probably your code doesn't use MPI.
If you're still unsure, contact us
and we'll help.
If you aren't sure whether your program uses the
OpenMP,
then it probably doesn't.
But, here's how you can find out for sure:
-
Log in to
topdawg.
-
At the Unix prompt,
change directory
(
cd)
to the subdirectory where your source code lives.
-
At the Unix prompt,
type this command:
grep "pragma" *.[cChH]* |
grep "omp"
-
At the Unix prompt,
type this command:
egrep -e '\!\$[oO][mM][pP]'
*.[fF]*
If, in response to any of these grep commands,
you get anything back other than the Unix prompt
(and especially anything that has the word "include"
or the word "parallel" in it),
then very probably your code uses OpenMP.
If not, then very probably your code doesn't use OpenMP.
If you're still unsure, contact us
and we'll help.
If you aren't sure whether your program uses the
POSIX
Threads (pthreads),
then it probably doesn't.
But, here's how you can find out for sure:
-
Log in to
topdawg.
-
At the Unix prompt,
change directory
(
cd)
to the subdirectory where your source code lives.
-
At the Unix prompt,
type this command:
grep -i "pthread" *.[cChHfF]*
If, in response to this grep command,
you get anything back other than the Unix prompt
(and especially anything that has the word "pthread" in it),
then very probably your code uses POSIX threads (pthreads).
If not, then very probably your code doesn't use POSIX threads.
If you're still unsure, contact us
and we'll help.
On topdawg,
we support the following compilers:
-
Intel compiler family (all version 9.1):
-
ifort
(formerly known as ifc)
for Fortran 77/90/95
-
icc
for C
-
icpc
for C++
-
Portland Group compiler family (all version 6.1-6):
-
pgf90
for Fortran 90/95
-
pgf77
for Fortran 77
-
pghpf
for High Performance Fortran
-
pgcc
for C
-
pgCC
for C++
-
GNU compiler family:
-
gfortran
for Fortran 90/95 (version 4.1.1)
-
g77
for Fortran 77 (version 3.4.6)
-
gcc
for C (version 3.4.6)
OR
gcc4
for C (version 4.1.1)
-
g++
for C++ (version 3.4.6)
OR
g++4
for C++ (version 4.1.1)
-
Numerical Algorithms Group (NAG) compiler family:
-
f95
for Fortran 77/90/95 (version 5.1)
Note that NAG f95 is a Fortran-to-C translator
that then calls gcc
(see above).
Each family of compilers has its own unique compiler options;
you can find complete lists both in their man pages
and in their
manuals.
So, the following are merely the recommended options
for reasonably vanilla source codes.
-
Intel compiler family (all version 9.1):
-O -tpp7 -xP
-
Portland Group compiler family (all version 6.1-6):
-fastsse
-
GNU compiler family:
-O
-
Numerical Algorithms Group (NAG) compiler family:
-O
If you've determined that your program uses no parallelism
—
that it's a "serial" (non-parallel) program
—
then here's how to compile.
-
Choose the compiler that you want.
-
Try the recommended compiler options.
-
You're also welcome to try other compiler options,
as described in the
compiler
manuals
and
man pages.
You may find it valuable to try compiling and running your code
with each of the compiler families,
and with various combinations of compiler options,
to find the best combination for your code.
Please bear in mind that some codes can only be compiled by
a subset of the compilers that are available on topdawg.
OpenMP is supported by the following compilers,
using the associated compiler option:
-
GNU
gfortran:
-fopenmp
-
Intel
icc, icpc, ifort:
-openmp
-
PGI
pgcc, pgCC, pgf77,
pgf90, pghpf:
-mp
POSIX threads (pthreads) are supported
by all of the compilers installed on topdawg,
using the associated compiler option.
To compile with pthreads,
all you need to do is,
in the final link step of compiling,
append the following at the end of the link command:
1. Before Compiling an MPI Program
Before compiling an MPI program,
you need to set some
environment variables:
-
MPI_COMPILER
(or
MPIENV),
which indicates
the family of compilers that you're using
(ENV is short for "environment");
-
MPI_INTERCONNECT
(or
MPIDEV),
which indicates
the kind of communication that you're using
(DEV is short for "device").
-
MPI_VENDOR,
which indicates
a subcategory of communication that you're using;
you only need to set this environment variable
under one certain specific circumstance.
A quick discussion of how to set environment variables is below.
MPI_COMPILER
(or
MPIENV)
This environment variable indicates which compiler family to use.
OSCER supports
multiple
compiler families on topdawg (see above).
Possible values you can set
MPI_COMPILER
(or
MPIENV)
to are:
-
intel
or
intel9
Either of these values indicates that
you want to compile using
the Intel family of compilers.
-
pgi
This value indicates that
you want to compile using
the Portland Group family of compilers.
-
gnu
or
gcc
Either of these values indicates that
you want to compile using
the GNU family of compilers.
-
nag
This value indicates that you want to compile using the
NAG compiler for Fortran 77/90/95 and the
GNU family of compilers for C and C++.
NOTE:
The NAG family includes only a Fortran 90/95 compiler,
which can also be used to compile Fortran 77 code,
but for C and C++ the GNU compilers are used.
(NAG f95 is a Fortran-to-C translator that invokes
gcc,
so these compilers should be fully compatible.)
MPI_INTERCONNECT
(or
MPIDEV)
This environment variable indicates
how to send MPI messages between processors.
Possible values you can set
MPI_INTERCONNECT
(or
MPIDEV)
to are:
-
ib
This is the default,
and in general you should use this
unless you have a very good reason to do otherwise
(for example, if Infiniband is incompatible with
the compiler you're using,
as shown in the
compatibility matrix
below.
This means that MPI messages will be sent over
the high performance Infiniband interconnect,
using the native Infiniband software drivers,
known as
MVAPICH.
This is the fastest way to communicate among multiple nodes.
NOTE:
A complete table of compiler/interconnect compatibilities
is below.
-
p4
This means that MPI messages will be sent
using the slower TCP/IP software driver
that is used for standard Ethernet
(e.g., for sending data on the Internet).
This is a substantially slower way to communicate among multiple nodes,
and should be used only if you have a very good reason
(such as needing the PGI or NAG compilers).
-
shmem
This means that MPI messages should be sent inside RAM.
This should only be used if your MPI executable is guaranteed
to be run on no more than 2 processors, and always within
a single individual node.
MPI_VENDOR
This environment variable indicates
some more specifics
how to send MPI messages between processors
via Infiniband.
The only possible value you can set
MPI_VENDOR
to is:
-
openmpi
This means that MPI messages should be sent using
OpenMPI,
rather than
MVAPICH.
If the
MPI_VENDOR
environment variable
is not defined,
but
MPI_INTERCONNECT
is set to
ib,
then
MVAPICH
will be used
as the Infiniband communication mechanism.
2. Compiler/Interconnect Compatibility Matrix
|
COMPILER
|
INTERCONNECT
|
|
|
ib
(Infiniband)
|
ib/openmpi
|
p4
(Gigabit Ethernet)
|
shmem
(inside a single node)
|
gcc
|
X
|
X
|
X
|
X
|
g++
|
|
X
|
X
|
X
|
g77
|
X
|
X
|
X
|
X
|
gcc4
|
X
|
X
|
X
|
X
|
g++4
|
X
|
X
|
X
|
X
|
gfortran
|
X
|
X
|
X
|
X
|
icc
|
X
|
X
|
X
|
X
|
icpc
|
X
|
X
|
X
|
X
|
ifort
|
X
|
X
|
X
|
X
|
pgcc
|
X
|
X
|
X
|
X
|
pgCC
|
X
|
X
|
X
|
X
|
pgf77
|
X
|
X
|
X
|
X
|
pgf90
|
X
|
X
|
X
|
X
|
pghpf
|
|
|
|
|
f95
|
|
|
X
|
X
|
3. How to Set Environment Variables
How to set these environment variables
depends on which shell program
you are using.
If you're not sure which shell program you're using
(or if you're not sure what a "shell" program is),
then type this at the Unix prompt and then press the "Enter" key:
ps
The response will look something like this:
PID TTY TIME CMD
28826 pts/19 00:00:00 tcsh
30567 pts/19 00:00:00 ps
If your shell is tcsh,
then to set the environment variables,
type this at the Unix prompt:
setenv MPI_COMPILER intel
setenv MPI_INTERCONNECT ib
If your shell is
bash or ksh or zsh,
then to set the environment variables,
type this at the Unix prompt:
export MPI_COMPILER=intel
export MPI_INTERCONNECT=ib
If your shell is "sh",
then to set the environment variables,
type this at the Unix prompt:
MPI_COMPILER=intel
MPI_INTERCONNECT=ib
export MPI_COMPILER
export MPI_INTERCONNECT
4. How to Compile an MPI Program
After You've Set the Environment Variables
After you've set the environment variables,
compile using one of the following compile commands:
-
mpicc
for C code
-
mpiCC
for C++ code
-
mpif77
for Fortran 77 code
-
mpif90
for Fortran 90 code
For example,
to compile an MPI code written in Fortran 90
using the Intel compiler family,
the compile command might start with:
mpif90 -O -tpp7 -xP ...
WARNING!
Be sure that every single compile command
in your compilaton procedure,
including the final compile command for linking the executable,
uses the EXACT SAME COMPILER OPTIONS!
Except for very small, very brief non-MPI runs
on 1 or 2 CPUs within a single node,
ALL
runs should be performed using the LSF batch queue system.
To start,
you'll want to get a copy of one of the following files:
-
~hneeman/example_nonparallel.bsub
Use this file to run any job that uses only 1 node,
either because it uses a single CPU,
or because it uses 2 threads (e.g., POSIX threads or OpenMP).
-
~hneeman/example_sharedmem.bsub
Use this file to run any job that uses
2 CPUs inside a single node,
via either OpenMP or with POSIX threads (pthreads).
-
~hneeman/example_mpi.bsub
Use this file to run any job that uses multiple nodes
with MPI over Infiniband,
using 2 CPUs per node.
-
~hneeman/example_mpi_ppn1.bsub
Use this file to run any job that uses multiple nodes
with MPI over Infiniband,
using 1 CPU per node.
You should avoid this one unless you have a really good reason
to use 1 CPU per node.
-
~hneeman/example_mpi_gige.bsub
Use this file to run any job that uses multiple nodes
with MPI over Gigabit Ethernet,
using 2 CPUs per node.
-
~hneeman/example_mpi_ppn1_gige.bsub
Use this file to run any job that uses multiple nodes
with MPI over Gigabit Ethernet,
using 1 CPU per node.
You should avoid this one unless you have a really good reason
to use 1 CPU per node.
Once you've identified which file to use, do this:
cp ~hneeman/example_nonparallel.bsub whatever.bsub
where
whatever.bsub
is the name of the batch script file that you want to create;
typically,
you'll replace
whatever
with the name of your executable or the experiment or something.
You should be able to modify this file
to suit your needs.
It contains detailed information
about how to set up and run a job.
IMPORTANT NOTE!!!
You MUST
use the absolute FULL PATH
for your executable!
DON'T
use a relative path nor leave out the path!
For help, please contact:
|
