---Getting started with DIEL---

This file will guide you through setting up the openDIEL and its dependencies. We will
test your installation and help you become familiar with running the openDIEL by walking you
through a few examples which demonstrate a number of key features.

DIEL is a work-in-progress and is designed to work on many platforms. If you need assistance,
please see the contact information at http://cfdlab.utk.edu/openDIEL. We appreciate your
interest in this project and will be more than happy to assist you!

Since you are reading this file, let's assume you have already downloaded openDIEL.tar.gz from
cfdlab.utk.edu/openDIEL and extracted it to a directory where you have adequate permissions to
compile and run binaries.


---SECTION ONE: Running the fanTest---
Overview: The fanTest compares the performance of the TSU against MPI function calls.
Note: $(IEL_HOMEa) is the absolute path to the top level openDIEL directory.

Step 1: Build MPICH

The openDIEL is designed to run in a parallel computing environment. MPICH, an implementation
of the MPI standard, is vital to many of the operations carried out by the openDIEL. Currently
the openDIEL looks for all MPICH related files under $(IEL_HOMEa)/lib/mpich/,
$(IEL_HOMEa)/include/mpich/, and $(IEL_HOMEa)/bin/mpich/. The top level Makefile is capable of
compiling and installing MPICH into the appropriate directories. To do this run the following
command from the top level directory.

make mpich >& make.mpich.out &

To check on the progress of the build you can run the following command.

tail -f make.mpich.out

Alternatively, if you already have MPICH installed on your system you can simply edit the
Makefile.inc file. Make sure to edit the following items to match your system's settings.

MPICH_LIB=-L/path/to/mpich/libs -lmpich
MPICH_INC=-I/path/to/mpich/headers

cc=/path/to/mpicc
CC=/path/to/mpicxx
ftn=/path/to/mpif90
f77=/path/to/mpif77
f90=/path/to/mpif90


Step 2: Building and Running the fanTest

The fanTest will check to ensure the the basic functionality of the openDIEL is working. To run
this test you will need to first compile the fanTest by running the following command from the
top level directory.

make TESTS >& make.TESTS.out &

You will now need to compile the driver code to get the test to run. To do this run the
following commands.

cd modules/TESTS/driver
make
export LD_LIBRARY_PATH=/path/to/lib/exec/:/path/to/lib/comm/:$LD_LIBRARY_PATH
make exec

A successful run results in a series of "Initializing the executive..." lines followed by a
series of "rank X going to statically run 'fanTuple'" lines being printed to the screen.

A note regarding shared libraries: By default the openDIEL will build all libraires as shared
libraries. This means that the linking step will occur at runtime. Should you encounter a
situation where running an executable results in errors of the type, "error while loading
shared libraries:..." you will need to set you LD_LIBRARY_PATH to include the location of the
missing libraries. To avoid having to do this you can build statically by setting the 'L'
variable to "static" at the top of the Makefile.inc file in the top directory.


---SECTION TWO: Running the REDIFF Module---
Step 1: Build MPICH (Follow the steps in SECTION ONE)

Step 2: Build Trilinos

Trilinos is a popular software library at the core of many scientific applications. This
powerful library is put to use in our cardiac modeling software known as REDIFF (REaction
DIFFusion). Trilinos is a large library and will require a long time to compile. If you are on
a system where Trilinos is already installed you can simply change the Makefile.inc file to
suit your system's configuration. Be sure to change the following values.

TRILINOS_LIB=-L/path/to/trilinos/libs -laztecoo -lepetra -ltriutils -lteuchoscomm -lteuchoscore -lteuchosparameterlist -lteuchosnumerics -lteuchosremainder
TRILINOS_INC=-I/path/to/trilinos/headers

If your system does not have Trilinos installed you can install it by running the following
command from the top level directory.

make trilinos >& make.trilinos.out &

This will build and install the Trilinos library in the directories the openDIEL expects to
find Trilinos related files.

Step 2: Compiling REDIFF
Once you have either changed the Makefile.inc file or built Trilinos from source you will now
need to compile the REDIFF module. To do this run the following command from the top level
directory.

make REDIFF >& make.rediff.out &

A note regarding modMaker: Examining the REDIFF target in the Makefile will reveal that during
the compilation of the REDIFF module the modMaker script is run on the fetemplate.cpp source
file (fetemplate.cpp is at the core of REDIFF). modMaker is very important to the openDIEL due
to its ability to convert standard MPI programs into programs suitable to be run under the
openDIEL. This means users can take their parallel programs, run modMaker on them, and use them
as modules in the openDIEL. For more information on modMaker see modMakerREADME under /tools.

Step 3: Building the Driver
You will now need to compile the driver code to get the REDIFF module to run. To do this begin
by entering the driver directory ($(IEL_HOMEa)/modules/REDIFF/driver). If you list the contents
of this directory, you will find a driver.c file (which contains the new main function) and a
Makefile that has been generated by modMaker. While modMaker can do most of the work for this
Makefile, it cannot (yet) add dependencies such as TRILINOS to the linking stage, so you need
to do this yourself if your module requires them. REDIFF happens to require TRILINOS, so open
the Makefile and add the paths for the lib and include directories of TRILINOS (follow the
example for MPICH) and add them to the INCLUDES and LDFLAGS variables as follows.

TRILINOS_INC=-I../../../include/trilinos

TRILINOS_LIB=-L../../../lib/trilinos -laztecoo -lepetra -ltriutils -lteuchoscomm -lteuchoscore -lteuchosparameterlist -lteuchosnumerics -lteuchosremainder

INCLUDES=$(EXEC_INC) -I/path/to/REDIFF/REDIFF $(COMM_INC) $(MPICH_INC) $(TRILINOS_INC)

LDFLAGS= $(EXEC_LIB) $(COMM_LIB) -L/path/to/REDIFF/REDIFF -lmodrediff $(CLIBS) $(MPICH_LIB) $(TRILINOS_LIB)

Now, save the file and type make:

make

Assuming this completed without error, Congratulations! You have built your
first DIEL program. REDIFF happens to need several input files to be in the working directory.
Example input files reside in the modules/REDIFF/Example-cube directory. Copy them to the
driver directory:

cp ../Example-cube/* .

If you are using shared libraries then you will need to set your LD_LIBRARY_PATH as follows:
export LD_LIBRARY_PATH=/path/to/lib/trilinos:/path/to/modules/REDIFF/REDIFF:/path/to/lib/comm:/path/to/lib/exec:$LD_LIBRARY_PATH

The Makefile generated by modMaker has an exec target that will execute the program:

make exec