Introduction to MPI on the CRAY Darter

Parallel Programming with MPI on the CRAY Darter
Joint Institute for Computational Sciences
March 27, 2015


The purpose of this lab is to become familiar with MPI account setup and documentation as well as compiling and running elementary MPI programs on CRAY Darter.


  1. Introduction to MPI on the CRAY Darter
    1. Workstation Login and Environment Setup
    2. MPI Programming - Exercise Information
    3. General Information
    4. Hello World 1 - The Minimal MPI Program
    5. Hello World 2 - Hello Again
    6. Pi Calculation
    7. Timing an MPI Code
    8. PingPong - Calculating Transfer Rates
    9. Ring 1 - Sending Messages around a Ring
    10. Simple Array Assignment
    11. Matrix Multiplication
    12. 2D Laplace Equation
  2. Other Important Information
    1. Logging Off
    2. Documentation for MPI and mpich, and Additional Resources
    3. Acknowledgments

Workstation Login and Environment Setup

  1. First of all, you need to login to your temporary account on Darter via the Secure Shell protocol (SSH). To do this please refer to the NICS User Support page.
  2. On Darter, access to compute resources is managed by the Portable Batch System (PBS). Batch scripts are run on service nodes that have access to the home, project and software directories. Executables launched with the aprun command do not have access to these directories; they have access only to the Lustre scratch directories, /lustre/medusa/YOUR_USER_ID.
  3. In LAB-EXAMPLE folder you will find three subdirectories:


    First two folders contain all the material necessary for the workstation exercises written in either C or Fortran77. Select the directory that suits your programming preferences. You are now ready to proceed into the world of MPI.

MPI Programming - Exercise Information

In this lab you will utilize the most fundamental MPI calls necessary for development of any MPI code, as well as learn how to compile and run MPI code. During this lab you will encounter the following exercises:

  1. Hello World 1 - exercise (hello1directory)
  2. Hello World 2 - exercise (hello2directory)
  3. Pi Calculation - exercise (pical directory)
  4. Timing an MPI Code - exercise (timing directory)
  5. PingPong - exercise (pingpong directory)
  6. Ring 1 - exercise (ring directory)
  7. Simple Array Assignment - exercise (array directory)
  8. Matrix Multiplication - exercise (matmul directory)
  9. Laplace Equation - exercise (laplace directory)

But first, the following . . .

General Information

During the workshop exercises you may be asked to write a code; if possible write the codes on your own. To do so use the files with the .start extensions that are provided in the directories where programming is required.

For those of you wishing to concentrate only on the message passing aspects of the code, files with .template extensions have been provided in the exercise directories that require programming. To modify the template files, first copy filename._.template to filename.c or filename.f depending on whether you are using C or FORTRAN. For example:

% cp filename.c.template filename.c
% cp filename.f.template filename.f

Next, invoke your favorite text editor and modify the template by replacing all of the XXXXX's with the appropriate MPI calls.

For your convenience, there are completed solutions to the programming exercises available as files with .soln extensions. Remember, the only way to learn how to program is by actually programming, SO LOOK AT THE SOLUTIONS ONLY AS A LAST RESORT.

Finally, extra exercises have been provided at the end of some of the main exercises. Please, do not work on these until after you have completed the main exercises for the day. These extra exercises have been provided without templates or solutions, and your lab assistants may be able to give you only very general help with them. In other words, try them at your own risk.

Hello World 1 - The Minimal MPI Program

The objective of this exercise is not to write a code but to demonstrate the fundamentals of compiling an MPI program and submitting it via qsub.

  1. Examine the "Hello World!" program hello.c/hello.f. Notice that every process prints "Hello World!" and that the "Hello World!" program:
    1. Includes a header,
    2. Initializes MPI,
    3. Prints a "Hello World!" message, and
    4. Finalizes MPI

  2. Compile "Hello World!". For the version of MPI that we are using there are several ways to compile a program. We will use the commands cc and ftn to compile our C and FORTRAN programs, respectively. These compilers locate the MPI libraries and header files as needed. To compile, you can either compile at the prompt or use the provided makefile. If you use the provided makefile, first read it to understand what it is doing.
  3. For the "Hello World!" program, enter either

    % cc hello.c -o hello for C programs, or
    % ftn hello.f -o hello for FORTRAN programs

    or, enter

    % make

    Again, if you use the provided makefile, first make sure you understand what it is doing.

  4. Now we want to run the "Hello World!" program. This elementary problem will use 8 processors and will assign a rank to each of them. Then the program will output 8 lines depending on the rank of the process.
  5. Hello World (from masternode)
    Hello WORLD!!! (from worker node)
    Hello WORLD!!! (from worker node)
    Hello WORLD!!! (from worker node)
    Hello WORLD!!! (from worker node)
    Hello WORLD!!! (from worker node)
    Hello WORLD!!! (from worker node)
    Hello WORLD!!! (from worker node)
    Since a parallel program runs on several processors, there needs to be some method of starting the program on each of the different processors. On Darter this is done by using the Batch Scripts. Batch scripts can be used to run a set of commands on a systems compute partition. The batch script is a shell script containing PBS flags and commands to be interpreted by a shell. Batch scripts are submitted to the batch manager, PBS, where they are parsed. Based on the parsed data, PBS places the script in the queue as a job. Once the job makes its way through the queue, the script will be executed on the head node of the allocated resources. The batch script example below explains how to submit the batch scripts, and common usage tips.

    Example Batch Script

    1: #!/bin/bash
    2: #PBS -A XXXYYY
    3: #PBS -N test
    4: #PBS -j oe
    5: #PBS -l walltime=1:00:00,size=256
    7: cd $PBS_O_WORKDIR
    8: date
    9: aprun -n 256 ./a.out

    This batch script can be broken down into the following sections:

    Submitting Batch Jobs

    Batch scripts can be submitted for execution using the qsub command on Darter. For example, the following will submit the batch script named test.pbs:

    % qsub test.pbs

    If successfully submitted, a PBS job ID will be returned. This ID can be used to track the job.

    For more information about qsub see the man pages.

    Look at the running job page for more (and some redundant) information. In particular, look at the PBS commands for submitting jobs, removing jobs from the queue, etc.

  6. We will use the PBS pbssub script to submit our job for the hello1 exercise. After compiling the source code you will find an executable file in the current hellol directory. To submit the job to the CRAY Darter queue use the provided pbssubfile, but first be sure to examine the file.
  7. Submit the job with the command:

    % qsub pbssub

  8. Did the job enter the batch queue? Check, with the showq command. Where did the job run?

Hello World 2 - Hello Again!

The objective of this exercise it to become familiar with the basic MPI routines used in almost any MPI program. You are asked to write an SPMD(Single Process, Multiple Data) program where, again, each process checks its rank, and decides if it is the master (if its rank is 0), or a worker (if its rank is 1 or greater).

  1. The SPMD programs should:
    1. Include the header,
    2. Initialize MPI,
    3. Check its rank, and
      1. if the process is the master, then send a "Hello World!" message, in characters, to each of the workers
      2. if the process is a worker, then receive the "Hello World!" message and print it out
    4. Finalize MPI
  2. Compile your program at the command line or via the makefile. Run this code on 8 processes using qsub pbssub command. You can also run the code on 16, 24 etc processes keeping in mind that the number of requested cores must be the multiple of 8.

Pi Calculation

This program calculates pi using a simple integration of a tangent function.

  1. Review the mpi_pi.c/mpi_pi.f code to get an idea of what it does.
  2. Compile the code. A makefile has been provided for you. To compile with the makefile, simply type:make. Or to compile at the command line use the cc compiler for the C code or the ftn compiler for the FORTRAN code:

    % cc mpi_pi.c -o mpi_pi
    % ftn mpi_pi.f -o mpi_pi

  3. Run the code: mpi_pi using provided pbssub file

Timing an MPI code

The objective of this exercise is to investigate the amount of time required for message passing between two processes.

In this exercise different size messages are sent back and forth between two processes a number of times. Timings are made for each message before it is sent and after it has been received. The difference is computed to obtain the actual communication time. Finally, the average communication time and the bandwidth are calculated and output to the screen.

We will run this code on two nodes (one process on each node) passing messages of length 1, 100, 10,000, and 1,000,000. You can record your results in a table like the one below.

length time
1 0.000001 65.440140
100 0.000002 2936.930591
10,000 0.000052 12321.465896
1,000,000 0.005133 12468.521884
A makefile and pbssub files have been provided:
to compile the code, type make
to submit the job, type qsub pbssub

PingPong - Calculating Transfer Rates

The objective of this exercise is to introduce some intermediate MPI features, and to understand how a possible deadlock situation can occur during message passing.

Write a program (pingpong) in which two processes pass a message (a certain number of real or float numbers) back and forth (perhaps 100 times). You will use the MPI_Wtime() routine as a timer in the following exercise. This routine returns a time expressed in seconds, so in order to time something, two calls are needed and the difference should be taken between them to obtain the total elapsed time (in wall clock seconds).

  1. In the program, pingpong, it is safer to use MPI_Ssend, since MPI_Send may or may not be synchronous, and its use may result in a deadlock situation.
  2. Compile (make) pingpong and run it: (cc -o pingpong pingpong.c).
  3. Insert timing calls (see man MPI_Wtime) to estimate the time taken for one message on a one way trip. Calculate the transfer rate in bytes per second. What did you find?
  4. Add a loop around the timing calls changing the length of the message (length varies from 1 to 10001 in steps of 1000) to investigate how the time taken varies with the size of the message.

Ring 1 - Sending Messages around a Ring

Consider a set of processes arranged in a ring as shown below. Use a token passing method to compute the sum of the ranks of the processes.

/ \
0 2
\ /

Figure 1: Four processors arranged in a ring, messages are sent from 0 to 1 to 2 to 3 to 0 again, sum of ranks is 6.

Each processor stores its rank in MPI_COMM_WORLD in an integer and sends this value to the processor on its right. It then receives an integer from its left neighbor. It keeps track of the sum of all the integers received. The processors continue passing on the values they receive until they get their own rank back. Each process should finish by printing out the sum of the values. Use synchronous sends MPI_Ssend() (blocking) or MPI_Issend() (non-blocking) for this program. Watch out for deadlock situations. If you use non-blocking sends, make sure that you do not overwrite information. You are asked to use synchronous message passing because the standard send can be either buffered or synchronous, and you should learn to program for either possibility.

Simple Array Assignment

This is a simple array assignment used to demonstrate the distribution of data among multiple tasks and the communications required to accomplish that distribution.

The master distributes an equal portion of the array to each worker. Each worker receives its portion of the array and performs a simple value assignment to each of its elements. Each worker then sends its portion of the array back to the master. As the master receives a portion of the array from each worker, selected elements are displayed.

Note: For this example, the number of processes should be set to an odd number(aprun -n 7), to ensure even distribution of the array to numtasks-1 worker tasks.

Matrix Multiplication

This example is a simple matrix multiplication program.

The data is distributed among the workers who perform the actual multiplication in smaller blocks and send back their respective results to the master.

Note: The C and FORTRAN versions of this code differ because of the way arrays are stored/passed. C arrays are stored in row-major order while FORTRAN arrays are stored in column-major order.

2D Laplace Equation

This example solves a two-dimensional Laplace equation using the point Jacobi iteration method over a rectangular domain. The initial guess value of the function is zero. The boundaries are held at 100 throughout the calculation. Domain decomposition will be used for the parallel implementation of the problem. To run this exercise, run it on 4 processes (aprun -n 4).

Logging Off

Type exit to close the connection with the CRAY Darter machine.

Documentation for MPI and mpich, and additional resources

  • There are man pages available for MPI which should now be installed in your MANPATH. Look at the following man pages to see some introductory information about MPI.
  • % man MPI
    % man cc
    % man ftn
    % man qsub
    % man MPI_Init
    % man MPI_Finalize

  • You can also refer to NICS User Support Page for Darter specific MPI Implementation details.
  • The MPI man pages are also available online.
  • The WWW home for is at the Argonne National Lab. They also maintain a general MPI page.

  • Acknowledgments

    The original MPI training materials for workstations were developed under the Joint Information Systems Committee (JISC) New Technologies Initiative by the Training and Education Centre at Edinburgh Parallel Computing Centre (EPCC-TEC), University of Edinburgh, United Kingdom.

    Thanks also to Blaise Barney from Cornell University Theory Center for his modifications of the labs available through the MHPCC on the World Wide Web. These labs have since been modified for this workshop.

    Joint Institute for Computational Sciences