CSCI 5451 Assignment 2: Distributed and Shared Memory Programming

Additionally, MSI uses a software modules system to specify use of different libraries and code packages. To access the most current version of MPI, you will need to run the following command in your login shell.

# load mpi software into path
ln0006 [~]% module load ompi/4.0.0/gnu-8.2.0-centos7

# alternative using script in code pack directory
ln0006 [~]% source a2-code/msi-setup.sh

# check version of MPI
ln0006 [~]% which mpicc
/panfs/roc/msisoft/openmpi/el6/4.0.0/gnu-8.2.0.CentOS7/bin/mpicc

ln0006 [~]% which mpirun
/panfs/roc/msisoft/openmpi/el6/4.0.0/gnu-8.2.0.CentOS7/bin/mpirun

Note that the msi-setup.sh script is provided as an alternative and can be sourced.

The list of all modules available is seen via module avail. Note that the version of MPI we will use is the most current but not the default.

Use the sbatch job.sh command to submit a job to the SLURM job queue. To ease timing evaluations, a job script with required parameters has been provided for all of the problems. For instance, for the Heat problem, use the script heat-slurm.sh. Checking on the status of jobs in the queue is done via squeue -u username. The provided scripts have the following properties:

• Require you to change the directory of your code in it but otherwise do not need to be modified. Look for the line

    # ADJUST: location of executable
    cd ~/mesabi-a2-5451/
  

  in the script and change the target directory to wherever you have placed your code. Ensure it is compiled ahead of time.

• Will run MPI jobs for the required programs on the required number of processors
• Output for the job will be left in a file like heat-slurm.sh.job-156425531.out which starts with the job script name

• Lines in the script that are marked runtime: report wall clock time along with the problem parameters in it these appear like

  runtime: procs 16 width 102400 realtime 5.15
  

  and make it possible to use grep or other tools to quickly fish out the lines reporting runtimes for construction of the results.

• Will charge compute time to the CSCI 5451 class account via the --account option to SLURM
• The scripts have the correct MPI software module loaded at the beginning of them and will call mpirun on the program.

The session below demonstrates use of sbatch and squeue

ln0006 [mesabi-a2-5451]% sbatch heat-slurm.sh    # submit script to run for heat jobs
Submitted batch job 156504577

ln0006 [mesabi-a2-5451]% squeue -u kauffman      # check on the submitted jobs by user kauffman
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         156504535     small kmeans-s kauffman PD       0:00      6 (Priority)
         156504577     small heat-slu kauffman PD       0:00      6 (None)

# jobs are PD - Pending but not being run yet


# a useful command to "watch" the queue, press Ctl-c to kill the watch
ln0006 [mesabi-a2-5451]% watch 'squeue -u kauffman'

# later....
ln0006 [mesabi-a2-5451]% squeue -u kauffman      # check on the submitted jobs by user kauffman
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

ln0006 [mesabi-a2-5451]%                         # no jobs listed, all jobs done with results

ln0006 [mesabi-a2-5451]% ls *156504577*          # check for output job
heat-slurm.sh.job-156504577.out

Testing out codes on MSI machines is ideal as it is the same environment as what the timing evaluation will involve. The same commands can be used (load MPI module, compile, run with mpirun -np 4), with the minor limitation that there is a cap on the number of processors that can be tested interactively.

CSE Labs machines have an MPI installation (sort of) set up on the machines cuda01-cselabs.umn.edu to cuda05-cselabs.umn.edu. By logging in, one will have access on those machines to mpicc and mpirun commands. However, the configuration is a bit unwieldly and requires some setup. Here are some gotchyas to keep in mind and move you towards the MSI machines.

1. You must use a Hostfile via mpirun -hostfile hf.txt and due to the configuration problems on these systems, the hosts will need to be listed via their IP address rather than their symbolic name.
2. You will need to have SSH keys setup up for password-free login. For details on how to do this see this tutorial.
3. All of the cuda-NN machines will need to be in your authorized host file. The easiest way to do this is to log in to cuda01 then, from that machine, SSH to cuda02, then cuda03, and so forth up to cuda05. This only needs to be done once but if you fail to do so, MPI runs beyond the local number of processors will hang silently.

Surmounting these factors is not difficult but creates enough friction that MSI is easier and preferred for testing.

Setting up a home MPI installation is another option. In most cases a Linux or Unix system can install OpenMPI with a package manager and by using mpirun --oversubscribe one can test on a number of processors larger than available as physical cores on the system. Do not expect speedup in these cases but it is extremely useful for debugging.

A slightly modified version of the heat propagation simulation from HW1 and in-class discussion is in the code pack and called heat_serial.c. This program can be compiled and run with the provided Makefile as follows.

>> make heat_serial             # build program
gcc -g -Wall -o heat_serial heat_serial.c

>> ./heat_serial                # run with no args to show help info
usage: ./heat_serial max_time width print
  max_time: int
  width: int
  print: 1 print output, 0 no printing

>> ./heat_serial 10 8 1         # run for 10 timesteps with 8 "elements"
   |     0     1     2     3     4     5     6     7 
---+-------------------------------------------------
  0|  20.0  50.0  50.0  50.0  50.0  50.0  50.0  10.0 
  1|  20.0  35.0  50.0  50.0  50.0  50.0  30.0  10.0 
  2|  20.0  35.0  42.5  50.0  50.0  40.0  30.0  10.0 
  3|  20.0  31.2  42.5  46.2  45.0  40.0  25.0  10.0 
  4|  20.0  31.2  38.8  43.8  43.1  35.0  25.0  10.0 
  5|  20.0  29.4  37.5  40.9  39.4  34.1  22.5  10.0 
  6|  20.0  28.8  35.2  38.4  37.5  30.9  22.0  10.0 
  7|  20.0  27.6  33.6  36.3  34.7  29.8  20.5  10.0 
  8|  20.0  26.8  32.0  34.1  33.0  27.6  19.9  10.0 
  9|  20.0  26.0  30.5  32.5  30.9  26.5  18.8  10.0 

>> ./heat_serial 10 8 0         # same run but don't print output, useful for timing as output takes a while

>> ./heat_serial 12 5 1         # run for 12 timesteps with 5 columns / elements
   |     0     1     2     3     4 
---+-------------------------------
  0|  20.0  50.0  50.0  50.0  10.0 
  1|  20.0  35.0  50.0  30.0  10.0 
  2|  20.0  35.0  32.5  30.0  10.0 
  3|  20.0  26.2  32.5  21.2  10.0 
  4|  20.0  26.2  23.8  21.2  10.0 
  5|  20.0  21.9  23.8  16.9  10.0 
  6|  20.0  21.9  19.4  16.9  10.0 
  7|  20.0  19.7  19.4  14.7  10.0 
  8|  20.0  19.7  17.2  14.7  10.0 
  9|  20.0  18.6  17.2  13.6  10.0 
 10|  20.0  18.6  16.1  13.6  10.0 
 11|  20.0  18.0  16.1  13.0  10.0 

In our Thu 02-Feb Lecture we discussed an MPI version of the heat transfer problem and it may be worthwhile to review this lecture to recall some details of data distribution and communication.

The central task of this problem is to create an MPI version of this program named heat_mpi which performs the same task but uses MPI calls to perform the heat calculations on distributed memory machines. Once completed, this program can be run as follows.

>> make heat_mpi                               # build MPI version of heat program
mpicc -g -Wall -o heat_mpi heat_mpi.c

>> source mpiopts.sh                           # set the MPIOPTS env variable, used to suppress warnings

>> mpirun $MPIOPTS -np 2 ./heat_mpi 10 8 1     # run using 2 procs, 10 steps, 8 elements = 4 per proc
   |     0     1     2     3     4     5     6     7 
---+-------------------------------------------------
  0|  20.0  50.0  50.0  50.0  50.0  50.0  50.0  10.0 
  1|  20.0  35.0  50.0  50.0  50.0  50.0  30.0  10.0 
  2|  20.0  35.0  42.5  50.0  50.0  40.0  30.0  10.0 
  3|  20.0  31.2  42.5  46.2  45.0  40.0  25.0  10.0 
  4|  20.0  31.2  38.8  43.8  43.1  35.0  25.0  10.0 
  5|  20.0  29.4  37.5  40.9  39.4  34.1  22.5  10.0 
  6|  20.0  28.8  35.2  38.4  37.5  30.9  22.0  10.0 
  7|  20.0  27.6  33.6  36.3  34.7  29.8  20.5  10.0 
  8|  20.0  26.8  32.0  34.1  33.0  27.6  19.9  10.0 
  9|  20.0  26.0  30.5  32.5  30.9  26.5  18.8  10.0 

>> mpirun $MPIOPTS -np 4 ./heat_mpi 6 12 1     # run using 4 procs, 6 steps, 12 elements = 3 per proc 
   |     0     1     2     3     4     5     6     7     8     9    10    11 
---+-------------------------------------------------------------------------
  0|  20.0  50.0  50.0  50.0  50.0  50.0  50.0  50.0  50.0  50.0  50.0  10.0 
  1|  20.0  35.0  50.0  50.0  50.0  50.0  50.0  50.0  50.0  50.0  30.0  10.0 
  2|  20.0  35.0  42.5  50.0  50.0  50.0  50.0  50.0  50.0  40.0  30.0  10.0 
  3|  20.0  31.2  42.5  46.2  50.0  50.0  50.0  50.0  45.0  40.0  25.0  10.0 
  4|  20.0  31.2  38.8  46.2  48.1  50.0  50.0  47.5  45.0  35.0  25.0  10.0 
  5|  20.0  29.4  38.8  43.4  48.1  49.1  48.8  47.5  41.2  35.0  22.5  10.0 

>> time mpirun $MPIOPTS -np 4 ./heat_mpi 6 12 0 # same as above but suppress output and time the run
real	0m0.168s    # wall clock time to report for the run
user	0m0.102s
sys	0m0.073s

• Name your program heat_mpi.c to be compatible with the provided Makefile. It has a target to build both heat_serial and heat_mpi if you name the source file heat_mpi.c.

• The serial version of the program provided accepts 3 command line arguments:

  1. Number of time steps (rows in output)
  2. Width of the rod in elements (columns of output)
  3. 1 or 0 to indicate whether final output should be printed or suppressed.

  The MPI version should allow for the same arguments so that runs like the following will work.

    >> mpirun $MPIOPTS -np 4 ./heat_mpit 10 40 1   # 4 procs, 10 timesteps, width 40, show output 
    ...                                           # output for the run
  
    >> mpirun $MPIOPTS -np 4 ./heat_mpi 10 40 0   # same but no output
    >> 
  

• There is a small script called mpiopts.sh which can set options for MPI runs to suppress unnecessary warning messages. While experimenting with your programs in a shell, you can source this script via the following.

    >> source mpiopts.sh            # sets variable MPIOPTS
    
    >> echo $MPIOPTS                # show value of MPIOPTS
    --mca opal_warn_on_missing_libcuda 0
    
    >> . mpiopts.sh                 # same as using "source" to execute script in current shell
  

• Divide the problem data so that each processor owns only a portion of the columns of the heat matrix as discussed in class.
• Utilize send and receives or the combined MPI_Sendrecv to allow processors to communicate with neighbors.
• Utilize a collective communication operation at the end of the computation to gather all results on Processor 0 and have it print out the entire results matrix if command line args indicate this is necessary.
• Verify that the output of your MPI version is identical to the output of the serial version which is provided. There are a series of automated tests that help with this which are described later.
• To be compatible with the automated tests, heat_mpi must produce an exit code of 0; e.g. return 0 at the end of main() as is done in heat_serial.c.

• Your MPI version is only required to work correctly in the following situations:

  • The width of the rod in elements is evenly divisible by the number of processors being run.
  • The width of the rod is at least three times the number of processors so that each processor would have at least 3 columns associated with it.

  That means the following configurations should work or fail as indicated.

  #Procs	Width	Works?	Notes
  1	1	no	not enough cols
  1	2	no	not enough cols
  1	3	yes	take special care for 1 proc
  4	4	no	only 1 column per proc
  4	8	no	only 2 columns per proc
  4	12	yes	at least 3 cols per proc
  4	16	yes	at least 3 cols per proc
  4	15	no	uneven cols
  3	9	yes	3 cols per proc, evenly divisible
  4	40	yes	evenly divisible, >= 3 cols per proc

  Runs that are marked with "no" in the "Works?" column will not be tested so are free to do anything (segfault, work correctly, print an error and exit immediately, etc.).

Included with the project code is the file A2-WRITEUP.txt which has a timing table to fill in and a few discussion questions which should be answered.

Time your runs on MSI's Mesabi Cluster. You can SSH to it via ssh myX500@mesabi.msi.umn.edu. Gathering data for the timing table is eased via the provided heat-slurm.sh job script which will run jobs with each of the parameters in the timing table listed. Timing results are printed with the string runtime: prepended to each line. See the Earlier examples of how to use sbatch, check the queue via squeue and find the output files for the completed jobs.

A battery of automated tests are provided to evaluate whether heat_mpi is producing correct results on some small examples. These are present in the file test_heat.org and are run via the testy script. This can be done manually or via make test-prob1. Compliant programs will give results that look like the following.

>> unset DEBUG                  # enusre that DEBUG output is disabled

>> make test-prob1              # build prob1 program and run tests
mpicc -g -Wall -o heat_mpi heat_mpi.c
./testy test_heat.org
============================================================
== testy test_heat.org
== Running 10 / 10 tests
1)  Procs=1 Width=20           : ok
2)  Procs=1 Width=20 Valgrind  : ok
3)  Procs=2 Width=20           : ok
4)  Procs=2 Width=20 Valgrind  : ok
5)  Procs=2 Width=20 No output : ok
6)  Procs=2 Width=6            : ok
7)  Procs=2 Width=6 Valgrind   : ok
8)  Procs=4 Width=20           : ok
9)  Procs=4 Width=20 Valgrind  : ok
10) Procs=4 Steps=30 Width=40  : ok
============================================================
RESULTS: 10 / 10 tests passed

Failed tests will provide a results file with information that can be studied to gain insight into detected problems with the programs.

Tests are limited to 4 processors max. Some tests run codes under Valgrind to detect memory problems and help diagnose segmentation faults.

1. kmeans_mpi will accept the same command line arguments as the Serial version. As all MPI programs, it will be launched via mpirun such as in

   mpirun -np 4 kmeans_mpi mnist-data/digits_all_1e2.txt 10 outdir1
   

2. The behavior and output of the MPI version will match the serial version in format; e.g. printed messages and produced output files.

3. All I/O will be performed only by the root processor. Only the root Processor 0 will read the data file specified for the program. It will send required information to other processors that are needed by them to contribute to the communication. At the end of the computation, the root processor will be sent all cluster assignments and cluster centers to be written to files by the root processor.

   This requirement is for learning purposes only: in a production setting, one would look to have each processor perform I/O operations in parallel if the system supported it. Such parallel I/O would be facilitated by a binary storage format rather than our text-based version. However, this assignment is designed to gain practice using MPI calls and distributing data then accumulating presents a good opportunity for this.

4. The algorithm implemented in the MPI version will follow an input partitioning approach as was discussed in lecture. Other possibilities exist but input partitioning is the most likely to be generally useful and lead to speedups.

Some of the pictures below mention 'Assignment' which is now 'Project' and may mention some files that are not part of the current project. The process of uploading submission is otherwise the same.

1. In a terminal, change to your project code directory and type make zip which will create a zip file of your code. A session should look like this:

      > cd Desktop/5451/a2-code      # location of assignment code
   
      > ls 
      Makefile    dense_pagerank_mpi.c    heat_serial.c
      ...
   
      > make zip                     # create a zip file using Makefile target
      rm -f a2-code.zip
      cd .. && zip "a2-code/p1-code.zip" -r "a2-code"
        adding: a2-code/ (stored 0%)
        adding: a2-code/Makefile (deflated 68%)
        adding: a2-code/dense_pagerank_mpi.c (deflated 69%)
        adding: a2-code/test_dense_pagerank_mpi.org (deflated 71%)
        ...
      Zip created in a2-code.zip
   
      > ls a2-code.zip
      a2-code.zip
   

2. Log into Gradescope and locate and click 'Assignment 2' which will open up submission

   

3. Click on the 'Drag and Drop' text which will open a file selection dialog; locate and choose your a2-code.zip file

   

4. This will show the contents of the Zip file and should include your C source files along with testing files and directories.

   

5. Click 'Upload' which will show progress uploading files. It may take a few seconds before this dialog closes to indicate that the upload is successful. Note: there is a limit of 256 files per upload; normal submissions are not likely to have problems with this but you may want to make sure that nothing has gone wrong such as infinite loops creating many files or incredibly large files.

   WARNING: There is a limit of 256 files per zip. Doing make zip will warn if this limit is exceeded but uploading to Gradescope will fail without any helpful messages if you upload more the 256 files in a zip.

   

6. Once files have successfully uploaded, the Autograder will begin running the command line tests and recording results. These are the same tests that are run via make test.

   

7. When the tests have completed, results will be displayed summarizing scores along with output for each batch of tests.

   

8. Don't forget to add you partner to your submission after uploading. Only one partner needs to submit the code.

You may wish to review the policy on late project submission which will cost 1 Engagement Point per day late. No projects will be accepted more than 48 hours after the deadline.

https://www-users.cs.umn.edu/~kauffman/5451/syllabus.html