MPICH is a free, open-source implementation of MPI, an industry standard Message Passing Interface. Code written with MPI library calls is now portable to virtually all computer platforms with only a recompilation. MPICH comes from Argonne National Labs and Mississippi State University. We currently have version 1.2.5.2 installed, this is the most recent version as of 13 January 2004. Since we have dual processor machines on our Beowulf clusters, we have configured MPICH to use shared memory communication inside of a node and TCP/IP (ethernet) communication to other nodes in the cluster (for the interested, this corresponds to the "ch_p4" device with "-comm=shared" option).
The basic MPI commands are mpicc, mpif77, mpif90, and mpirun. To compile, use mpicc just as you would gcc - it accepts the same options for optimization, debug, etc. The mpicc script will append a number of search directories for the MPI libraries and headers, and then call gcc. Note that you should NOT include your own options to find the MPI libraries or headers - this will be done automatically. The script mpif77 which will compile Fortran 77 programs. For Fortran 90, use mpif90.
To compile an MPI code for MPICH, use:
mpicc -O -o runme runme.c
where -O is the usual compiler optimization switch, and runme is the program to be compiled. Since mpicc is just a wrapper to a sequential compiler (maybe gcc or Intel's icc), it accepts all the usual switches and options. It will add the necessary search paths for the MPI libraries and headers, so you don't need to worry about where they are.
If you are doing it "by hand," then you may need something like the following:
gcc -I/opt/mpich/include -o runme runme.c -L/opt/mpich/lib -lmpich
Of course, you'll need to add any other compiler optimizations or libraries that you may need.
To use MPICH with the Intel compilers you can do one of two things:
At the compile line:
mpicc -cc=/opt/intel/bin/icc ...args...
mpif77 -fc=/opt/intel/bin/ifc ...args...
In your .cshrc file:
setenv MPICH_CC /opt/intel/bin/icc
setenv MPICH_CLINKER /opt/intel/bin/icc
setenv MPICH_F77 /opt/intel/bin/ifc
setenv MPICH_FLINKER /opt/intel/bin/ifc
To force the use of the Gnu compilers, use:
mpicc -cc=/usr/bin/gcc ...args...
mpif77 -fc=/usr/bin/g77 ...args...
Or similar changes to your .cshrc file.
To run an MPI code under SGE, you'll need to make an SQE submission script first. See the SGE Queuing System web page for details on how to create such a script. You will use the following to launch your parallel job:
mpirun -np $NSLOTS -machinefile $TMPDIR/machines runme -arg1 -arg2
where -arg1 and -arg2 are arguments to your code (if needed). The -np and -machinefile arguments are for the mpirun program itself. The variable $NSLOTS is set automatically by SGE to the number of "slots" or CPUs that you were actually given (recall that you can request a range of CPUs from SGE, $NSLOTS says how many you were actually given).
A list of the actual machines you were given is in a file in $TMPDIR/machines. This is a simple text file with one hostname per line, it is used by MPICH to spawn your jobs on the proper machines (otherwise, our automated job-killer program might remove your remote processes).
If you get errors from MPICH saying that p4_shmalloc returned NULL, you may need to alter the amount of memory used by MPICH. The P4_GLOBMEMSIZE variable can be made larger by changing your .cshrc file and adding:
setenv P4_GLOBMEMSIZE 16777216
This is the max amount that you can use under Linux. Note that increasing this amount reduces the amount of memory left for your program to use, so do not raise P4_GLOBMEMSIZE unless you need to.