Table of Contents

Intercomm 2.0

InterComm-2.0 uses MPI for inter-node communication via P^nMPI.  PnMPI is distributed & automatically built with the InterComm-2.0 release.

There is a bug with OpenMPI v 1.4.3 which interferes with InterComm:

"- Modified a memcpy() call in the openib btl connection setup to use
memmove() instead because of the possibility of an overlapping
copy (as identified by valgrind)."

  1. Download
    1. via FTP (coming soon)
      cd $INSTALL_DIR/src
      wget http://cism.hao.ucar.edu/files/InterComm-2.0.tar.gz
      tar -zxvf InterComm-2.0.tar.gz
      
    2. via SVN (login required)
      svn checkout https://proxy.subversion.ucar.edu/cism_CSE/trunk/InterComm-2.0-wilsone
      
  2. Configure, build and install
    ./configure CC=$CC CXX=$CXX FC=$FC MPICC=$MPICC MPICXX=$MPICXX MPIF90=$MPIF90 \
                --with-ppp=${INSTALL_DIR}/P++/install \
                --prefix=${INSTALL_DIR}/InterComm-2.0
    gmake
    gmake install
    
  3. You are now ready to install the next prerequisite, Overture.

User Guide

Download the InterComm-2.0 user guide.

Troubleshooting

InterComm-2.0 was developed at the University of Maryland by the Chaos group. The code is unsupported and debugging InterComm can be full of chaos. The library works very well on some systems (typically using the Intel-12.x and OpenMPI-1.4.4). However, the code can fail in a variety of confusing ways. If you happen to see an error message, it might look like

IC_Setprogname: dlopen error
ERROR: NULL IC_getSubComm pointer from virtual
IC_Initialize failed at rank : 0

If InterComm still fails after trying all the obvious things (ie. try source $INTERCOMM/lib/build.env to set proper environment variables), then you may need to get your hands dirty. The first step is to build InterComm-2.0 with debug messages enabled. Extract a fresh copy of InterComm-2.0 source code and pass along an additional flag to configure: the --enable-debug flag.

Debugging PnMPI

When --enable-debug doesn't give you enough information, you can try enabling PnMPI debug messages. Try setting DEBUGLEVEL in src/pnmpi/common/Makefile.common. Sadly, there is no documentation on this. However, looking at the PnMPI source for debug.h sheds some light. There are several levels of verbosity you can set in a hexadecimal system:

#define DBGLEVEL1  0x0001  /* entry and exit prints */
#define DBGLEVEL2  0x0002  /* module load and instantiation */
#define DBGLEVEL3  0x0004  /* entry and exit of layers */
#define DBGLEVEL4  0x0008  /* arguments and parse information */

For very verbose messages, sum the DBGLEVEL hex numbers. The max verbosity would have DEBUGLEVEL=0x000F (1+2+4+8) in src/pnmpi/common/Makefile.common.

Testing PnMPI

There is a sample program distributed with PnMPI. Look in src/pnmpi/demo. Use gmake to build executables. Run programs with

This can be useful to determine where there's a problem with InterComm or its PnMPI dependency.