Info Catalog fftw3: FFTW MPI Performance Tips fftw3: Distributed-memory FFTW with MPI fftw3: FFTW MPI Reference

fftw3: Combining MPI and Threads

 
 6.11 Combining MPI and Threads
 ==============================
 
 In certain cases, it may be advantageous to combine MPI
 (distributed-memory) and threads (shared-memory) parallelization.  FFTW
 supports this, with certain caveats.  For example, if you have a cluster
 of 4-processor shared-memory nodes, you may want to use threads within
 the nodes and MPI between the nodes, instead of MPI for all
 parallelization.
 
    In particular, it is possible to seamlessly combine the MPI FFTW
 routines with the multi-threaded FFTW routines (Multi-threaded
 FFTW).  However, some care must be taken in the initialization code,
 which should look something like this:
 
      int threads_ok;
 
      int main(int argc, char **argv)
      {
          int provided;
          MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
          threads_ok = provided >= MPI_THREAD_FUNNELED;
 
          if (threads_ok) threads_ok = fftw_init_threads();
          fftw_mpi_init();
 
          ...
          if (threads_ok) fftw_plan_with_nthreads(...);
          ...
 
          MPI_Finalize();
      }
 
    First, note that instead of calling 'MPI_Init', you should call
 'MPI_Init_threads', which is the initialization routine defined by the
 MPI-2 standard to indicate to MPI that your program will be
 multithreaded.  We pass 'MPI_THREAD_FUNNELED', which indicates that we
 will only call MPI routines from the main thread.  (FFTW will launch
 additional threads internally, but the extra threads will not call MPI
 code.)  (You may also pass 'MPI_THREAD_SERIALIZED' or
 'MPI_THREAD_MULTIPLE', which requests additional multithreading support
 from the MPI implementation, but this is not required by FFTW.) The
 'provided' parameter returns what level of threads support is actually
 supported by your MPI implementation; this _must_ be at least
 'MPI_THREAD_FUNNELED' if you want to call the FFTW threads routines, so
 we define a global variable 'threads_ok' to record this.  You should
 only call 'fftw_init_threads' or 'fftw_plan_with_nthreads' if
 'threads_ok' is true.  For more information on thread safety in MPI, see
 the MPI and Threads
 (http://www.mpi-forum.org/docs/mpi-20-html/node162.htm) section of the
 MPI-2 standard.
 
    Second, we must call 'fftw_init_threads' _before_ 'fftw_mpi_init'.
 This is critical for technical reasons having to do with how FFTW
 initializes its list of algorithms.
 
    Then, if you call 'fftw_plan_with_nthreads(N)', _every_ MPI process
 will launch (up to) 'N' threads to parallelize its transforms.
 
    For example, in the hypothetical cluster of 4-processor nodes, you
 might wish to launch only a single MPI process per node, and then call
 'fftw_plan_with_nthreads(4)' on each process to use all processors in
 the nodes.
 
    This may or may not be faster than simply using as many MPI processes
 as you have processors, however.  On the one hand, using threads within
 a node eliminates the need for explicit message passing within the node.
 On the other hand, FFTW's transpose routines are not multi-threaded, and
 this means that the communications that do take place will not benefit
 from parallelization within the node.  Moreover, many MPI
 implementations already have optimizations to exploit shared memory when
 it is available, so adding the multithreaded FFTW on top of this may be
 superfluous.
 

Info Catalog fftw3: FFTW MPI Performance Tips fftw3: Distributed-memory FFTW with MPI fftw3: FFTW MPI Reference