From 0a28aeed62c3ea7cfe7d5b74d761f92d56c91c54 Mon Sep 17 00:00:00 2001
From: Craig Warren <craig_warren@me.com>
Date: Wed, 21 Jun 2017 14:31:41 +0100
Subject: [PATCH] Added info on using MPI to task farm to multiple GPUs on a
 node.

---
 docs/source/gpu.rst | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/docs/source/gpu.rst b/docs/source/gpu.rst
index f1e191f5..43ad9f2c 100644
--- a/docs/source/gpu.rst
+++ b/docs/source/gpu.rst
@@ -1,8 +1,8 @@
 .. _gpu:
 
-**********
-Using GPUs
-**********
+*****
+GPGPU
+*****
 
 The most computationally intensive parts of gprMax, which are the FDTD solver loops, can optionally be executed using General-purpose computing on graphics processing units (GPGPU). This has been achieved through use of the NVIDIA CUDA programming environment, therefore a `NVIDIA CUDA-Enabled GPU <https://developer.nvidia.com/cuda-gpus>`_ is required to take advantage of the GPU-based solver.
 
@@ -12,11 +12,11 @@ Extra installation steps for GPU usage
 The following steps provide guidance on how to install the extra components to allow gprMax to run on your GPU:
 
 1. Install the `NVIDIA CUDA Toolkit <https://developer.nvidia.com/cuda-toolkit>`_. You can follow the Installation Guides in the `NVIDIA CUDA Toolkit Documentation <http://docs.nvidia.com/cuda/index.html#installation-guides>`_
-2. Install the pycuda Python module. Open a Terminal (Linux/macOS) or Command Prompt (Windows), navigate into the top-level gprMax directory, and if it is not already active, activate the gprMax conda environment :code:`source activate gprMax` (Linux/macOS) or :code:`activate gprMax` (Windows). Run :code:`conda install pycuda`
+2. Install the pycuda Python module. Open a Terminal (Linux/macOS) or Command Prompt (Windows), navigate into the top-level gprMax directory, and if it is not already active, activate the gprMax conda environment :code:`source activate gprMax` (Linux/macOS) or :code:`activate gprMax` (Windows). Run :code:`pip install pycuda`
 
 
-Run a test model
-================
+Running gprMax using GPU(s)
+===========================
 
 Open a Terminal (Linux/macOS) or Command Prompt (Windows), navigate into the top-level gprMax directory, and if it is not already active, activate the gprMax conda environment :code:`source activate gprMax` (Linux/macOS) or :code:`activate gprMax` (Windows)
 
@@ -25,3 +25,18 @@ Run one of the test models:
 .. code-block:: none
 
     (gprMax)$ python -m gprMax user_models/cylinder_Ascan_2D.in -gpu
+
+
+Combining MPI and GPU usage
+---------------------------
+
+Message Passing Interface (MPI) has been utilised to implement a simple task farm that can be used to distribute a series of models as independent tasks. This is described in more detail in the :ref:`OpenMP, MPI, HPC section <openmp-mpi>`. MPI can be combined with the GPU functionality to allow a series models to be distributed to multiple GPUs on the same machine (node). For example, to run a B-scan that contains 60 A-scans (traces) on a system with 4 GPUs:
+
+.. code-block:: none
+
+    (gprMax)$ python -m gprMax user_models/cylinder_Bscan_2D.in -n 60 -mpi 5 -gpu
+
+.. note::
+
+    The argument given with `-mpi` is number of MPI tasks, i.e. master + workers, for MPI task farm. So in this case, 1 master (CPU) and 4 workers (GPU cards).
+