Merge pull request #185 from coding-for-reproducible-research/update_parallel_computing_course

Integrate PR from old parallel computing course into new material

Author: liamjberrisford

individual_modules/parallel_computing/collective_comms.ipynb

@@ -114,7 +114,7 @@
     "\n",
     "## Global MPI operations\n",
     "\n",
-    "For distributed memory problems, its difficult to get a holistic view of your entire data set as it doesnt exist in any one place. This means that performing global operations such as calculating the sum or product of a distributed data set also requires MPI. Fortunately, MPI has several functions that make this easier. Lets create a large set of data and scatter it across our processes, as before:\n",
+    "For distributed memory problems, it's difficult to get a holistic view of your entire data set as it doesn't exist in any one place. This means that performing global operations such as calculating the sum or product of a distributed data set also requires MPI. Fortunately, MPI has several functions that make this easier. Lets create a large set of data and scatter it across our processes, as before:\n",
     "\n",
     "```python\n",
     "if comm.Get_rank() == 0:\n",

individual_modules/parallel_computing/complete_files/multiprocessing_fractal.py

@@ -0,0 +1,50 @@
+# Run with command:
+# $ python multiprocessing_fractal.py
+import numpy as np
+import warnings
+import matplotlib.pyplot as plt
+from functools import partial
+from multiprocessing import Pool
+
+def complex_grid(extent, n_cells):
+    mesh_range = np.arange(-extent, extent, extent/n_cells)
+    x, y = np.meshgrid(mesh_range * 1j, mesh_range)
+    z = x + y
+
+    return z
+
+
+def julia_set(grid, num_iter, c):
+
+    fractal = np.zeros(np.shape(grid))
+
+    # Iterate through the operation z := z**2 + c.
+    for j in range(num_iter):
+        # Catch the warnings because they are annoying
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
+            grid = grid ** 2 + c
+            index = np.abs(grid) < np.inf
+        fractal[index] = fractal[index] + 1
+
+    return fractal
+
+c = -0.83 - 0.22 * 1j
+extent = 2
+cells = 2000
+
+grid = complex_grid(extent, cells)
+
+# Parameters for multiprocessing
+n_processes = 5
+n_slices = 2000
+
+# Split up the grid to distribute to processes
+sliced_grid = np.array_split(grid, n_slices)
+with Pool(processes=n_processes) as p:
+    fractals = p.map(partial(julia_set, num_iter=80, c=c) , sliced_grid)
+
+fractal = np.concatenate(fractals)
+
+#plt.imshow(fractal, extent=[-extent, extent, -extent, extent], aspect='equal')
+#plt.show()

individual_modules/parallel_computing/multiprocessing_fractal.ipynb

@@ -0,0 +1,96 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0d031ccb-54d9-44a8-8282-63064ec52ba0",
+   "metadata": {},
+   "source": [
+    "# Python Multiprocessing\n",
+    "\n",
+    "## Learning Objectives\n",
+    "\n",
+    "By the end of this lesson, learners will be able to:\n",
+    "\n",
+    "- Differentiate between message-passing and multiprocessing approaches in parallel programming.\n",
+    "- Implement Python's `multiprocessing` library to parallelize a fractal generation task within a single code instance.\n",
+    "- Set up a pool of worker processes using `Pool(processes=n_processes)` and delegate tasks across these processes with the `p.map()` function.\n",
+    "- Use `functools.partial` to manage function parameters that remain constant across parallel tasks, optimizing code reuse.\n",
+    "- Divide a computational grid into slices and assign each slice to a worker process to handle independently.\n",
+    "- Close a pool of processes in Python's multiprocessing model once tasks are completed, resuming the main program.\n",
+    "- Evaluate the performance of the multiprocessing approach by timing code execution with varying numbers of slices and processes, and compare results with the serial version in `fractal_complete.py`.\n",
+    "\n",
+    "\n",
+    "## Fractal example with Python multiprocessing\n",
+    "\n",
+    "In the previous lessons we have seen *message passing* being used to communicate data between multiple running instances of the code.\n",
+    "An alternative approach is to use *multi-processing*, where-by we launch one instance of our code which in turn launches new threads with access to the same memory.\n",
+    "\n",
+    "In `multiprocessing_fractal.py`, the previous fractal example has been implemented using `multiprocessing` from the python standard library.\n",
+    "Most of the code follows the same structure as the parallel fractal example.\n",
+    "\n",
+    "For the multi-processing model, we set up a *pool* of workers, `Pool(processes=n_processes)`, assigned to `p`.\n",
+    "The work can then be delegated out to these workers using the [`p.map()`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map) method.\n",
+    "This `map` method (equivalent to the builtin [`map`](https://docs.python.org/3/library/functions.html#map)) takes two arguments: a function to run (our fractal function), and a collection of inputs to pass to the function (different regions of the grid to be processed in parallel).\n",
+    "\n",
+    "```{note}\n",
+    "To pass in the parameters that don't change over grid regions, we've used [`functools.partial`](https://docs.python.org/3/library/functools.html#functools.partial):\n",
+    "\n",
+    "``` python\n",
+    "partial_julia_set = partial(julia_set, num_iter=80, c=-0.83 - 0.22 * 1j)\n",
+    "```\n",
+    "\n",
+    "This would be essentially equivalent to defining a new function:\n",
+    "\n",
+    "``` python\n",
+    "def partial_julia_set(grid):\n",
+    "    return julia_set(grid, num_iter=80, c=-0.83  -0.22 * 1j)\n",
+    "```\n",
+    "\n",
+    "You may be familiar with *lambda* expressions, but these cannot be passed in to the `multiprocessing.Pool.map` function.\n",
+    "In this script, we have split up the grid into `n_slices` vertical slices and assigned a pool of of `n_processes` workers.\n",
+    "These workers each take a slice, calculate the result saving the output into `fractals`, then work on a new slice.\n",
+    "When there are no more slices to work on, the pool is *closed* and the program resumes.\n",
+    "We can see how we can speed up the code by timing the full script running with different values of `n_slices` and `n_processes`.\n",
+    "Compare these numbers against the previous serial example in `fractal_complete.py`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b45c5b07-4d05-4705-93fe-fd841171e4cc",
+   "metadata": {},
+   "source": [
+    "# Complete File\n",
+    "[Download complete multiprocessing_fractal.py file](complete_files/multiprocessing_fractal.py)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b3b04584-c002-47f5-8f1c-66cb00ebe4d3",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

individual_modules/parallel_computing/parallel_fractal.ipynb

@@ -20,7 +20,7 @@
     "\n",
     "## Solving a problem in parallel\n",
     "\n",
-    "In the previous three sections we have built up a foundation enough to be able to tackle a simple problem in parallel. In this case, the problem we will attempt to solve is constructing a fractal. This kind of problem is often known as \"embarassingly parallel\" meaning that each element of the result has no dependency on any of the other elements, meaning that we can solve this problem in parallel without too much difficulty. Let's get started by creating a new script - `parallel_fractal.py```:\n",
+    "In the previous three sections we have built up a foundation enough to be able to tackle a simple problem in parallel. In this case, the problem we will attempt to solve is constructing a fractal. This kind of problem is often known as \"embarrassingly parallel\" meaning that each element of the result has no dependency on any of the other elements, meaning that we can solve this problem in parallel without too much difficulty. Let's get started by creating a new script - `parallel_fractal.py`:\n",
     "\n",
     "## Setting up our problem\n",
     "\n",
@@ -61,7 +61,7 @@
     "    return fractal\n",
     "```\n",
     "\n",
-    "This function calculates how many iterations it takes for each element in the complex grid to reach infinity (if ever) when operated on with the equation `x = x**2 + c```. The function itself is not the focus of this exercise as much as it is a way to make the computer perform some work! Let's use these functions to set up our problem in serial, without any parallelism:\n",
+    "This function calculates how many iterations it takes for each element in the complex grid to reach infinity (if ever) when operated on with the equation `x = x**2 + c`. The function itself is not the focus of this exercise as much as it is a way to make the computer perform some work! Let's use these functions to set up our problem in serial, without any parallelism:\n",
     "\n",
     "```python\n",
     "\n",
@@ -75,7 +75,7 @@
     "fractal = julia_set(grid, 80, c)\n",
     "```\n",
     "\n",
-    "If we run the python script (```python fractal.py```) it takes a few seconds to complete (this will vary depending on your machine), so we can already see that we are making our computer work reasonably hard with just a few lines of code. If we use the `time` command we can get a simple overview of how much time and resource are being used:\n",
+    "If we run the python script (`python fractal.py`) it takes a few seconds to complete (this will vary depending on your machine), so we can already see that we are making our computer work reasonably hard with just a few lines of code. If we use the `time` command we can get a simple overview of how much time and resource are being used:\n",
     "\n",
     "```\n",
     "$ time python parallel_fractal_complete.py\n",
@@ -137,7 +137,7 @@
     "mpirun -n 4 python parallel_fractal.py  37.23s user 21.70s system 370% cpu 15.895 total\n",
     "```\n",
     "\n",
-    "We can see that running the problem in parallel has greatly increased the speed of the function, but that the speed increase is directly proportional to the resource we are using (i.e. using 4 cores doesnt make the process 4 times faster). This is due to the increased overhead induced by MPI communication procedures, which can be quite expensive (as metioned in previous chapters).\n",
+    "We can see that running the problem in parallel has greatly increased the speed of the function, but that the speed increase is directly proportional to the resource we are using (i.e. using 4 cores doesn't make the process 4 times faster). This is due to the increased overhead induced by MPI communication procedures, which can be quite expensive (as mentioned in previous chapters).\n",
     "The way that a program performance changes based on the number of processes it runs on is often referred to as its \"scaling behaviour\". Determining how your problem scales across multiple processes is a useful exercise and is helpful when it comes to porting your code to a larger scale HPC machine.\n",
     "\n",
     "### Download Complete Parallel File \n",

individual_modules/parallel_computing/simple_communication.ipynb

@@ -57,25 +57,25 @@
     "```\n",
     "\n",
     "Now, if we run this script in parallel we no longer get the error, because the variable now exists on the second rank thanks to the `send`/`recv` methods.\n",
-    "In order to add an additional layer of safety to this process, we can add a tag to the message. This is an integer ID which ensures that the message is being recieved is being correctly used by the recieving process. This can be simply achieved by modifying the code to match the following:\n",
+    "In order to add an additional layer of safety to this process, we can add a tag to the message. This is an integer ID which ensures that the message is being received is being correctly used by the receiving process. This can be simply achieved by modifying the code to match the following:\n",
     "\n",
     "```python\n",
     "    comm.send(var, dest=1, tag=23)\n",
     "...\n",
     "    var = comm.recv(source=0, tag=23)\n",
     "```\n",
     "\n",
-    "The types of communications provided by the `send```/```recv` methods are known as blocking communications, as there is a chance that the send process won't return until it gets a signal that the data has been recieved successfully. This means that sending large amounts of data between processes can result in significant stoppages to the program. In practice, the standard for this is not implemented uniformly, so the blocking/non-blocking nature of the communication can be dynamic or depend on the size of the message being passed.\n",
+    "The types of communications provided by the `send```/```recv` methods are known as blocking communications, as there is a chance that the send process won't return until it gets a signal that the data has been received successfully. This means that sending large amounts of data between processes can result in significant stoppages to the program. In practice, the standard for this is not implemented uniformly, so the blocking/non-blocking nature of the communication can be dynamic or depend on the size of the message being passed.\n",
     "Before we start the next example, we can add the line `comm.barrier()` in our Python script to make sure that our processes only proceed once all other processes have reached this point, which will stop us getting confused about the output of our program.\n",
     "\n",
     "## Non-blocking communications\n",
     "\n",
-    "In some instances, it might make sense for communications to only be non-blocking, which will enable the sending rank to continue with its process without needing to wait for confirmation of a potentially large message to be recieved. In this case, we can use the explicitly non-blocking methods, `isend` and `irecv`.\n",
+    "In some instances, it might make sense for communications to only be non-blocking, which will enable the sending rank to continue with its process without needing to wait for confirmation of a potentially large message to be received. In this case, we can use the explicitly non-blocking methods, `isend` and `irecv`.\n",
     "The syntax is very similar for the sending process:\n",
     "```python\n",
     "    comm.send(var, dest=1, tag=23)\n",
     "```\n",
-    "but the recieving process has more to unpack. The `comm.irecv` method returns a request object, which can be unpacked with the `wait` method which then returns the data:\n",
+    "but the receiving process has more to unpack. The `comm.irecv` method returns a request object, which can be unpacked with the `wait` method which then returns the data:\n",
     "\n",
     "```python\n",
     "if comm.Get_rank() == 0:\n",

programme_information/parallel_computing.ipynb

@@ -167,7 +167,7 @@
     "\n",
     "### MPI for Python (MPI4Py)\n",
     "\n",
-    "The first part of this workshop is focussed on distributed memory parallelism with MPI, making use of the Python programming language. There are many different interfaces to MPI for many different languages, so we've chosen Python for the benefits it provides to write examples in an easy-to-understand format. Whilst the specific syntax of the commands learned in this part of the course wont be applicable across different languages, the overall code structures and concepts are highly transferrable, so once you have a solid grasp of the fundamentals of MPI you should be able to take thoses concepts to any language with an MPI interface and write parallel code!\n",
+    "The first part of this workshop is focussed on distributed memory parallelism with MPI, making use of the Python programming language. There are many different interfaces to MPI for many different languages, so we've chosen Python for the benefits it provides to write examples in an easy-to-understand format. Whilst the specific syntax of the commands learned in this part of the course wont be applicable across different languages, the overall code structures and concepts are highly transferable, so once you have a solid grasp of the fundamentals of MPI you should be able to take those concepts to any language with an MPI interface and write parallel code!\n",
     "\n",
     "The python package that we will be using in this course to implement MPI command is the [MPI4Py package](https://mpi4py.readthedocs.io/en/stable/), which can be installed via pip as follows:\n",
     "```\n",