Technical Appendix (#79)

* work in progress

* dis

Doesn't feel worth adding a cross-reference for this section.

* Understanding memory

* Move hashing data structures to technical appendix.

* Fix some dead links etc on review.

* Update episodes/optimisation-data-structures-algorithms.md

Co-authored-by: Jost Migenda <[email protected]>

* Update episodes/optimisation-data-structures-algorithms.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/technical-appendix.md

Co-authored-by: Jost Migenda <[email protected]>

* Update episodes/optimisation-data-structures-algorithms.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/technical-appendix.md

Co-authored-by: Jost Migenda <[email protected]>

* Update episodes/optimisation-latency.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/technical-appendix.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/technical-appendix.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/technical-appendix.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/acknowledgements.md

Co-authored-by: Jost Migenda <[email protected]>

* Update learners/technical-appendix.md

Co-authored-by: Jost Migenda <[email protected]>

---------

Co-authored-by: Jost Migenda <[email protected]>

Author: Robadob

config.yaml

@@ -70,13 +70,14 @@ episodes:
 - long-break1.md
 - optimisation-numpy.md
 - optimisation-use-latest.md
-- optimisation-memory.md
+- optimisation-latency.md
 - optimisation-conclusion.md
 
 # Information for Learners
 learners: 
 - setup.md
 - registration.md
+- technical-appendix.md
 - acknowledgements.md
 - ppp.md
 - reference.md

episodes/optimisation-conclusion.md

@@ -54,10 +54,9 @@ Your feedback enables us to improve the course for future attendees!
     - Where feasible, the latest version of Python and packages should be used as they can include significant free improvements to the performance of your code.
     - There is a risk that updating Python or packages will not be possible to due to version incompatibilities or will require breaking changes to your code.
     - Changes to packages may impact results output by your code, ensure you have a method of validation ready prior to attempting upgrades.
-- How the Computer Hardware Affects Performance
-    - Sequential accesses to memory (RAM or disk) will be faster than random or scattered accesses.
-      - This is not always natively possible in Python without the use of packages such as NumPy and Pandas
+- How Latency Affects Performance
     - One large file is preferable to many small files.
+    - Network requests can be parallelised to reduce the impact of fixed overheads.
     - Memory allocation is not free, avoiding destroying and recreating objects can improve performance.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::

episodes/optimisation-data-structures-algorithms.md

@@ -156,14 +156,13 @@ Since Python 3.6, the items within a dictionary will iterate in the order that t
 
 ### Hashing Data Structures
 
-Python's dictionaries are implemented as hashing data structures.
-Explaining how these work will get a bit technical, so let's start with an analogy:
+Python's dictionaries are implemented as hashing data structures, we can understand these at a high-level with an analogy:
 
 A Python list is like having a single long bookshelf. When you buy a new book (append a new element to the list), you place it at the far end of the shelf, right after all the previous books.
 
 ![A bookshelf corresponding to a Python list.](episodes/fig/bookshelf_list.jpg){alt="An image of a single long bookshelf, with a large number of books."}
 
-A hashing data structure is more like a bookcase with several shelves, labelled by genre (sci-fi, romance, children's books, non-fiction, …) and author surname. When you buy a new book by Jules Verne, you might place it on the shelf labelled "Sci-Fi, V–Z".
+A Python dictionary is more like a bookcase with several shelves, labelled by genre (sci-fi, romance, children's books, non-fiction, …) and author surname. When you buy a new book by Jules Verne, you might place it on the shelf labelled "Sci-Fi, V–Z".
 And if you keep adding more books, at some point you'll move to a larger bookcase with more shelves (and thus more fine-grained sorting), to make sure you don't have too many books on a single shelf.
 
 ![A bookshelf corresponding to a Python dictionary.](episodes/fig/bookshelf_dict.jpg){alt="An image of two bookcases, labelled "Sci-Fi" and "Romance". Each bookcase contains shelves labelled in alphabetical order, with zero or few books on each shelf."}
@@ -186,25 +185,14 @@ In practice, therefore, this trade-off between memory usage and speed is usually
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
+When a value is inserted into a dictionary, its key is hashed to decide on which "shelf" it should be stored. Most items will have a unique shelf, allowing them to be accessed directly. This is typically much faster for locating a specific item than searching a list.
 
-::::::::::::::::::::::::::::::::::::: callout
-
-### Technical explanation
-
-Within a hashing data structure each inserted key is hashed to produce a (hopefully unique) integer key.
-The dictionary is pre-allocated to a default size, and the key is assigned the index within the dictionary equivalent to the hash modulo the length of the dictionary.
-If that index doesn't already contain another key, the key (and any associated values) can be inserted.
-When the index isn't free, a collision strategy is applied. CPython's [dictionary](https://github.com/python/cpython/blob/main/Objects/dictobject.c) and [set](https://github.com/python/cpython/blob/main/Objects/setobject.c) both use a form of open addressing whereby a hash is mutated and corresponding indices probed until a free one is located.
-When the hashing data structure exceeds a given load factor (e.g. 2/3 of indices have been assigned keys), the internal storage must grow. This process requires every item to be re-inserted which can be expensive, but reduces the average probes for a key to be found.
-
-![An visual explanation of linear probing, CPython uses an advanced form of this.](episodes/fig/hash_linear_probing.png){alt="A diagram demonstrating how the keys (hashes) 37, 64, 14, 94, 67 are inserted into a hash table with 11 indices. This is followed by the insertion of 59, 80 and 39 which require linear probing to be inserted due to collisions."}
-
-To retrieve or check for the existence of a key within a hashing data structure, the key is hashed again and a process equivalent to insertion is repeated. However, now the key at each index is checked for equality with the one provided. If any empty index is found before an equivalent key, then the key must not be present in the data structure.
 
+::::::::::::::::::::::::::::::::::::: callout
 
 ### Keys
 
-Keys will typically be a core Python type such as a number or string. However, multiple of these can be combined as a Tuple to form a compound key, or a custom class can be used if the methods `__hash__()` and `__eq__()` have been implemented.
+A dictionary's keys will typically be a core Python type such as a number or string. However, multiple of these can be combined as a tuple to form a compound key, or a custom class can be used if the methods `__hash__()` and `__eq__()` have been implemented.
 
 You can implement `__hash__()` by utilising the ability for Python to hash tuples, avoiding the need to implement a bespoke hash function.
 

episodes/optimisation-memory.md episodes/optimisation-latency.mdepisodes/optimisation-memory.md renamed to episodes/optimisation-latency.md

@@ -1,83 +1,33 @@
 ---
-title: "Understanding Memory"
+title: "Understanding Latency"
 teaching: 30
 exercises: 0
 ---
 
 :::::::::::::::::::::::::::::::::::::: questions
 
-- How does a CPU look for a variable it requires?
-- What impact do cache lines have on memory accesses?
 - Why is it faster to read/write a single 100 MB file, than 100 files of 1 MB each?
+- How many orders of magnitude slower are disk accesses than RAM?
+- What's the cost of creating a list?
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 ::::::::::::::::::::::::::::::::::::: objectives
 
-- Able to explain, at a high-level, how memory accesses occur during computation and how this impacts optimisation considerations.
 - Able to identify the relationship between different latencies relevant to software.
+- Demonstrate how to implement parallel network requests.
+- Justify the re-use of existing variables over creating new ones.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
-## Accessing Variables
-
-The storage and movement of data plays a large role in the performance of executing software.
-
-<!-- Brief summary of hardware -->
-Modern computer's typically have a single processor (CPU), within this processor there are multiple processing cores each capable of executing different code in parallel.
-
-Data held in memory by running software is exists in RAM, this memory is faster to access than hard drives (and solid-state drives).
-But the CPU has much smaller caches on-board, to make accessing the most recent variables even faster.
-
-![An annotated photo of a computer's hardware.](episodes/fig/annotated-motherboard.jpg){alt="An annotated photo of inside a desktop computer's case. The CPU, RAM, power supply, graphics cards (GPUs) and harddrive are labelled."}
-
-<!-- Read/operate on variable ram->cpu cache->registers->cpu -->
-When reading a variable, to perform an operation with it, the CPU will first look in its registers. These exist per core, they are the location that computation is actually performed. Accessing them is incredibly fast, but there only exists enough storage for around 32 variables (typical number, e.g. 4 bytes).
-As the register file is so small, most variables won't be found and the CPU's caches will be searched.
-It will first check the current processing core's L1 (Level 1) cache, this small cache (typically 64 KB per physical core) is the smallest and fastest to access cache on a CPU.
-If the variable is not found in the L1 cache, the L2 cache that is shared between multiple cores will be checked. This shared cache, is slower to access but larger than L1 (typically 1-3MB per core).
-This process then repeats for the L3 cache which may be shared among all cores of the CPU. This cache again has higher latency to access, but increased size (typically slightly larger than the total L2 cache size).
-If the variable has not been found in any of the CPU's cache, the CPU will look to the computer's RAM. This is an order of magnitude slower to access, with several orders of magnitude greater capacity (tens to hundreds of GB are now standard).
-
-Correspondingly, the earlier the CPU finds the variable the faster it will be to access.
-However, to fully understand the cache's it's necessary to explain what happens once a variable has been found.
-
-If a variable is not found in the caches, it must be fetched from RAM.
-The full 64 byte cache line containing the variable, will be copied first into the CPU's L3, then L2 and then L1.
-Most variables are only 4 or 8 bytes, so many neighbouring variables are also pulled into the caches.
-Similarly, adding new data to a cache evicts old data.
-This means that reading 16 integers contiguously stored in memory, should be faster than 16 scattered integers
-
-Therefore, to **optimally** access variables they should be stored contiguously in memory with related data and worked on whilst they remain in caches.
-If you add to a variable, perform large amount of unrelated processing, then add to the variable again it will likely have been evicted from caches and need to be reloaded from slower RAM again.
-
-<!-- Latency/Throughput typically inversely proportional to capacity -->
-It's not necessary to remember this full detail of how memory access work within a computer, but the context perhaps helps understand why memory locality is important.
-
-![An abstract diagram showing the path data takes from disk or RAM to be used for computation.](episodes/fig/hardware.png){alt='An abstract representation of a CPU, RAM and Disk, showing their internal caches and the pathways data can pass.'}
-
-::::::::::::::::::::::::::::::::::::: callout
-
-Python as a programming language, does not give you enough control to carefully pack your variables in this manner (every variable is an object, so it's stored as a pointer that redirects to the actual data stored elsewhere).
-
-However all is not lost, packages such as `numpy` and `pandas` implemented in C/C++ enable Python users to take advantage of efficient memory accesses (when they are used correctly).
-
-:::::::::::::::::::::::::::::::::::::::::::::
-
-<!-- TODO python code example 
-```python
-
-```-->
 
 ## Accessing Disk
 
 <!-- Read data from a file it goes disk->disk cache->ram->cpu cache/s->cpu -->
-When accessing data on disk (or network), a very similar process is performed to that between CPU and RAM when accessing variables.
+When reading data from a file, it is first transferred from the disk to the disk cache and then to the RAM (the computer's main memory, where variables are stored).
+The latency to access files on disk is another order of magnitude higher than accessing normal variables.
 
-When reading data from a file, it transferred from the disk, to the disk cache, to the RAM.
-The latency to access files on disk is another order of magnitude higher than accessing RAM.
-
-As such, disk accesses similarly benefit from sequential accesses and reading larger blocks together rather than single variables.
+As such, disk accesses benefit from sequential accesses and reading larger blocks together rather than single variables.
 Python's `io` package is already buffered, so automatically handles this for you in the background.
 
 However before a file can be read, the file system on the disk must be polled to transform the file path to its address on disk to initiate the transfer (or throw an exception).
@@ -158,7 +108,7 @@ An even greater overhead would apply.
 
 ## Accessing the Network
 
-When transfering files over a network, similar effects apply. There is a fixed overhead for every file transfer (no matter how big the file), so downloading many small files will be slower than downloading a single large file of the same total size.
+When transferring files over a network, similar effects apply. There is a fixed overhead for every file transfer (no matter how big the file), so downloading many small files will be slower than downloading a single large file of the same total size.
 
 Because of this overhead, downloading many small files often does not use all the available bandwidth. It may be possible to speed things up by parallelising downloads.
 
@@ -227,7 +177,9 @@ Latency can have a big impact on the speed that a program executes, the below gr
 
 ![A graph demonstrating the wide variety of latencies a programmer may experience when accessing data.](episodes/fig/latency.png){alt="A horizontal bar chart displaying the relative latencies for L1/L2/L3 cache, RAM, SSD, HDD and a packet being sent from London to California and back. These latencies range from 1 nanosecond to 140 milliseconds and are displayed with a log scale."}
 
-The lower the latency typically the higher the effective bandwidth (L1 and L2 cache have 1 TB/s, RAM 100 GB/s, SSDs up to 32 GB/s, HDDs up to 150 MB/s), making large memory transactions even slower.
+L1/L2/L3 caches are where your most recently accessed variables are stored inside the CPU, whereas RAM is where most of your variables will be found.
+
+The lower the latency typically the higher the effective bandwidth (L1 and L2 cache have 1&nbsp;TB/s, RAM 100&nbsp;GB/s, SSDs up to 32 GB/s, HDDs up to 150&nbsp;MB/s), making large memory transactions even slower.
 
 ## Memory Allocation is not Free
 
@@ -335,9 +287,8 @@ Line #      Hits         Time  Per Hit   % Time  Line Contents
 
 ::::::::::::::::::::::::::::::::::::: keypoints
 
-- Sequential accesses to memory (RAM or disk) will be faster than random or scattered accesses.
-  - This is not always natively possible in Python without the use of packages such as NumPy and Pandas
 - One large file is preferable to many small files.
+- Network requests can be parallelised to reduce the impact of fixed overheads.
 - Memory allocation is not free, avoiding destroying and recreating objects can improve performance.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::

episodes/optimisation-using-python.md

@@ -150,134 +150,6 @@ operatorSearch: 28.43ms
 
 An easy approach to follow is that if two blocks of code do the same operation, the one that contains less Python is probably faster. This won't apply if you're using 3rd party packages written purely in Python though.
 
-::::::::::::::::::::::::::::::::::::: callout
-
-### Python bytecode
-
-
-You can use `dis` to view the bytecode generated by Python, the amount of byte-code more strongly correlates with how much code is being executed by the Python interpreter. However, this still does not account for whether functions called are implemented using Python or C.
-
-The pure Python search compiles to 82 lines of byte-code.
-
-```python
-import dis
-
-def manualSearch():
-    ls = generateInputs()
-    ct = 0
-    for i in range(0, int(N*M), M):
-        for j in range(0, len(ls)):
-            if ls[j] == i:
-                ct += 1
-                break
-
-dis.dis(manualSearch)
-```
-```output
- 11           0 LOAD_GLOBAL              0 (generateInputs)
-              2 CALL_FUNCTION            0
-              4 STORE_FAST               0 (ls)
-
- 12           6 LOAD_CONST               1 (0)
-              8 STORE_FAST               1 (ct)
-
- 13          10 LOAD_GLOBAL              1 (range)
-             12 LOAD_CONST               1 (0)
-             14 LOAD_GLOBAL              2 (int)
-             16 LOAD_GLOBAL              3 (N)
-             18 LOAD_GLOBAL              4 (M)
-             20 BINARY_MULTIPLY
-             22 CALL_FUNCTION            1
-             24 LOAD_GLOBAL              4 (M)
-             26 CALL_FUNCTION            3
-             28 GET_ITER
-        >>   30 FOR_ITER                24 (to 80)
-             32 STORE_FAST               2 (i)
-
- 14          34 LOAD_GLOBAL              1 (range)
-             36 LOAD_CONST               1 (0)
-             38 LOAD_GLOBAL              5 (len)
-             40 LOAD_FAST                0 (ls)
-             42 CALL_FUNCTION            1
-             44 CALL_FUNCTION            2
-             46 GET_ITER
-        >>   48 FOR_ITER                14 (to 78)
-             50 STORE_FAST               3 (j)
-
- 15          52 LOAD_FAST                0 (ls)
-             54 LOAD_FAST                3 (j)
-             56 BINARY_SUBSCR
-             58 LOAD_FAST                2 (i)
-             60 COMPARE_OP               2 (==)
-             62 POP_JUMP_IF_FALSE       38 (to 76)
-
- 16          64 LOAD_FAST                1 (ct)
-             66 LOAD_CONST               2 (1)
-             68 INPLACE_ADD
-             70 STORE_FAST               1 (ct)
-
- 17          72 POP_TOP
-             74 JUMP_FORWARD             1 (to 78)
-
- 15     >>   76 JUMP_ABSOLUTE           24 (to 48)
-        >>   78 JUMP_ABSOLUTE           15 (to 30)
-
- 13     >>   80 LOAD_CONST               0 (None)
-             82 RETURN_VALUE
-```
-
-Whereas the `in` variant only compiles to 54.
-
-```python
-import dis
-
-def operatorSearch():
-    ls = generateInputs()
-    ct = 0
-    for i in range(0, int(N*M), M):
-        if i in ls:
-            ct += 1
-
-dis.dis(operatorSearch)
-```
-```output
-  4           0 LOAD_GLOBAL              0 (generateInputs)
-              2 CALL_FUNCTION            0
-              4 STORE_FAST               0 (ls)
-
-  5           6 LOAD_CONST               1 (0)
-              8 STORE_FAST               1 (ct)
-
-  6          10 LOAD_GLOBAL              1 (range)
-             12 LOAD_CONST               1 (0)
-             14 LOAD_GLOBAL              2 (int)
-             16 LOAD_GLOBAL              3 (N)
-             18 LOAD_GLOBAL              4 (M)
-             20 BINARY_MULTIPLY
-             22 CALL_FUNCTION            1
-             24 LOAD_GLOBAL              4 (M)
-             26 CALL_FUNCTION            3
-             28 GET_ITER
-        >>   30 FOR_ITER                10 (to 52)
-             32 STORE_FAST               2 (i)
-
-  7          34 LOAD_FAST                2 (i)
-             36 LOAD_FAST                0 (ls)
-             38 CONTAINS_OP              0
-             40 POP_JUMP_IF_FALSE       25 (to 50)
-
-  8          42 LOAD_FAST                1 (ct)
-             44 LOAD_CONST               2 (1)
-             46 INPLACE_ADD
-             48 STORE_FAST               1 (ct)
-        >>   50 JUMP_ABSOLUTE           15 (to 30)
-
-  6     >>   52 LOAD_CONST               0 (None)
-             54 RETURN_VALUE
-```
-
-:::::::::::::::::::::::::::::::::::::::::::::
-
 
 ## Example: Parsing data from a text file