From f50e7ae218ca8321443cb8e2478d8f7a141c16b8 Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Fri, 14 Nov 2025 16:03:03 +0200 Subject: [PATCH 01/10] WIP: started porting Cython material from CSC HPC Python course --- content/cython.rst | 632 +++++++++++++++++++++++++ content/img/cython/fractal.svg | 32 ++ content/img/cython/unboxing-boxing.svg | 587 +++++++++++++++++++++++ content/index.rst | 2 + 4 files changed, 1253 insertions(+) create mode 100644 content/cython.rst create mode 100644 content/img/cython/fractal.svg create mode 100644 content/img/cython/unboxing-boxing.svg diff --git a/content/cython.rst b/content/cython.rst new file mode 100644 index 00000000..3818bf57 --- /dev/null +++ b/content/cython.rst @@ -0,0 +1,632 @@ +.. _cython: + +Cython +====== + +.. questions:: + + - Q1 + - Q2 +.. objectives:: + + - O1 + - O2 + +Interpreted languages like Python are rather slow to execute compared to +languages like C or Fortran that are compiled to machine code before execution. +Python in particular is both strongly typed and dynamically typed: this means +that all variables have a type that matters for operations that +can be performed on the variable, and that the type is determined only during +runtime by the Python interpreter. The interpreter does a lot of +"unboxing" of variable types when performing operations, and this comes with +significant overhead. For example, when just adding two integers + +.. code:: python + + a = 7 + b = 6 + c = a + b + +the Python interpreter needs to: + + 1. Check the types of both operands + 2. Check whether they both support the **+** operation + 3. Extract the function that performs the **+** operation (due to operator + overloading objects can have a custom definition for addition) + 4. Extract the actual values of the objects + 5. Perform the **+** operation + 6. Construct a new integer object for the result ("boxing") + +TODO non-transparent figure + + .. image:: img/cython/unboxing-boxing.svg + :class: center + :width: 90.0% + +Meanwhile in languages like C, the types are known at compilation time, which +allows the compiler to optimize many of the above steps away for better +performance at runtime. + +Scientific programs often include computationally expensive sections (e.g. +simulations of any kind). So how do we make Python execute our code faster in +these situations? Well that's the neat part: we don't! Instead, we write the +performance critical parts in a faster language and make them usable from +Python. + +Cython +------ + +Cython is an optimising static compiler for Python that also provides +its own programming language as a superset for standard Python. + +`Cython `__ is designed to provide C-like +performance for a code that is mostly written in Python by adding only a +few C-like declarations to an existing Python code. As such, Cython +provides the best of the both Python and C worlds: the good programmer +productivity of Python together with the high performance of C. +Especially for scientific programs performing a lot of numerical +computations, Cython can speed up the execution times more than an order +of magnitude. Cython makes it also easy to interact with external C/C++ +code. + +Cython works by transferring existing Python/Cython code into C code, +albeit in a form that is generally not easily readable by humans. The +resulting C-code calls functions from Python C application programming +interface (API), and thus requires an existing Python compiler and +runtime system. The Cython generated code is compiled into a Python +module. Normally, this module cannot be used as such, but needs to be +imported from a Python program that uses the functionality implemented +in Cython. + +The main mechanism of how Cython speeds up Python programs is by adding +static declarations for variables. Thus, one loses some of the dynamic +nature of Python when working with Cython. This works best for +fundamental data types (integers, floats, strings) and contiguous arrays +(such as NumPy arrays), operations on lists and dictionaries do not +normally benefit much from Cython. + +In summary, Cython can alleviate the following Python performance +bottlenecks discussed in Week 1: + +- Interpretation overhead +- Unboxing / boxing Python objects +- Overheads of Python control structures +- Function call overheads + +Creating Cython modules +------------------------ + +Normally, when working with Cython one does not Cythonize the whole +program but only selected modules. + +Suppose we have a Python module named **my_module.py** that defines a +function called **add**: + +.. code:: python + + def add(x, y): + result = x + y + return result + +This function could then be used from some other Python code for example as: + +.. code:: python + + from my_module import add + + z = add(4, 5) + +Cython can transform this Python code into an equivalent C-code utilizing +the Python API as: + +.. code:: bash + + $ cython my_module.py + +The result is a file **my_module.c**, which could be compiled into a Python +extension module using a C-compiler. One can investigate the generated **.c** +file but it is not really meant for humans to read (already this simple +function results in over 4000 lines of C code)! + +A typical Cython project is separated into plain Python modules (file +extension **.py**) and Cython modules (extension **.pyx**). One usually uses +established build tools to Cythonize and compile code that +estalished ubild tools to Cythonize and compile the **.pyx** files while +leaving the **.py** files as such. One common approach is to use +**setuptools** (see section on packaging) with a Cython stage specified in +**setup.py**: + +.. code:: python + + from setuptools import setup + from Cython.Build import cythonize + + setup( + name='My cool app', + ext_modules=cythonize("my_module.pyx"), + ) + +In larger real-world projects one would list all **.pyx** files here. + +One can then create the C-extension module with: + +.. code:: bash + + $ python3 setup.py build_ext --inplace + running build_ext + building 'my_module' extension + creating build + creating build/temp.linux-x86_64-3.6 + gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c my_module.c -o build/temp.linux-x86_64-3.6/my_module.o + gcc -pthread -shared ... + +where the ``--inplace`` option places the C-extension in the current +directory. The end result is a .so file containing the C-extension that +can be imported and used just the same as the pure Python module: + +.. code:: python + + from my_module import add + + z = add(4, 5) + +As the C-extension implements the fully dynamic Python code (just using +the Python C-API), transforming the pure Python module into C-extension +gives normally only very modest speed-ups. However, as we will discuss +in the following steps, by adding Cython language extensions into the +code (so it is no longer valid Python code) it is possible to achieve +much more significant performance improvements. + + +Adding static type information +------------------------------ + +What if one knows that e.g. in a certain function the variables have +always the same type? That's where Cython steps in: Cython allows one to +add static typing information so that boxing and unboxing are not +needed, and one can operate directly with the actual values. + +When Cythonizing a Python code, static type information can be added +either: + +- In function signatures by prefixing the formal arguments by their + type +- By declaring variables with the **cdef** Cython keyword, followed by + the the type + +For example, a simple Python function adding two objects could be +Cythonized as follows: + +.. code:: python + + def add (int x, int y): + cdef int result + result = x + y + return result + +The function works now only with integers but with less boxing/unboxing +overheads. + +The types provided in Cython code are C types, and the variables with +type information are pure C variables and not Python objects. When +calling a Cythonized function from Python, there is an automatic +conversion from the Python object of actual arguments to the C value of +formal argument, and when returning a C variable it is converted to +corresponding Python object. Automatic conversions are carried out also +in most cases within the Cython code where both Python objects and C +variables are involved. + +The table below lists the most common C types and their corresponding +Python types. More information can be found in the `Cython +documentation `__. + +================= ============= +From Python types To C types +================= ============= +int int, long +int, float float, double +str/bytes char \* +================= ============= + +============= =============== +From C types To Python types +============= =============== +int, long int +float, double float +char \* str/bytes +============= =============== + +“Boxing” +-------- + +- In Python, everything is an object + +.. image:: img/cython/unboxing-boxing.svg + :class: center + :width: 90.0% + +Static type declarations +------------------------ + +- Cython extended code should have .pyx ending + + - Cannot be run with normal Python + +- Types are declared with ``cdef`` keyword + + - In function signatures only type is given + +.. container:: column + + .. code:: python + + def integrate(f, a, b, N): + s = 0 + dx = (b - a) / N + for i in range(N): + s += f(a + i * dx) + return s * dx + +.. container:: column + + .. code:: python + + def integrate(f, double a, double b, int N): + cdef double s = 0 + cdef int i + cdef double dx = (b - a) / N + for i in range(N): + s += f(a + i * dx) + return s * dx + +.. _static-type-declarations-1: + +Static type declarations +------------------------ + +- Pure Python: 5.55 s +- Static type declarations in kernel: 100 ms + +.. container:: column + + .. code:: python + + def kernel(double zr, double zi, ...): + cdef int count = 0 + + while ((zr*zr + zi*zi) < (lim*lim)) + and count < cutoff: + zr = zr * zr - zi * zi + cr + zi = zr * zr - zi * zi + cr + count += 1 + + return count + +.. container:: column + + .. image:: img/cython/fractal.svg + :class: center + :width: 80.0% + +Function call overhead +---------------------- + +- Function calls in Python can involve lots of checking and “boxing” +- Overhead can be reduced by declaring functions to be C-functions + + - **cdef** keyword: functions can be called only from Cython + - **cpdef** keyword: generate also Python wrapper + +.. container:: column + + .. code:: python + + def integrate(f, a, b, N): + s = 0 + dx = (b - a) / N + for i in range(N): + s += f(a + i * dx) + return s * dx + +.. container:: column + + .. code:: python + + cdef double integrate(f, double a, ...): + cdef double s = 0 + cdef int i + cdef double dx = (b - a) / N + for i in range(N): + s += f(a + i * dx) + return s * dx + +Using C functions +----------------- + +- Static type declarations in kernel: 100 ms +- Kernel as C function: 69 ms + +.. container:: column + + .. code:: python + + cdef int kernel(double zr, double zi, ...): + cdef int count = 0 + while ((zr*zr + zi*zi) < (lim*lim)) + and count < cutoff: + zr = zr * zr - zi * zi + cr + zi = zr * zr - zi * zi + cr + count += 1 + return count + +.. container:: column + + .. image:: img/cython/fractal.svg + :class: center + :width: 80.0% + +NumPy arrays with Cython +------------------------- + +- Cython supports fast indexing for NumPy arrays +- Type and dimensions of array have to be declared + +.. code:: python + + import numpy as np # normal NumPy import + cimport numpy as cnp # import for NumPY C-API + + def func(): # declarations can be made only in function scope + cdef cnp.ndarray[cnp.int_t, ndim=2] data + data = np.empty((N, N), dtype=int) + + ... + + for i in range(N): + for j in range(N): + data[i,j] = ... # double loop is done in nearly C speed + +Compiler directives +------------------- + +- Compiler directives can be used for turning of certain Python + features for additional performance + + - boundscheck (False) : assume no IndexErrors + - wraparound (False): no negative indexing + - … + +.. code:: python + + import numpy as np # normal NumPy import + cimport numpy as cnp # import for NumPY C-API + + import cython + + @cython.boundscheck(False) + def func(): # declarations can be made only in function scope + cdef cnp.ndarray[cnp.int_t, ndim=2] data + data = np.empty((N, N), dtype=int) + +Final performance +----------------- + +- Pure Python: 5.5 s +- Static type declarations: 100 ms +- Kernel as C function: 69 ms +- Fast indexing and directives: 15 ms + +Where to add types? +------------------- + +- Typing everything reduces readibility and can even slow down the + performance +- Profiling should be first step when optimising + + +Profiling Cython code +--------------------- + +- By default, Cython code does not show up in profile produced by + cProfile +- Profiling can be enabled for entire source file or on per function + basis + +.. code:: python + + # cython: profile=True + import cython + + @cython.profile(False) + cdef func(): + ... + +.. code:: python + + # cython: profile=False + import cython + + @cython.profile(True) + cdef func(): + ... + +Summary +------- + +- Cython is optimising static compiler for Python +- Possible to add type declarations with Cython language +- Fast indexing for NumPy arrays +- At best cases, huge speed ups can be obtained + + - Some compromise for Python flexibility + +Further functionality in Cython +------------------------------- + +- Using C structs and C++ classes in Cython +- Exceptions handling +- Parallelisation (threading) with Cython +- … + +Interfacing external libraries +------------------------------ + +Increasing performance with compiled code +----------------------------------------- + +- There are Python interfaces for many high performance libraries +- However, sometimes one might want to utilize a library without Python + interface + + - Existing libraries + - Own code written in C or Fortran + +- Python C-API provides the most comprehensive way to extend Python +- CFFI, Cython, and f2py can provide easier approaches + +CFFI +---- + +- C Foreign Function Interface for Python +- Interact with almost any C code +- C-like declarations within Python + + - Can often be copy-pasted from headers / documentation + +- ABI and API modes + + - ABI does not require compilation + - API can be faster and more robust + - Only API discussed here + +- Some understanding of C required + +Creating Python interface to C library +-------------------------------------- + +- In API mode, CFFI is used for building a Python extension module that + provides interface to the library +- One needs to write a *build* script that specifies: + + - the library functions to be interfaced + - name of the Python extension + - instructions for compiling and linking + +- CFFI uses C compiler and creates the shared library +- The extension module can then be used from Python code. + +Example: Python interface to C math library +------------------------------------------- + +.. code:: python + + from cffi import FFI + ffibuilder = FFI() + + ffibuilder.cdef(""" + double sqrt(double x); // list all the function prototypes from the + double sin(double x); // library that we want to use + """) + + ffibuilder.set_source("_my_math", # name of the Python extension + """ + #include // Some C source, often just include + """, + library_dirs = [], # location of library, not needed for C + # C standard library + libraries = ['m'] # name of the library we want to interface + ) + + ffibuilder.compile(verbose=True) + +.. _example-python-interface-to-c-math-library-1: + +Example: Python interface to C math library +------------------------------------------- + +- Building the extension + +.. code:: bash + + python3 build_mymath.py + generating ./_mymath.c + running build_ext + building '_mymath' extension + ... + gcc -pthread -shared -Wl,-z,relro -g ./_mymath.o -L/usr/lib64 -lm -lpython3.6m + -o ./_mymath.cpython-36m-x86_64-linux-gnu.so + +- Using the extension + +.. code:: python + + from _mymath import lib + + a = lib.sqrt(4.5) + b = lib.sin(1.2) + +- Python ``float``\ s are automatically converted to C ``double``\ s + and back + +Passing NumPy arrays to C code +------------------------------ + +- Only simple scalar numbers can be automatically converted Python + objects and C types +- In C, arrays are passed to functions as pointers +- A “pointer” object to NumPy array can be obtained with ``cast`` and + ``from_buffer`` functions + +.. _passing-numpy-arrays-to-c-code-1: + +Passing NumPy arrays to C code +------------------------------ + +.. container:: column + + - C function adding two arrays + + .. code:: c + + // c = a + b + void add(double *a, double *b, double *c, int n) + { + for (int i=0; i + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/img/cython/unboxing-boxing.svg b/content/img/cython/unboxing-boxing.svg new file mode 100644 index 00000000..3085c75f --- /dev/null +++ b/content/img/cython/unboxing-boxing.svg @@ -0,0 +1,587 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + Object + Integer + + + int 7 + + + + otherobjectdata + + + + + + Object + Integer + + + int 6 + + + + otherobjectdata + + + Check the types:integers + + + + + + + int 7 + + + + + + + int 6 + + + + = + + + + + int 13 + + + + + Object + Integer + + + int 13 + + + + otherobjectdata + + + + + diff --git a/content/index.rst b/content/index.rst index ac71a0a0..89724915 100644 --- a/content/index.rst +++ b/content/index.rst @@ -87,6 +87,7 @@ to learn yourself as you need to. 45 min ; :doc:`dependencies` 30 min ; :doc:`binder` 60 min ; :doc:`packaging` + 30 min ; :doc:`cython` .. toctree:: @@ -113,6 +114,7 @@ to learn yourself as you need to. parallel packaging web-apis + cython .. toctree:: :maxdepth: 1 From 91a2f858f7db20f301097be0e14f6aed06df3946 Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Tue, 18 Nov 2025 17:11:54 +0200 Subject: [PATCH 02/10] Mostly finished draft of Cython section --- content/cython.rst | 683 ++++++++----------------- content/img/cython/fractal.svg | 32 -- content/img/cython/unboxing-boxing.svg | 265 ++++------ 3 files changed, 330 insertions(+), 650 deletions(-) delete mode 100644 content/img/cython/fractal.svg diff --git a/content/cython.rst b/content/cython.rst index 3818bf57..98bef95f 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -12,6 +12,17 @@ Cython - O1 - O2 + +.. callout:: + + Using Cython requires that you have a working environment for compiling + C code. This goes beyond the software requirements for this course, so the + teaching will be given in form of demonstrations and no exercises. + You may still follow along with the code examples if you have a C compiler + installed, in which case you can install Cython to your Conda environment + with `conda install cython`. + + Interpreted languages like Python are rather slow to execute compared to languages like C or Fortran that are compiled to machine code before execution. Python in particular is both strongly typed and dynamically typed: this means @@ -37,8 +48,6 @@ the Python interpreter needs to: 5. Perform the **+** operation 6. Construct a new integer object for the result ("boxing") -TODO non-transparent figure - .. image:: img/cython/unboxing-boxing.svg :class: center :width: 90.0% @@ -51,56 +60,43 @@ Scientific programs often include computationally expensive sections (e.g. simulations of any kind). So how do we make Python execute our code faster in these situations? Well that's the neat part: we don't! Instead, we write the performance critical parts in a faster language and make them usable from -Python. +Python. This is called extending Python, and usually involves writing C-code +with Python-specific boilerplate and compiling this as a shared library. + +Here we discuss one popular approach for extending Python with compiled code: +using a tool called Cython. Cython ------ -Cython is an optimising static compiler for Python that also provides -its own programming language as a superset for standard Python. - -`Cython `__ is designed to provide C-like -performance for a code that is mostly written in Python by adding only a -few C-like declarations to an existing Python code. As such, Cython -provides the best of the both Python and C worlds: the good programmer -productivity of Python together with the high performance of C. -Especially for scientific programs performing a lot of numerical -computations, Cython can speed up the execution times more than an order -of magnitude. Cython makes it also easy to interact with external C/C++ -code. - -Cython works by transferring existing Python/Cython code into C code, -albeit in a form that is generally not easily readable by humans. The -resulting C-code calls functions from Python C application programming -interface (API), and thus requires an existing Python compiler and -runtime system. The Cython generated code is compiled into a Python -module. Normally, this module cannot be used as such, but needs to be -imported from a Python program that uses the functionality implemented -in Cython. - -The main mechanism of how Cython speeds up Python programs is by adding -static declarations for variables. Thus, one loses some of the dynamic -nature of Python when working with Cython. This works best for -fundamental data types (integers, floats, strings) and contiguous arrays -(such as NumPy arrays), operations on lists and dictionaries do not -normally benefit much from Cython. - -In summary, Cython can alleviate the following Python performance -bottlenecks discussed in Week 1: - -- Interpretation overhead -- Unboxing / boxing Python objects -- Overheads of Python control structures -- Function call overheads - -Creating Cython modules ------------------------- +`Cython `__ is a framework for writing Python-like code +that can be processed with the Cython compiler to produce optimized code. +Cython is designed to provide C-like performance for code that is mostly +written in Python by adding only a few C-like declarations to existing +Python code. As such, Cython provides the best of the both worlds: +the good programmer productivity of Python together with the high performance +of C. Cython also makes it easy to interact with external C/C++ code. + +The Cython compiler processes code written in Python, or more +commonly the Cython extension of Python language, turns it into valid C-code +which is then compiled into a Python extension module using a C compiler +(GCC, Clang, MSVC, ...). The Cython programming language is a a superset of +Python that adds C-like static type declarations and other features that +make it possible to generate efficient machine code. + +.. callout:: -Normally, when working with Cython one does not Cythonize the whole -program but only selected modules. + Unlike plain Python code, Cython code must be compiled ahead of time before + it can be executed. This is usually done during the build phase of a + project. Note that Cython is *not* a just-in-time (JIT) compiler like e.g. + Numba, although you *can* call the Cython compiler at runtime for JIT-like + behavior if you really want to. -Suppose we have a Python module named **my_module.py** that defines a -function called **add**: + +Your first Cython module +------------------------ + +Suppose we have a Python module called **my_module.py** that contains: .. code:: python @@ -108,114 +104,124 @@ function called **add**: result = x + y return result -This function could then be used from some other Python code for example as: +Cython allows one to compile **my_module.py** directly to machine code while +still allowing its contents to be imported and used from Python code. We can +Cynthonize the module "manually" from command line: -.. code:: python +.. code:: bash - from my_module import add + $ cythonize -i my_module.py - z = add(4, 5) +This produces a file called **my_module.c**, full of C code. One can +investigate the generated **.c** file but it is not really meant for humans to +read, because of all the boilerplate that Cython adds in order to make the +compiled code available to Python. Already this simple function results in +over 7000 lines of C code! -Cython can transform this Python code into an equivalent C-code utilizing -the Python API as: +The option **-i** (meaning inplace) tells Cython to also compile the generated +**.c** file into an extension module in the same directory. +This could also be done manually by invoking a C-compiler of your choice. +On Linux/Mac systems the compiled module will be called something +like **my_module.cpython-314-x86_64-linux-gnu.so**, on Windows the suffix will +be **.pyd**. -.. code:: bash +The extension module can be imported from Python in the same way as one would +import a pure Python module, e.g.: - $ cython my_module.py +.. code:: python -The result is a file **my_module.c**, which could be compiled into a Python -extension module using a C-compiler. One can investigate the generated **.c** -file but it is not really meant for humans to read (already this simple -function results in over 4000 lines of C code)! + from my_module import add + z = add(4, 5) -A typical Cython project is separated into plain Python modules (file -extension **.py**) and Cython modules (extension **.pyx**). One usually uses -established build tools to Cythonize and compile code that -estalished ubild tools to Cythonize and compile the **.pyx** files while -leaving the **.py** files as such. One common approach is to use -**setuptools** (see section on packaging) with a Cython stage specified in -**setup.py**: -.. code:: python +Usually when working with Cython, one does not Cythonize the whole program but +only selected modules. A typical Cython project is separated into plain Python +modules (file suffix **.py**), and Cython code files (suffix **.pyx**). +The **.pyx** files will usually contain Cython-specific code like static type +information, so that they are not valid Python code anymore and must be +Cythonized before use. - from setuptools import setup - from Cython.Build import cythonize +.. callout:: - setup( - name='My cool app', - ext_modules=cythonize("my_module.pyx"), - ) + Real-world project don't usually invoke Cython from the command line and + instead use an established build tool like **setuptools** to handle the + Cythonization during the project's build phase. More info is available on + the `Cython documentation `__. + See also the course page on packaging. (TODO: link.) -In larger real-world projects one would list all **.pyx** files here. -One can then create the C-extension module with: +Using Cython with Jupyter +------------------------- -.. code:: bash +Jupyter has an `extension ` +for supporting Cython compilation directly inside notebooks, assuming your +environment has Cython installed. - $ python3 setup.py build_ext --inplace - running build_ext - building 'my_module' extension - creating build - creating build/temp.linux-x86_64-3.6 - gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c my_module.c -o build/temp.linux-x86_64-3.6/my_module.o - gcc -pthread -shared ... +We first load the Cython extension, e.g. in the very first cell: :: -where the ``--inplace`` option places the C-extension in the current -directory. The end result is a .so file containing the C-extension that -can be imported and used just the same as the pure Python module: + %load_ext Cython + +We can Cythonize cell contents using the magic `%%cython` and executing it: .. code:: python - from my_module import add + %%cython + def add(x, y): + result = x + y + return result - z = add(4, 5) -As the C-extension implements the fully dynamic Python code (just using -the Python C-API), transforming the pure Python module into C-extension -gives normally only very modest speed-ups. However, as we will discuss -in the following steps, by adding Cython language extensions into the -code (so it is no longer valid Python code) it is possible to achieve -much more significant performance improvements. + +There is also `%%cython --annotate` which is useful for analyzing the +generated C code. Adding static type information ------------------------------ -What if one knows that e.g. in a certain function the variables have -always the same type? That's where Cython steps in: Cython allows one to -add static typing information so that boxing and unboxing are not -needed, and one can operate directly with the actual values. +So far our Cythonized extension module is rather dumb. We have reduced some +of interpreting overhead by compiling the code, but it's still using Python's +fully dynamic type system with the same boxing and unboxing overhead as in +standard Python. This is because there are no type declarations in the code +that Cython could use for optimizations. When Cythonizing a Python code, static type information can be added either: - In function signatures by prefixing the formal arguments by their - type + type. - By declaring variables with the **cdef** Cython keyword, followed by - the the type + the the type. -For example, a simple Python function adding two objects could be -Cythonized as follows: +To make Cythonize a function that adds two integers and returns the result as +an integer, we would write: .. code:: python - def add (int x, int y): + def add(int x, int y): cdef int result result = x + y return result The function works now only with integers but with less boxing/unboxing -overheads. +overheads. Store this as **my_module.pyx** (note the file extension) and +Cythonize as before: + +.. code:: bash + + $ cythonize -i my_module.pyx -The types provided in Cython code are C types, and the variables with -type information are pure C variables and not Python objects. When -calling a Cythonized function from Python, there is an automatic -conversion from the Python object of actual arguments to the C value of -formal argument, and when returning a C variable it is converted to -corresponding Python object. Automatic conversions are carried out also -in most cases within the Cython code where both Python objects and C -variables are involved. +Import this into Python and confirm that it works as expected with integers. +However, if passing floating-point numbers the function is forced to interpret +the inputs as integers before performing the addition. For example, +**add(1.2, 2.7)** would return 3. This happens because there is an automatic +conversion from the input Python objects (floating point numbers) to the +declared C-types when calling the Cythonized function from Python. +Similarly the returned C variable is converted to a corresponding Python +object. +To make the function work with floats we'd instead declare the types to be +either **float** (32-bit) or **double** (64-bit) type instead of **int**. The table below lists the most common C types and their corresponding Python types. More information can be found in the `Cython documentation `__. @@ -236,395 +242,150 @@ float, double float char \* str/bytes ============= =============== -“Boxing” --------- - -- In Python, everything is an object - -.. image:: img/cython/unboxing-boxing.svg - :class: center - :width: 90.0% - -Static type declarations ------------------------- - -- Cython extended code should have .pyx ending - - - Cannot be run with normal Python - -- Types are declared with ``cdef`` keyword - - - In function signatures only type is given - -.. container:: column - - .. code:: python - - def integrate(f, a, b, N): - s = 0 - dx = (b - a) / N - for i in range(N): - s += f(a + i * dx) - return s * dx - -.. container:: column - - .. code:: python - - def integrate(f, double a, double b, int N): - cdef double s = 0 - cdef int i - cdef double dx = (b - a) / N - for i in range(N): - s += f(a + i * dx) - return s * dx - -.. _static-type-declarations-1: - -Static type declarations ------------------------- - -- Pure Python: 5.55 s -- Static type declarations in kernel: 100 ms - -.. container:: column - - .. code:: python - - def kernel(double zr, double zi, ...): - cdef int count = 0 - - while ((zr*zr + zi*zi) < (lim*lim)) - and count < cutoff: - zr = zr * zr - zi * zi + cr - zi = zr * zr - zi * zi + cr - count += 1 - - return count - -.. container:: column - - .. image:: img/cython/fractal.svg - :class: center - :width: 80.0% - -Function call overhead ----------------------- - -- Function calls in Python can involve lots of checking and “boxing” -- Overhead can be reduced by declaring functions to be C-functions - - - **cdef** keyword: functions can be called only from Cython - - **cpdef** keyword: generate also Python wrapper - -.. container:: column - - .. code:: python - - def integrate(f, a, b, N): - s = 0 - dx = (b - a) / N - for i in range(N): - s += f(a + i * dx) - return s * dx - -.. container:: column - - .. code:: python - - cdef double integrate(f, double a, ...): - cdef double s = 0 - cdef int i - cdef double dx = (b - a) / N - for i in range(N): - s += f(a + i * dx) - return s * dx - -Using C functions ------------------ - -- Static type declarations in kernel: 100 ms -- Kernel as C function: 69 ms - -.. container:: column - - .. code:: python - - cdef int kernel(double zr, double zi, ...): - cdef int count = 0 - while ((zr*zr + zi*zi) < (lim*lim)) - and count < cutoff: - zr = zr * zr - zi * zi + cr - zi = zr * zr - zi * zi + cr - count += 1 - return count - -.. container:: column - - .. image:: img/cython/fractal.svg - :class: center - :width: 80.0% - -NumPy arrays with Cython -------------------------- - -- Cython supports fast indexing for NumPy arrays -- Type and dimensions of array have to be declared - -.. code:: python - - import numpy as np # normal NumPy import - cimport numpy as cnp # import for NumPY C-API - - def func(): # declarations can be made only in function scope - cdef cnp.ndarray[cnp.int_t, ndim=2] data - data = np.empty((N, N), dtype=int) - - ... - - for i in range(N): - for j in range(N): - data[i,j] = ... # double loop is done in nearly C speed - -Compiler directives -------------------- - -- Compiler directives can be used for turning of certain Python - features for additional performance - - - boundscheck (False) : assume no IndexErrors - - wraparound (False): no negative indexing - - … - -.. code:: python - - import numpy as np # normal NumPy import - cimport numpy as cnp # import for NumPY C-API - - import cython - - @cython.boundscheck(False) - def func(): # declarations can be made only in function scope - cdef cnp.ndarray[cnp.int_t, ndim=2] data - data = np.empty((N, N), dtype=int) - -Final performance ------------------ - -- Pure Python: 5.5 s -- Static type declarations: 100 ms -- Kernel as C function: 69 ms -- Fast indexing and directives: 15 ms - -Where to add types? -------------------- - -- Typing everything reduces readibility and can even slow down the - performance -- Profiling should be first step when optimising +Using Numpy arrays with Cython +------------------------------ +Cython has built-in support for Numpy arrays. -Profiling Cython code ---------------------- +As discussed in the Numpy lectures (TODO: LINK), Numpy arrays provide great performance +for vectorized operations. In contrast, thing like **for**-loops over Numpy +arrays should be avoided because of interpreting overhead inherent to Python +**for**-loops. There is also overhead from accessing individual elements of +Numpy arrays. -- By default, Cython code does not show up in profile produced by - cProfile -- Profiling can be enabled for entire source file or on per function - basis +With Cython we can bypass both restrictions and write efficient loops over +Numpy arrays. Consider e.g. a double loop that sets values of a 2D array: .. code:: python - - # cython: profile=True - import cython - - @cython.profile(False) - cdef func(): - ... + + import numpy as np + + def slow_looper(N): + """""" + data = np.empty((N, N), dtype=int) + + counter = 0 + for i in range(N): + for j in range(N): + data[i, j] = counter + counter += 1 + + +We can Cythonize this as before to optimize the **for**-loops. A quick check +with **timeit** shows that with **N=100**, the pure Python version takes 820μs +and the Cythonized version (without any static typing) takes 700μs. This is +nice, but we are still bottlenecked by array lookups and assignments, i.e. the +**[]** operator, which invokes Python code. + +We can get a huge speedup by adding a static type declaration for the Numpy +array, and for the other variables too while we are at it. To do this we must +import compile-time information about the Numpy module using the +Cython-specific `cimport` keyword, then use Cython's Numpy interface to +declare the array's datatype and dimensions: .. code:: python + + import numpy as np # Normal NumPy import + cimport numpy as cnp # Import for NumPY C-API - # cython: profile=False - import cython - - @cython.profile(True) - cdef func(): - ... - -Summary -------- - -- Cython is optimising static compiler for Python -- Possible to add type declarations with Cython language -- Fast indexing for NumPy arrays -- At best cases, huge speed ups can be obtained - - - Some compromise for Python flexibility - -Further functionality in Cython -------------------------------- - -- Using C structs and C++ classes in Cython -- Exceptions handling -- Parallelisation (threading) with Cython -- … - -Interfacing external libraries ------------------------------- - -Increasing performance with compiled code ------------------------------------------ - -- There are Python interfaces for many high performance libraries -- However, sometimes one might want to utilize a library without Python - interface - - - Existing libraries - - Own code written in C or Fortran - -- Python C-API provides the most comprehensive way to extend Python -- CFFI, Cython, and f2py can provide easier approaches - -CFFI ----- - -- C Foreign Function Interface for Python -- Interact with almost any C code -- C-like declarations within Python - - - Can often be copy-pasted from headers / documentation - -- ABI and API modes - - - ABI does not require compilation - - API can be faster and more robust - - Only API discussed here - -- Some understanding of C required - -Creating Python interface to C library --------------------------------------- - -- In API mode, CFFI is used for building a Python extension module that - provides interface to the library -- One needs to write a *build* script that specifies: - - - the library functions to be interfaced - - name of the Python extension - - instructions for compiling and linking - -- CFFI uses C compiler and creates the shared library -- The extension module can then be used from Python code. - -Example: Python interface to C math library -------------------------------------------- - -.. code:: python + def fast_looper(int N): + """""" - from cffi import FFI - ffibuilder = FFI() + # Static declaration: 2D array of integers + cdef cnp.ndarray[cnp.int32_t, ndim=2] data + data = np.empty((N, N), dtype=np.int32) + + cdef int counter = 0 + # double loop is done at nearly C speed + for i in range(N): + for j in range(N): + data[i, j] = counter + counter += 1 - ffibuilder.cdef(""" - double sqrt(double x); // list all the function prototypes from the - double sin(double x); // library that we want to use - """) - ffibuilder.set_source("_my_math", # name of the Python extension - """ - #include // Some C source, often just include - """, - library_dirs = [], # location of library, not needed for C - # C standard library - libraries = ['m'] # name of the library we want to interface - ) +Cythonizing and running the function with **timeit** shows that the function +now only takes 3.30μs with **N = 100**. This is ~250 times faster than the +pure Python implementation! - ffibuilder.compile(verbose=True) +.. callout:: -.. _example-python-interface-to-c-math-library-1: + `cimport numpy` needs access to Numpy C-headers which are usually included + in Python distributions. This usually works out of the box for Jupyter + notebooks. However, if using the command line `cythonize` tool you may need + to manually set include paths for the C compiler knows where to find the + headers. Refer to `the docs __` + for more details. -Example: Python interface to C math library -------------------------------------------- +.. callout:: -- Building the extension + It is good practice to also call `cnp.import_array()` after doing the + `cimport` of Numpy. This is required for accessing attributes (like + `.shape`) of typed Numpy arrays. -.. code:: bash - python3 build_mymath.py - generating ./_mymath.c - running build_ext - building '_mymath' extension - ... - gcc -pthread -shared -Wl,-z,relro -g ./_mymath.o -L/usr/lib64 -lm -lpython3.6m - -o ./_mymath.cpython-36m-x86_64-linux-gnu.so +More Numpy indexing enhancements +-------------------------------- -- Using the extension +When indexing arrays, Numpy does some bounds checking in an attempt to catch +logic errors (e.g. attempting to access element at index 100 of an array of +length 10). Numpy also checks for negative indices to support wraparound +syntax like **a[-1]**. We can tell Cython to disable these checks for some extra +performance: .. code:: python - from _mymath import lib - - a = lib.sqrt(4.5) - b = lib.sin(1.2) + import numpy as np + cimport numpy as cnp + cimport cython -- Python ``float``\ s are automatically converted to C ``double``\ s - and back - -Passing NumPy arrays to C code ------------------------------- - -- Only simple scalar numbers can be automatically converted Python - objects and C types -- In C, arrays are passed to functions as pointers -- A “pointer” object to NumPy array can be obtained with ``cast`` and - ``from_buffer`` functions - -.. _passing-numpy-arrays-to-c-code-1: - -Passing NumPy arrays to C code ------------------------------- + @cython.boundscheck(False) + @cython.wraparound(False) + def fast_looper(int N): + # ... Same function body as above ... -.. container:: column - - C function adding two arrays +Whether these decorators *actually* result in faster code or not depends on +how complicated your array usage is. In this simple example there is likely +no measurable improvement: even if the checks are kept, modern compilers and +processors are rather good at predicting unlikely branches and optimize the +execution accordingly ("branch prediction"). - .. code:: c +Disabling bounds checking of course means that out-of-bounds indexing will go +undetected and lead to undefined behavior. It may crash your program or cause +memory corruption, so be very careful if using these decorators! - // c = a + b - void add(double *a, double *b, double *c, int n) - { - for (int i=0; i`__): - .. code:: python +- Only Cythonize the modules/functions for which performance is *really* + needed. Profiling tools help at identifying such bottlenecks. +- Static type declarations work the best for fundamental data types (integers, + floats, strings) and for contiguous arrays. Operations on lists and + dictionaries do not usually benefit much from Cython. - from add_module import ffi, lib +TODO: when to use other C extension stuff - a = np.random.random((1000000,1)) - aptr = ffi.cast("double *", ffi.from_buffer(a)) - ... +Further reading +--------------- - lib.add(aptr, bptr, cptr, len(a)) +- Newer usage of Numpy arrays (memory views) https://cython.readthedocs.io/en/latest/src/userguide/numpy_tutorial.html#numpy-tutorial +- cpdef keyword for functions - - “pointer” objects resemble C pointers and can result easily in - Segmentation faults! -.. _summary-1: Summary ------- -- External libraries can be interfaced in various ways -- CFFI provides easy interfacing to C libraries - - - System libraries and user libraries - - Python can take care of some datatype conversions - - “pointer” objects are needed for NumPy arrays +- TODO Acknowledgements ---------------- diff --git a/content/img/cython/fractal.svg b/content/img/cython/fractal.svg deleted file mode 100644 index 7f154dcf..00000000 --- a/content/img/cython/fractal.svg +++ /dev/null @@ -1,32 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/content/img/cython/unboxing-boxing.svg b/content/img/cython/unboxing-boxing.svg index 3085c75f..1c791268 100644 --- a/content/img/cython/unboxing-boxing.svg +++ b/content/img/cython/unboxing-boxing.svg @@ -2,21 +2,23 @@ + inkscape:version="1.2.2 (b0a8486541, 2022-12-01)" + sodipodi:docname="unboxing-boxing.svg" + inkscape:export-filename="unboxing-boxing.png" + inkscape:export-xdpi="96" + inkscape:export-ydpi="96" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns="http://www.w3.org/2000/svg" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:dc="http://purl.org/dc/elements/1.1/"> @@ -46,76 +48,18 @@ - - - - - - - - - + inkscape:swatch="solid"> - - - - + inkscape:window-y="0" + inkscape:window-maximized="1" + inkscape:showpageshadow="2" + inkscape:pagecheckerboard="0" + inkscape:deskcolor="#d1d1d1" /> @@ -143,10 +90,24 @@ image/svg+xml - + + + + + style="fill:#d7faff;fill-opacity:0.843621;fill-rule:nonzero;stroke:#000000;stroke-width:0.264583;stroke-opacity:1" /> Object Integer + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:7.76111px;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;stroke-width:0.264583">Integer @@ -190,14 +151,14 @@ height="11.906249" width="41.01041" id="rect6000" - style="fill:none;fill-opacity:0.30638302;fill-rule:nonzero;stroke:#000000;stroke-width:0.86500001;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.99170125" /> + style="fill:none;fill-opacity:0.306383;fill-rule:nonzero;stroke:#000000;stroke-width:0.865;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.991701" /> otherotherobjectdata + + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:16.9333px;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;stroke-width:0.264583">+ @@ -255,21 +216,21 @@ height="75.67083" x="86.821129" y="63.410713" - style="fill:#d7faff;fill-opacity:0.84362135;fill-rule:nonzero;stroke:#000000;stroke-width:0.26458332;stroke-opacity:1" /> + style="fill:#d7faff;fill-opacity:0.843621;fill-rule:nonzero;stroke:#000000;stroke-width:0.264583;stroke-opacity:1" /> Object Integer + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:7.76111px;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;stroke-width:0.264583">Integer @@ -287,14 +248,14 @@ height="11.906249" width="41.01041" id="rect6043" - style="fill:none;fill-opacity:0.30638302;fill-rule:nonzero;stroke:#000000;stroke-width:0.86500001;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.99170125" /> + style="fill:none;fill-opacity:0.306383;fill-rule:nonzero;stroke:#000000;stroke-width:0.865;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.991701" /> otherotherobjectdata Check the types:Check the types:integers + inkscape:connector-curvature="0" + id="path59" /> int 7 + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:5.64444px;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;stroke-width:0.264583">int 7 + style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> @@ -414,14 +376,14 @@ height="11.906242" width="25.664574" id="rect7726" - style="fill:none;fill-opacity:0.30638302;fill-rule:nonzero;stroke:#000000;stroke-width:0.86500001;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.99170125" /> + style="fill:none;fill-opacity:0.306383;fill-rule:nonzero;stroke:#000000;stroke-width:0.865;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.991701" /> + = @@ -462,7 +424,7 @@ transform="translate(90.80878,2.9104166)" id="g7754"> int 13 + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:5.64444px;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;stroke-width:0.264583">int 13 Object @@ -510,9 +472,9 @@ id="text8814" y="85.333336" x="93.791794" - style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:7.76111126px;line-height:1.25;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:7.76111px;line-height:1.25;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.264583" xml:space="preserve"> int 13 + style="font-style:normal;font-variant:normal;font-weight:500;font-stretch:normal;font-size:5.64444px;font-family:Inconsolata;-inkscape-font-specification:'Inconsolata, Medium';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;stroke-width:0.264583">int 13 + style="fill:none;fill-opacity:0.306383;fill-rule:nonzero;stroke:#000000;stroke-width:0.865;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.991701" /> otherobjectdata - From e56111193534f42f0da3dd2a2d1d85ea532a0257 Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Wed, 19 Nov 2025 13:40:28 +0200 Subject: [PATCH 03/10] More about Python extensions in general + fix links --- content/cython.rst | 79 ++++++++++++++++++++++++++++------------------ 1 file changed, 48 insertions(+), 31 deletions(-) diff --git a/content/cython.rst b/content/cython.rst index 98bef95f..de9406e7 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -1,16 +1,20 @@ .. _cython: -Cython -====== +Extending Python with Cython +============================ .. questions:: - - Q1 - - Q2 + - How does runtime performance of Python compare to languages like C, C++ + or Fortran? + - How do we use code written in other languages from within Python? In what + situations is this useful? + + .. objectives:: - - O1 - - O2 + - Understand how compiled extension modules can speed up code execution. + - Understand the basics of Cython. .. callout:: @@ -23,10 +27,13 @@ Cython with `conda install cython`. +Python and performance +---------------------- + Interpreted languages like Python are rather slow to execute compared to -languages like C or Fortran that are compiled to machine code before execution. -Python in particular is both strongly typed and dynamically typed: this means -that all variables have a type that matters for operations that +languages like C or Fortran that are compiled to machine code ahead of +execution. Python in particular is both strongly typed and dynamically typed: +this means that all variables have a type that matters for operations that can be performed on the variable, and that the type is determined only during runtime by the Python interpreter. The interpreter does a lot of "unboxing" of variable types when performing operations, and this comes with @@ -60,8 +67,20 @@ Scientific programs often include computationally expensive sections (e.g. simulations of any kind). So how do we make Python execute our code faster in these situations? Well that's the neat part: we don't! Instead, we write the performance critical parts in a faster language and make them usable from -Python. This is called extending Python, and usually involves writing C-code -with Python-specific boilerplate and compiling this as a shared library. +Python. + +This is called extending Python, and usually involves writing C-code +with Python-specific boilerplate and compiling this as a shared library, which +in this context is called a **Python extension module**. +Most scientific Python libraries (Numpy, Scipy etc) do exactly this: their +computationally intensive parts are either written in a compiled language, +or they call an external library written in such language. + +When working on your own Python project, you may find that there is a C +library that does exactly what you need, but it doesn't provide a Python +interface. Or you may have computationally intensive code that doesn't +vectorize nicely for Numpy. In cases like these it can be useful to write +your own extension modules that you then import into your Python code. Here we discuss one popular approach for extending Python with compiled code: using a tool called Cython. @@ -73,7 +92,7 @@ Cython that can be processed with the Cython compiler to produce optimized code. Cython is designed to provide C-like performance for code that is mostly written in Python by adding only a few C-like declarations to existing -Python code. As such, Cython provides the best of the both worlds: +Python code. As such, Cython aims to provide the best of the both worlds: the good programmer productivity of Python together with the high performance of C. Cython also makes it easy to interact with external C/C++ code. @@ -147,15 +166,14 @@ Cythonized before use. instead use an established build tool like **setuptools** to handle the Cythonization during the project's build phase. More info is available on the `Cython documentation `__. - See also the course page on packaging. (TODO: link.) + See also the :doc:`course page on packaging `. Using Cython with Jupyter ------------------------- -Jupyter has an `extension ` -for supporting Cython compilation directly inside notebooks, assuming your -environment has Cython installed. +Jupyter supports Cython compilation directly inside notebooks via `an extension `__, +assuming your environment has Cython installed. We first load the Cython extension, e.g. in the very first cell: :: @@ -214,11 +232,11 @@ Cythonize as before: Import this into Python and confirm that it works as expected with integers. However, if passing floating-point numbers the function is forced to interpret the inputs as integers before performing the addition. For example, -**add(1.2, 2.7)** would return 3. This happens because there is an automatic -conversion from the input Python objects (floating point numbers) to the -declared C-types when calling the Cythonized function from Python. -Similarly the returned C variable is converted to a corresponding Python -object. +**add(1.4, 2.7)** would return 3. This happens because there is an automatic +conversion from the input Python objects to the +declared C-types, in this case integers, when calling the Cythonized function +from Python. Similarly the returned C variable is converted to a corresponding +Python object. To make the function work with floats we'd instead declare the types to be either **float** (32-bit) or **double** (64-bit) type instead of **int**. @@ -247,11 +265,11 @@ Using Numpy arrays with Cython Cython has built-in support for Numpy arrays. -As discussed in the Numpy lectures (TODO: LINK), Numpy arrays provide great performance -for vectorized operations. In contrast, thing like **for**-loops over Numpy -arrays should be avoided because of interpreting overhead inherent to Python -**for**-loops. There is also overhead from accessing individual elements of -Numpy arrays. +As discussed in the :doc:`Numpy lectures `, Numpy arrays provide +great performance for vectorized operations. In contrast, thing like +**for**-loops over Numpy arrays should be avoided because of interpreting +overhead inherent to Python **for**-loops. There is also overhead from +accessing individual elements of Numpy arrays. With Cython we can bypass both restrictions and write efficient loops over Numpy arrays. Consider e.g. a double loop that sets values of a 2D array: @@ -280,7 +298,7 @@ nice, but we are still bottlenecked by array lookups and assignments, i.e. the We can get a huge speedup by adding a static type declaration for the Numpy array, and for the other variables too while we are at it. To do this we must import compile-time information about the Numpy module using the -Cython-specific `cimport` keyword, then use Cython's Numpy interface to +Cython-specific **cimport** keyword, then use Cython's Numpy interface to declare the array's datatype and dimensions: .. code:: python @@ -291,7 +309,7 @@ declare the array's datatype and dimensions: def fast_looper(int N): """""" - # Static declaration: 2D array of integers + # Type declaration: 2D array of 32-bit integers cdef cnp.ndarray[cnp.int32_t, ndim=2] data data = np.empty((N, N), dtype=np.int32) @@ -313,7 +331,7 @@ pure Python implementation! in Python distributions. This usually works out of the box for Jupyter notebooks. However, if using the command line `cythonize` tool you may need to manually set include paths for the C compiler knows where to find the - headers. Refer to `the docs __` + headers. Refer to `the docs `__ for more details. .. callout:: @@ -378,8 +396,7 @@ Further reading --------------- - Newer usage of Numpy arrays (memory views) https://cython.readthedocs.io/en/latest/src/userguide/numpy_tutorial.html#numpy-tutorial -- cpdef keyword for functions - +- TODO Summary From c4e8904392b2b6453f3ef17a7600e17e089f84ce Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Fri, 21 Nov 2025 11:11:15 +0200 Subject: [PATCH 04/10] Cython formatting fixes etc --- content/cython.rst | 43 ++++++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/content/cython.rst b/content/cython.rst index de9406e7..5a3bb612 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -97,11 +97,11 @@ the good programmer productivity of Python together with the high performance of C. Cython also makes it easy to interact with external C/C++ code. The Cython compiler processes code written in Python, or more -commonly the Cython extension of Python language, turns it into valid C-code -which is then compiled into a Python extension module using a C compiler -(GCC, Clang, MSVC, ...). The Cython programming language is a a superset of -Python that adds C-like static type declarations and other features that -make it possible to generate efficient machine code. +commonly the Cython extension of Python language, and turns it into valid +C-code which is then compiled into a Python extension module using a +C compiler (GCC, Clang, MSVC, ...). The Cython programming language is a +superset of Python that adds C-like static type declarations and other +features that make it possible to generate efficient machine code. .. callout:: @@ -125,7 +125,7 @@ Suppose we have a Python module called **my_module.py** that contains: Cython allows one to compile **my_module.py** directly to machine code while still allowing its contents to be imported and used from Python code. We can -Cynthonize the module "manually" from command line: +Cythonize the module "manually" from command line: .. code:: bash @@ -179,7 +179,7 @@ We first load the Cython extension, e.g. in the very first cell: :: %load_ext Cython -We can Cythonize cell contents using the magic `%%cython` and executing it: +We can Cythonize cell contents using the magic `%%cython`: .. code:: python @@ -189,6 +189,7 @@ We can Cythonize cell contents using the magic `%%cython` and executing it: return result +The compiled function can then be called from other cells. There is also `%%cython --annotate` which is useful for analyzing the generated C code. @@ -198,10 +199,10 @@ Adding static type information ------------------------------ So far our Cythonized extension module is rather dumb. We have reduced some -of interpreting overhead by compiling the code, but it's still using Python's +of the interpreting overhead by compiling the code, but it's still using Python's fully dynamic type system with the same boxing and unboxing overhead as in standard Python. This is because there are no type declarations in the code -that Cython could use for optimizations. +that Cython could use to optimize. When Cythonizing a Python code, static type information can be added either: @@ -211,7 +212,7 @@ either: - By declaring variables with the **cdef** Cython keyword, followed by the the type. -To make Cythonize a function that adds two integers and returns the result as +To make Cython function that adds two integers and returns the result as an integer, we would write: .. code:: python @@ -222,7 +223,7 @@ an integer, we would write: return result The function works now only with integers but with less boxing/unboxing -overheads. Store this as **my_module.pyx** (note the file extension) and +overhead. Store this as **my_module.pyx** (note the file extension) and Cythonize as before: .. code:: bash @@ -303,8 +304,8 @@ declare the array's datatype and dimensions: .. code:: python - import numpy as np # Normal NumPy import - cimport numpy as cnp # Import for NumPY C-API + import numpy as np # Normal Numpy import + cimport numpy as cnp # Import for Numpy C-API def fast_looper(int N): """""" @@ -330,8 +331,8 @@ pure Python implementation! `cimport numpy` needs access to Numpy C-headers which are usually included in Python distributions. This usually works out of the box for Jupyter notebooks. However, if using the command line `cythonize` tool you may need - to manually set include paths for the C compiler knows where to find the - headers. Refer to `the docs `__ + to manually set include paths for the C compiler. + Refer to `the docs `__ for more details. .. callout:: @@ -347,8 +348,8 @@ More Numpy indexing enhancements When indexing arrays, Numpy does some bounds checking in an attempt to catch logic errors (e.g. attempting to access element at index 100 of an array of length 10). Numpy also checks for negative indices to support wraparound -syntax like **a[-1]**. We can tell Cython to disable these checks for some extra -performance: +syntax like **a[-1]**. We can tell Cython to disable these checks for some +extra performance: .. code:: python @@ -395,9 +396,9 @@ TODO: when to use other C extension stuff Further reading --------------- -- Newer usage of Numpy arrays (memory views) https://cython.readthedocs.io/en/latest/src/userguide/numpy_tutorial.html#numpy-tutorial -- TODO - +- Cython `memory views `__ + are a newer and more general way of interfacing with Numpy arrays and other buffer-like objects. +- `Calling C functions from Cython `__ Summary ------- @@ -407,4 +408,4 @@ Summary Acknowledgements ---------------- -This material has been adapted from the "Python for HPC" course by CSC - IT Center for Science. \ No newline at end of file +This material has been adapted from the "Python for HPC" course by CSC - IT Center for Science. From 9ca3f087ab450391c7b7643ff5ddc67cb21e9abf Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Fri, 21 Nov 2025 11:52:05 +0200 Subject: [PATCH 05/10] cython chapter: add list of other similar tools --- content/cython.rst | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/content/cython.rst b/content/cython.rst index 5a3bb612..933403d1 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -261,6 +261,7 @@ float, double float char \* str/bytes ============= =============== + Using Numpy arrays with Cython ------------------------------ @@ -388,10 +389,21 @@ Cython (see also `Cython docs `__: part of Python standard library. +- `CFFI `__: somewhat similar to `ctypes` but with more features and probably better for large projects. +- `pybind11 `__: very robust and modern way of creating extension modules. C++ only. -TODO: when to use other C extension stuff Further reading --------------- @@ -400,10 +412,6 @@ Further reading are a newer and more general way of interfacing with Numpy arrays and other buffer-like objects. - `Calling C functions from Cython `__ -Summary -------- - -- TODO Acknowledgements ---------------- From 461d89c8e12a4e3a25407e6817a5a2418e5fa38c Mon Sep 17 00:00:00 2001 From: Lauri Niemi <113029612+niemilau@users.noreply.github.com> Date: Fri, 21 Nov 2025 13:09:50 +0200 Subject: [PATCH 06/10] Update content/cython.rst Co-authored-by: Ashwin V. Mohanan <9155111+ashwinvis@users.noreply.github.com> --- content/cython.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/cython.rst b/content/cython.rst index 933403d1..d5abb06a 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -171,6 +171,12 @@ Cythonized before use. Using Cython with Jupyter ------------------------- +.. important:: + + Due to a `known issue`_ with ``%%cython -a`` in ``jupyter-lab`` we have to use the ``jupyter-nbclassic`` interface + for this episode. + +.. _known issue: https://github.com/cython/cython/issues/7319 Jupyter supports Cython compilation directly inside notebooks via `an extension `__, assuming your environment has Cython installed. From 3ddadd38a458b80628bd830adb185aa6d1ac7003 Mon Sep 17 00:00:00 2001 From: Lauri Niemi <113029612+niemilau@users.noreply.github.com> Date: Fri, 21 Nov 2025 13:15:28 +0200 Subject: [PATCH 07/10] Update content/cython.rst Co-authored-by: Ashwin V. Mohanan <9155111+ashwinvis@users.noreply.github.com> --- content/cython.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/cython.rst b/content/cython.rst index d5abb06a..abd1c209 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -204,7 +204,7 @@ generated C code. Adding static type information ------------------------------ -So far our Cythonized extension module is rather dumb. We have reduced some +So far our Cythonized extension module is rather minimal. We have reduced some of the interpreting overhead by compiling the code, but it's still using Python's fully dynamic type system with the same boxing and unboxing overhead as in standard Python. This is because there are no type declarations in the code From 7c362e9ded9c1556567c86c17fd792677c9b40aa Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Fri, 21 Nov 2025 13:50:12 +0200 Subject: [PATCH 08/10] cython.rst: implemented some comments --- content/cython.rst | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/content/cython.rst b/content/cython.rst index abd1c209..b9906d2b 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -22,9 +22,9 @@ Extending Python with Cython Using Cython requires that you have a working environment for compiling C code. This goes beyond the software requirements for this course, so the teaching will be given in form of demonstrations and no exercises. - You may still follow along with the code examples if you have a C compiler - installed, in which case you can install Cython to your Conda environment - with `conda install cython`. + You may still follow along with the code examples but you will need to have + Cython and a working C compiler available. You can install both to your + Conda environment with `conda install -c conda-forge cython c-compiler`. Python and performance @@ -66,12 +66,13 @@ performance at runtime. Scientific programs often include computationally expensive sections (e.g. simulations of any kind). So how do we make Python execute our code faster in these situations? Well that's the neat part: we don't! Instead, we write the -performance critical parts in a faster language and make them usable from -Python. +performance critical parts in a faster language and make them usable +from Python. -This is called extending Python, and usually involves writing C-code -with Python-specific boilerplate and compiling this as a shared library, which -in this context is called a **Python extension module**. +This is called extending Python, and usually boils down to writing C-code +with Python-specific boilerplate, or using a specialized tool for generating +such C code from Python code (so-called *transpilers*). The C-code is compiled +into a shared library, in this context called a **Python extension module**. Most scientific Python libraries (Numpy, Scipy etc) do exactly this: their computationally intensive parts are either written in a compiled language, or they call an external library written in such language. @@ -197,8 +198,8 @@ We can Cythonize cell contents using the magic `%%cython`: The compiled function can then be called from other cells. -There is also `%%cython --annotate` which is useful for analyzing the -generated C code. +There is also `%%cython --annotate` (or `%%cython -a` for short) which is +useful for analyzing the generated C code. Adding static type information @@ -259,14 +260,6 @@ int, float float, double str/bytes char \* ================= ============= -============= =============== -From C types To Python types -============= =============== -int, long int -float, double float -char \* str/bytes -============= =============== - Using Numpy arrays with Cython ------------------------------ From e4c676202ec954bb40cb43054ab3deac798465d9 Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Fri, 21 Nov 2025 16:30:15 +0200 Subject: [PATCH 09/10] Updated cython.rst with Numba link, jupyter demo and better Objectives list --- content/cython.rst | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/content/cython.rst b/content/cython.rst index b9906d2b..b22e3b39 100644 --- a/content/cython.rst +++ b/content/cython.rst @@ -14,8 +14,10 @@ Extending Python with Cython .. objectives:: - Understand how compiled extension modules can speed up code execution. - - Understand the basics of Cython. - + - Build your first compiled extension module with Cython. + - Learn to optimize your Cython code with static type declarations. + - Learn to use Numpy arrays in Cython code and implement common performance + enhancements for Cythonized arrays. .. callout:: @@ -198,8 +200,18 @@ We can Cythonize cell contents using the magic `%%cython`: The compiled function can then be called from other cells. -There is also `%%cython --annotate` (or `%%cython -a` for short) which is -useful for analyzing the generated C code. +.. demo:: + + There is also `%%cython --annotate`, or `%%cython -a` for short, which is + useful for analyzing the generated C code. Try executing the code for + `add()` with this magic command in Jupyter. Upon doing so: + + 1. Estimate the amount of interactions with the Python runtime, by the intensity of the yellow background colour. + 2. You will be able to inspect the underlying C code. + + .. solution:: + + .. image:: img/cython/jupyter-cython-annotate.png Adding static type information @@ -395,14 +407,18 @@ Cython (see also `Cython docs `__ is a tool that compiles Python code to +optimized machine code on the fly without needing a manual compilation step. +It works with Numpy but does not support all of Python's features. + +For creating compiled extension modules there are a plethora of tools and +libraries. If you already have a working C/C++ codebase and would like to use it from Python, consider using one of the following: - `ctypes `__: part of Python standard library. - `CFFI `__: somewhat similar to `ctypes` but with more features and probably better for large projects. - `pybind11 `__: very robust and modern way of creating extension modules. C++ only. - +- `PyO3 `__ for Rust code. Further reading --------------- From d7490ddc0000d25b8b09d0bb8689685ab1cfb62e Mon Sep 17 00:00:00 2001 From: Lauri Niemi Date: Fri, 21 Nov 2025 16:31:37 +0200 Subject: [PATCH 10/10] Added image for cython section (forgot to include in previous commit) --- content/img/cython/jupyter-cython-annotate.png | Bin 0 -> 39122 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 content/img/cython/jupyter-cython-annotate.png diff --git a/content/img/cython/jupyter-cython-annotate.png b/content/img/cython/jupyter-cython-annotate.png new file mode 100644 index 0000000000000000000000000000000000000000..5ada11f76d2513e36c3121ae1f37db69f0427c46 GIT binary patch literal 39122 zcmd3O1y@zg8?PcriF7wecT0CSND9*3EnR|2cPNN-cSwgI9nxJ=(%p6E7_a|z?-#hP zvz8*~>@$1fnP1I*ucRP}gn))+&0@2zOdK_-gM)*QqdnK(D?WgSrWBe?Cl+}B{{4%Jo3drwzt{Eu4yQDX5RUE? z{qO6ip!3Xw!oy(xTpD}<2|Lt_Dv^~Aoo7N&e_kk@;0q1NjsCR|4KDlvX7TsuEZG0P z_Kmz>i278w=6~(h(UKr$QFE_kZHX7q=&^Yk9+2Tre-nQbpD2b7wd8+NLg7qfOvJT&dE?K;S#^Ho0!|N z!`iIZnTY0S3TmxC!Kf>uM^%xZ!lXAYy3zf(_jFQ^$!WcxrN{mpllFxa=+d>`NDsOf zc_e|b$eh=-Y6ZshbbX0Ti5qFY3pe4!MHG7h)p&LdyPQs8kCE`YJHzpHd(7E-6KK_B z95)6TxE8%Ci?pkI*Tz6IGWHub2>dOt_E z(wLEH&DNy*&5nUjE{)$?PsN@5fY0`N?v6HI&$%DDTj4V*TqhD=Y;RInPiki)pd1{o zgJr86OU(5BYd74=ypUnC#-gj@VuEMD6>2XZ3 z5PZN(G=%T7b}cq{o%`U!&hc%;fsc`5z##U7;j)GOI0JXsFp^&7eRsLR>UCkCEa=

!C+m29AU0bz5n+bKv>zw(ByM4<#ka*_+~;&$<7iUPH~Cmtu=VRtMkQm3 za6F2aHP+Dt+>WnP=#Gl@S{CfZ1(zH3Ja;)X2&oiOm=n4~FzI@~1|eoErU@L~UGIPN zLZNnC3T=LK1Abs!SL!EGs?wi_R-|rKiaOxJP?Or8D3-_7A)c-2u7;~2Y7Hb)0q-Xaro zGVQo*Pr&PhD2)M zo@YDjZlm&C$rsPY;_zP>#lI10;&M3Mu@o@6s(ybllO2q|VL!^j{`Uk&a>ROh5NcJ-m& zi4?fXQ@&8b4*FGSNN!PUNyH0<7K&oD;^|*jn1(B3&ukj!9;lZ-c3k~wap^S8cxisZ zny)EPs=V_RwU#uNT%TwHp|OLh<;KxSxF`9zkr#89+xx0242R__HPjH_aaiNkRw2>A zn+zC}US0PC0c2{aa6FEfCeL#`E({%$=-R075-*no_`v=*{bn4&|Jq)72Fi`Xr|hN{ zan8D?RRy|F`#DrlrDnNd2^mx4LBP@mqkVjvHjF>M%hp(ai7~ZMSw|ZKLTHSQL*Vs* z?%XVV_ICjdUd83FLHs{9?HOx$u{WN$k;c)#EO?P`h%sjm?G3$VJ(?rWILU-@sI<1< za?hz=0}Ad~<~vWpU~TN<-#ca&5)^ipxj6T8qI<-bG82__%Jgq15=Cj%`kCxCo@$)y z)rP1$vj3bmOXE1?@V-8audRK)IaQtsA0e57R75}f#V+v^zh|!~GXJw!Dn+c?%B2Vq zok_aVMl!ItSqI8i8gHuMPuuF>yk^jRwRBX$pm^8&eAf+mOtPNP$#?^4;_0eWB)3yR z`)2Ll_2=)kkt3jLKl)^h-yA<>*Bv-h;#KwdwX5yCreH5 zg|}DpNWJ`3lSj|b;moy8d^>g-{KsNKwNU^=!)IQv`)uLz{vNDc_FFsZfegW5RO7?jS?(9jy)VCRFOf@!Grg@xsC zjKK_FhwW2vM8F`Z{Q(6(XMJbf763DnBae@s3-gfo{Iiem+Sh61Y%FyK%^_=XyKfmz0IlUop8MLL$8;hJc+mLX(OF=@a?r!h8;;)i6IasY7hSTR94I&yzp+} zBw-Xf75Zvk#8~^{$1gp-?s}3 z@dYd`ed$xB)^JhycTsJYR@jHjnHr>;f z``e+~3p_4cf;`QFT!Ry$fP*&d8U z#R5j1JOtWpzV*`+FMK(amOb9Sm`rkV8S0wUlIClqUhqk)4NA*oeg9=v7hOwFrlMl( zer|DW7jZrf@`4_P+U&A3GqifUb`Um`&dUV{N&DzFsK;E`KCj=;j6NbkA>5;&=H;-O zU>DvhW3EHo3c`Hya3N`=RoY&zu?qD&5(SEkR^HLYN+iB4YCtil-*ne!yKKGw#ni#t ziZUDRQVP_?O2#N$LCQY*2`Nx8xEoK`9;-`e#_9jDq0*5x9dO)Etxj!zx-}k~BmeP38~l_a=`RQ?l?I)0 zLXp6ZJ0@-|n6NbMyOed}ecOp>31eDDJaKNC@98p(SO6Q8$T0fZGrY7uDZtM?j5F|K zjL2U^9CZhGQEi^Ro-ejtg;cq28$A;)*;nNLvR&YWZvW1<*YFDVW~8CD{^YQWxbFjp z6*M^t&3|}$GgjzBnR&LjGxlgk&?fv+!q=TuxR5`P0RV?AMW{{NF@viPf>(v*V#vP> zF$9_4qD!Hdw4X#qoBapPg`k=Ng1=;4Ci;u=AEHTN0R+$ZV5P*U;aOjUR@TYVh{0sV z<$-CP`_6z$nXj;_WP_b5CzjP}w|VOGv3s%N?McZ^9hJj+_Q-vGsQ#fK3*@p^XKl=W z+2DK<>Obv{N`PaQ*d{ye{45CDlTz%IJw^V6U@jGwgCqKK4KS;x*dm49reUgZC8tzPmI*k1=A7MEm+mKl+~1c=IjS z@|_;5s#x07(0p(48@q38mRn?=k1y>CBkEMUxSiP_XkL%~@OGDHu8dS>QZQ^>Yh{H( zqvUlew}3+8;o(m2cwwgV*`;QxH2_8$R6toT;gF(XiD%Y&)G0)m$Du1kF<9 z?K^`6N4K^kwxn_**&kB@?k&vZcU2and`GXkA5T2r8O@E9#8k>*j%_~YiTtwY)1c)! zlBO?w05*Ad`qULBrt{d})Jw<_#oniZQgQ&dECRlevb$CM(VFpBX_`-!wS3&S(rJ}D zSE?Gg&BnEHy=pQa)XXWjkxLRAS{h$s?P;1HP0+nk2!J@j~NheTrPh6XLy13INV}MkSNH*^)_ivLl#OHsArjR2l7`?|m z4s>0~bEOl$YCcxiA5ZE_5+ygo-*WbLsG(di?e;gZIYnDv1nnMzYLT{XkuY~{=hm)mb*b?YCT>ID|LkH5rPPgjT|U^DK-J&UK) zNZ?v{-A>;ehC@@-FxB8@*VP!k*AXxSz!(RjwP=mqqAdn*(dbVGO=7eFPSw9moB;*g zm;}?5gIajsGp#h+rhc~PUgOPb68YLqzSYLQObrh3oYw`(++Sw`t=gZfEEQ2HymQ_s z%0y)7l8zuaG_6`HP*Zlic^xBe)gCN|miy>QUp#M1m`ObC#{ELIEPJU`w#>%4T8Y=- z;b(i)&U9A6k!)FPwIkZ@GK2Q8AjE`{RSZuD8_jYObSi~0R}GcWPy0&`8&b_HWA%-H z^u$uFLEya1y1?Is=n6G>mtia@*O`YRgYk@#r}G9MpK#m3kP2P#!!B!GUpGJI-l0@q z8b?dlS&(((jbm7UQnYk@1e=6b$ng4DJ~|Z-Vqi4&_!m~sQ|tBqjdVr&PlI z*2=T2u~e7?KKBiF$T>ywqQ6fHFC z4R9n8Qe4^s{3f8yWR_K)ww>i3Bvf7_iSzj5vk z+c)c8TE9g*U1ksn=5=SevTlQGJY7lc*A8W2+<=cmFX>O!$lz`PA779rwOMrNB3!=! zJcyM=^WBhR445UBI@$_#I~9E?F>3%t@hDE2azBhE`CFk?{DCOKO!B3XbrU8s`YqF- zm-46EQ?1Amjnx*BC51U3=LVhAVYLxcNBlaYAB3iF$e=c3WqT6^shl=UN0i+%gnwp` z-y6Pgf3D{_@FC$vsjc*4GfQW0wXJSkL}#e0iJ$nkKP;Nb!IF8~PthBJi`MB{cL-T6 zQd`))UyW5XKHHhblM?(ICSyHcU!DB;>q{D5uZuEx5#Jy0sTvF6Eb#ZbA*>MsIRWH8 z;#I_gI>+a`(+J_6SI zdi~wiHcUb}&`)w45nn(5vq;~iJiufS+yVRg!}j_AU;`=o=WS-=7yVLyL5L94GNi5& zF|AYnr_A|B!2^vC7n98com#iPit#@7+l|2ejnlrd?Vqiul|s=kWP-vMUhJtppgg9# zt%l!1V2ZLNBZ(Y`_&>>_!O*vg{b@)y2^e~lsEAdItf&41=$UInR7PtlgT4RVl0Nu7 z40)WyXs%)!<8-B2;^py%?0mfo*j;r@1IEF&k%UD&`R!@>Dw3%YlTK|}Id$DoV32z* zbD(FPT$Y->B{!>K33ZGE=8JttACz>kWlDvc)FRJh7dBavc@=@#d?Z)MZ2TuK>htXF#WB$rKY}6AOo_i&&(e`u-$}q|>2nH~IL{RC z3Ualkj!N@BOV*0w;p9|@s4+mgaiYm?vz)#Yt=hR_*!;Nqy*JA@Q2?7<`O1uP+^Vnz zeJ=7{PC}@b zES9iNu&6buM~k%Bj2;%MWQRBu>U!>0OQ4o97%r?acDm{as2MIGlytlqso5GYlpXrS zA8vRB;Jwu4(V8fY=@Ef8CIzog$XR!blgEf_&eV4d1`Ht66)T^e?af>1G?j6F-IDwb z>AP#eOw3+s4v&a9+ujSbda_rMmq*>j1{Qsh1MXlFizr0GfoQ1ihvW%xKVCe=sv3(^ zy%tk%N?b*7;jtapKzrtdvoIKhh%KX|lR~dqCPd_YqMS=hc&_HbseBMf@5>;8g0kq` zLjymaVXC2>t3U|Z37)L{_!f#UOQuus8|(Uq$pRVt@y<)=8geYl!yn%qfeLobFIqdc z*LbXO9Eeyh|B;hYp8$b?o)HCle73@6TfHY``aDLM|M@3D{{}`WgBNPnuGi8M|U*&dRSK&t`Q}!IKkxA>9)KaHU>MjRuq|dNK$Pv zJUV5Pjb@NY)Cr0R>pV^^*|r9scAuAYdGfxbk=@gD z@7-r&>^RL{mI>-ZX}bp!Ek}uHShtv`>Pzh}wl+Ujeg1G?)$0R3qF-oUh!!18mflv? zZF_QwFy=#{R2~)M27a`ymu4>XA+a(lc=*Hr_u^BD6N+ml;U7Sz z@u)pf<)&v&N(>Q!>XCntDaLZ}pzZSSK+CssEm0b!F%3mPGvfIJy{89!^}q=}j=$8O z!MK2w&l`bt+#FOWd+e_+A<0LhKN?LepwT}HL#jA#2mlOE##8x{tK^oZ%8e}Q+^z9{ z_Yne$KC>5skV*W1_5axeK2+-y*9?9LTB$9*{X#3Q)JO*?50!^sf_(Qtn_^X=kgXG|$a(fi-axCye1UwMS= zKKr|)NV!24Rs*A~<$=7BE$c*{f9T`h>NH)tx~uh=m_V7{K_g!IV5GO4QpeH*+4}bZ zDuzn%b?4ijCcFOClr{@B?m0WUdJ85sz6%$X)17uBEwTO0kqJJ=HN!9NBTD)hPDtGSS5{Z(~WxA_wfBX8GT}vaS&Frv^H0GNNUwZO4 z$vYcEX-q(R<=AF6WSxd=HF#x2-lGpZxvX>Az@pEa zueQ?A;K7;9bk0Wi!(wGlQ=TUo>%N41FgxnN}}z`gJ@T5MO~c-|34`jDB^c5?beY`nhtYh1ByB z#ZUUYWquDQ=sHw{l*$h5 znL5JxU+W_@eOb6orpgRDtEYh2U)hL!*I5cg^d5Jd3GrW$A!-oLo~YP)!eC)wSE)Mi zNwZ1WwD%y7nU9f1L_CCu1#0k9i`o%dj=Z|qh1tB9^u=v)Y;-OdmcHWJvx_UG+0K#r z=7l+M7Sw%^)1LheQyPne+H(&I265MqqIek_$>j3WngQ_^@QfsezaDW!0jK1nFH0W} z1{HmTO_w4P4zmyf7QH!^T?tUfg@yx0TxM;))M51nyM+E#R%jq5q71si^3^`QuRs*P z>I=#?yS0V*^*r)>JPV(_iZlI~Yv#I-YYP!mKQ$w_WY=g-YMj2UvECE$qEC0EDt%}) z0n(tU@HX&a+^KNSNqdFz3siF>v^wex+JjB?0A0-n&`-dmHt^S!_k)g!1&eSs%N=zq zWQbtjX)N|V)j;zV#kOoQ8Mr~-(Q)yTvmWB!r?OC`{Ci_h8y2YFBdno3JQkCbl&!Iz zCO9VHw7kE&MZ;Y4BB;eN>rWzu_#prc^&Kp?!>BW9Haf2NZ)!fpV*d+Fb*KUUz?NY~ zN+^wMiLt<>muj;6xof$cK>pG>c;?u^y~c5McT3cNr+iwTyTe(SlW@lPPTmTFl+Y2Y zUmqG8l^;?4 z?b|b__-h{6zefeNM_ zWlDohiSt!?>`BsaJEovoosdcFewT}&Olfk{>Oh@cjaZ+m?RwR-+ zZ9V|FA`?NtZ8&q&MeHlJ*z9cr_VfyySx#JjkE)KYE?MxSyhk*dxSZGI8+i2oGhq3c zfZ4rnXP2*4nd1d0NPv4avka=OtebEYKSu3%p+;Z6YOc=1%Y{?>kH3a)X@o>bobJ(? z)8570(ebTdA3x@P^seNWzAskI@dIzU`sNcC88DC3>{A^MQ@*WLUzJMx7r_|!FB%yi zqy?JH`@2v!$+XdqG}*fKq|j8#-I!%mm5lKW#Ne?QzG1gKhW^C(Hw!Q(#pAMN{G+_y zZJ!ZvT8>_fXOa;FN5CIOE?{tKi(dWjAhkSDWIpI@w_JzVzw zP$r{aLiu;c1C2>23zThfnTWMNN(Dqdgv17f zAFw3iNmkx_Ix{!bI$ zfF(-emn8SEozW0MkcK;f{O?9Z;twG!Fb_M7|F!d79QfS-mtA!v$_UJI$Eys#}A-4be`j@;L14B;tdz1Q=8_|PuCcOH8 zMd1JET`#Jx^ z<(%lh5$O>jk*C&ffEA!B#K&ZBF>>;Ka^2zDe4A8w^PZyeE920=w?2zPgXBBRkayz= zO9h`<*)ow^?bFloE^g9pL=Yo2AStCx%)R0n9KQNxOMkeF&dme_Zyzdl1qli<;Imay{952JiVr_K-^=G(L%>;Yxs} z;YyG}+|zrtRH|bC%OIspLc!^?A@5WQo|g*T=mDYy1o!i%f{t@Gf% zIEAa+?`G1&b6YsmAVS$5gHN=w8_nbjp1YfhS)*9IRg|hJw2Ql-vy@p#h?Koq)987h z-SSrFEnOz9&3jL`JheTk3lpt?e`7&LvXp0@s@>H80#h&jLt*bl2#1*S?dIsV`Wxq@ zl<*JuZm-HGC8S??sULIsWDDLfWE}}4j}^Fj4VOB4j9O+|2ho|0A=Ndza7=S;&D)2v z3jUi7Vq&esF6J+bGo02P)GY5)UsLyFU=9zH(O1>Y7CUY&QjjIHO8ZN`O3;y?5z|Bu z{hzhIg=+tz3W`X8mR8?U<{6f^K~f68htzzslgWLO51L*{h=WBixxFZ>z5sz?zwF|I#wliR7IBfDs*^qo z_feDy&NruU`>09$ujSC>OE{S(U8d9J+IbzddoSujRISjSsh;UL?yVjkpQiEBy}x$5 zBw(VYRN$x)OS_HUD;rFWb;?x;*3&U+M02%?b}Hag`8T~uIN+XZR~d%tQECA7 zrVcV%=|6euDY#dxN;^kdQ%!ewWGPY3b2-VE8nBGV{d|*L@!NLGL{|Yvugr5JO0og< zlilVVD+pjyPAEeUnn^p=0Ys9#F77t;j z@3bWe)sn;==SQ>+W#i7dedG=+Vp`fOE38^89nY4C-qXFLS&e?nav9p~|6bJOzSh6M zJ~bxkD@9hTD3i>>x~^A5`(K3%;+RgrfjWfUpsX1{r5s;oSnA~=EZ^3sOR;dRQZZp} zFML|Y{IB<)!9MK5DPGNcp2Cxn!PZmmF1X3c`)zIN#$p+R!`?iFxE1=#b#zJxzwH4G-iV-TCV#G zsXVZaewmQ0kP6TaXw(YgK(t1p`Ff8x9~fTjHZx=P$l@i9}jS?r6}~{%Mp1U2q*KoJC<3? zQT^)W+IP@5gPAUsTeN1SDE(ieL!dqS2+X9-p89Vgdh7;Q<|HAZ@Rf=`Uyt|2AmFX zEp|7ZFKoxY<^4U}K=bO)6o(IQEjQ{a0=!-el58=0=$)?()L_T#{L^krAZn|97J4Wf z&$jJ*f1Q?{fE0T-swk*s=~BMnvCUAZ-xi<_Bya|A3+*ig$M}1o?$WC*(7PypXat5t z10B+=p7&{dHk0#q!b@A0>-(Ey#qD5x+cLlvOxF4mhb*Tnva_+CZv!t<9YA|#y$|_T zAW=fuqOC&j-Sd*$akUH*AM?Igep_-iJc#}3b(>1!N43V+4%#;{sOl`y-3rbBG1v1!){?NlVBzvOE|6Gv_HupKgJ;6LZgl|uFN4?ZEGp==uuCZ5Aql>%*c-D|&Yg9v%TlBUvfvc%>p9iUChQidV^^OWoo z!$JT@*XO<`7RX&!A@eFBZD>-fF%<~)i8$TcZ2=W4eAY%JAg5yk>}7SL`OHu&+eJ2vWU%|kQ%M=e+2u%`9M;y5iFkIec+(=a~=tQ zHaEEbhONgwCvdU!s9j~HTduxAs9fB4IwmSg-@^K1HWtr*k?A$CbJ@Ii7>5M!8}-Ed z`1LkNvh&TkRn`_O=i_uKJT{Z4SHD}@G%$u zV<%Jmdy)Jg6YgN~g(L{F{w9$YZyy9MpCSIy`Pkf6SgJCeB{t+EKZG$rHU^O6uc@ZzYMAqJ$|2)nBV z>zF9}T%BwZk>jf++pkH=MbfIXn)G5Qo5Q{C6{c$=e_=3^REXsw@m6^FT|@*{SV*NfgO2G3*qgbxGe&~1 z24RtjoNqtB&W~WQBxL!PPH>4HH;Ka#d8EW#HiQ-4;CLI3%^|TRJRK;Kw@0O17}l}{ zb4o5X+vIKDQs3FnEn@xcYe~n`yK{|vH=pW-y1{a-N>kGllDMZ~V<)-h{)Zq4Odx5O zK`=F)B3g{Ti!|Z7?<6uipW6;Mrm<&|{&^-Oqq8vivyTf$?`PpazAY!F!05PmS~aw4 zypVd`@C>;>!X(bG=^9xkWFW;af$1@%_a@r+(c>rfF>-JRjGTi}~qq*DBZ;S!G!@o3o3+n+VBZPPR^=kX%%L=fuhBs5`I_=_yxqaG?K zI-)Dm6drHdqnY$_+Y0?QsK}?`^dMzgmgTv>N`pcYye#UR27(u_XG*XSwMFNBOy*SVKN` zZs7zWsXvvQ$|qkT2yGXE*~W+?*Hja@yD(besoFTLD3%9cf&&2uL!!3xb6ndlJVlP% z{zA#sa_7j@vr*eDa^Es|MddpnZ?In)$cuht9)I`@*f0{)c-)RzdRACRZH`&OTb&4R zES|EP_S_tt18GETrCVo%I8PyksmKYBe51S1+bmZhRZP+u@0JOySg^#DF1wINoBlmXjlyWxhD6{4}zp6J=xWVPH6r_H3>l( zS~lFwrGPW-ii5#Ph-@`UwH^tZXpi!d%hwNWCKxz*3KzxnA}=a=K^TxX+_nRp>vBx#W1rGMH25>0)xmB9BVuIT~)t zZJK(5ib464C=;2b>5&BUOn{Rx%JUp?YVE4LWXgv?OAlBcCaS_#CDG4HrG+oKI_Sz% z&g{>VKY#du2zQQijK?$|Tt3|2NrV@bdXM353lg5OfX#_2%XPD73KfQOW#&5@O5q+2 zLx4d{2|Pb}ij%j*h=aSyghOQWC3jkhs|2^zFJIVNcyZeQ3(V5m5jtaC~t{bN)xxc>~%QhC=Ya)QcG+v!ELC{DaV9Qb1;Tv!ogJk2Zo!WnhM( zjQ#`BBq9)*1-i@dAIC)r15jK4O_IW2ekKHk3XxezpJV-x%<|jw?Qs5gYwx1K=iu7P zasD*{nj9?4|GVixk%Ei(<8VUmkOb!;BQ0TAl%&TC%{r#}{)ueQEk37z9T4}_M@ zJ@^EZ9-!=s_HM_)n<&2(w3w?c7ZVq+^C>SccU+6tu=Ks1G`K^!`*IIzWd8^ zL-bGy%TOpDLC8uZ|Cy##y!#inERmkLo(AN?pW@u4)l9v{JlN{Lw3If8{wC4QaGRYd z=D3crIo?=G zxJjB(0OT1!8pv73ao1|Gx#`!Y`~{R#iwb z)#ltcQc2akS!Orq-zuS`TPy$SQ*yMWlcxqJv_)F`O;jW4fq8}`j^k?5L3*!+Pu7@@ zmI4i4xJFk9T>TX1vG;2SKv(%dmJY%J@|Ds93b{Ef$D)94+lwrCLjukMcxzkaDDpxM z;iP%fl&_5CtD+GTr&+5Be!v2qz{{&H2KCH7AN@0^>2#lMUPjlu7}QD8)|y9mC+dYf z!6Aw1316Qo2PZ*7jQc263KP2L>e30#rD;@T#P!P!@uef{Y1YlxFbMA&&t@)<;Ni)bnyFQ4tYat? zkti#yCNEEhGsQ8VMHgws78!4mKB3m5rLKi@D$qJu@Z66F9h`Kt=yP4%=K{xN!~k;! z;G1l^NYI!_k|;Q)ArOllqEM(_Yz)$;ruPnSW9OSZzksBmLZzJPS5Q1o{p(9PA(tM)ZLE`+T^Exlx50+ znyE9V>c~%ppPANIQRl_KeY8DQa?gXmf#e2|y)2B3833Yk{41T|qgIUtXTIt*uNrHA zNA<6k0+GYMeqhoQ7+w5{pDR#Qf}P#4ZJ)FNU90eY5C4=>C~X=_gMP#yxx=B!-Gki^ zA5v7Ju;9!ap(%`^C>y)98#U?G3X1kyunUDBm7fq=p3Mq+M`~X!RGp++7r&%ZzrT~7 zEg(+f8!vdzVHvl-sOLqR(7aM~xbjWATBgW$zGK9#(&cBAW`VD!rE=woxA~TmVrV>f zrKY7h%Pdn1{R9%%>za1k$n!%b!I4j%3iL?A6(_frHj9i2&Df}o#+%n;S=*koBW|Bh zulJ>Qm2W!=v{dQG-HUwhqz##!^EBx+LdLU0HS3&!RQRsq(-DMMvaqmB5W3A7*vwRc z!xL5sYGwMuR!gn^lq_9{Slu=)3QddpfAUkznGv&knj z&1com|2T97^nHf#x(WzE)R+KFjArprL42SYBrvof6q`;r{)sFmRXfZYNLiTXb94c_ z`V(Dby#Rw+SwCm5>)zaSy!sbg-L;rm6}RkalRm87QCkn2RgO(?$S;8wBY~E50*D#? zb=Y+3{VGA-S+GXSJl(#&wZzwT8=u$+)Z9m!=S#e=+kCwn##Ya4b7?JRCdaEweJ8E# z;SCRPm93cDwz^g0eZ;8E6qBmD) zRC6?sH-;v!v;1MOP|oic0Ag{6)4L?0lfl=G{7tNvH65@GnyU;jYh3>_% z-kfhkv#sOsXs33?iZ|zB^z2KhCH{Ba~E<1Cr|62GSFxQfKa>D1cKo3ELG z*E0bo%5K3UgiP(Ae$F_5aPCI7n8S4TrN6)7K{t71ZEp!U3WVTFuvPit}i+|}Nz+s?PAK^}*oHv3v5ogE&h?eN86$JHBKVS5{Z;~Az zdHAs81PPxL2OOM{iKjIQ$7So>34r1T`Du-#3X{IDqc9`)^K{QNP*)}Q{WWygzT2Lp z>0-YY0ARkvIXsh%&qlmIQ+N(y8u+sd`IU%z%Se*Jd1+*eJW3OnNjxhOX3)sMqIAJb zy>F}j;$*f&o}S_%p_8$Ir@SURQG(eM15?5jJzXh|#R!<1yt7}5gLDCA&rZ8*Z2*oj z0c?Qpa>X0#DH5p3bXxn8V?Qg@5}A~@+DBsyajWWYc-4yCa(3dxt*3WACn|TUL327%QzIoBFEKYW!P?z}Y?Y z6Q7$y$u{aiAt*|HeZ-mK-o*SNtXeu7_>CDjXPq1fY!s4#04q|har_es>I^JW;x2M=0gZx6w$UX)`wj<$Em9icF)6+B7r_M3wi@q(mmv1i!I}A^EXStMI zyV!Yghz`XsOhiAYek933VDh;^ahs@jvFh^JNnK89&>oUw@k9tkG(fe~dN~}J;jF|v zdzX%68Sa$&k@v0O-GylUt!0KfGrB65VFw&~smT{Y0PeGyND`IM+d8~uB9(^d#;%fo z@m`e}fbcLrBL=negSaJazJqx4CFMaIV~dV6kN>gzn28w%jpu`uV)Jxc%`audUIrFzmN5(y5Cq*p|`r2s-Yjj z>hFHoo|wj)%=Y#!?vq)Y$8vG#b(z$q0{QhAP)+en`iN2<6zW=V95a5!ZuJc~euLRx zGv+>NW)}#LGnJw4+XRc{5e9Xtk#G$G$0Q5@@y0_nWeBgDV*GeZ9Af^x7B~Ikc!;dp z3RVDTz1HL2J6bwcjQ!m!rO2(PP?H4ESah0kKDVchp~{Ocodp>0XPK3(^fmn_ksc#a zE0(=7xI(Cvm5oOugh?6`ppN{=`M%j+{A^$G!#bH#6ZVJ48fdR3?5*4p<7mS zwjA)L0&znMtcctaL3m%eW^dSnabwM~e3Twmh@6U4V-R`e%YIYpX@*r#23Y z*(`<`4kp5mzg=XNNXWF%54XTQac9EoC~5y5TO@+VSK=V)j~wgkUlNEWB}`B%tNax2ywrG zK@OpaE0&r|^v{2t^ECu&@|7}-z9O>0Mr+@_Dn}*KczRS)&Vy0J*6Q3$tbr;Nlv*s7 z0b!trwR0CKE;Q`b_w5|J;#+=hEN>Nhh)8K)EL`5^PQnNu^0Y{FS%DvQ=efe>!mgJz zFMiivR_eSGFK?*p8P`L(GJcN}6wNf|0jM1PR(8bw#^Z?jPa7?NvjClx&DIzk)Y?iP zl!!-^(;}$Z<>ANPD&;MVeSCl=-QjJ4(zNi1jXzE+_qhB#eaOHj6JksXX_KM^`Q22F zBvma&WVQaNS{w|W{~G2iy~z$oiQ^UdwBaOTxqj-z-Rl?s@vjmEE=ScUwzV`5=>s}M zdP93+l^z+Y)J{{#A@o*pGNDj)%_WOr1SE~l8gqjJ1bW;zuSuZ2{~k?fB?jc^uKW33 zZpPZXZ)Rql>Q!C}iGQSasaE8N^-pj8{yiJuAzG~xR5DXlfNUWj14p8|@3HkCs<`wO zFp_#^A9~ z!fHr?mTSnAqI@IbxHkXq_3lvx9mOn!g#xrxnflB@h^P)miA<^is(bOpW+RbY6FE z^K7@v5_RAs*ZIp6e^{a@a0;P!FqQXq9iD^qdnNlAPvf!j@DU}NF+Q8WsKZQ!NhpxF zC9yT`&6ooxjZF{N<51}LSy2(WDFeO0iFS|>T6dd0b{jGz4OHQ^){ zmU!1&@{(E5qq;udt4Fk}%$E;`clLEu>jiHKx%l@K2b)g?eC0^O&pRL}b*3DfmgbP& zq{98a*(lPZW6>ma%`URC= zP_n8hvp=m*KEp*p7u1607l)P-TA~mW3&2{AN*Sf*#W4N!9e1W!{5jZISdBMrS|pxOAkTnl5~hkJ4?NIO06t7c|vqS?0Zwr2Tp`DQuHxKl#+P4LmrQi zE0L5lulrW>qDCYLVf{xfa5D%Q2+L%2T*lRozKbGXj=d$U6)Ll5(gDB6jN!DINrI*A zaIKX8#5b+urTP7f?R@=u`VU2ro1=m}>j1c#TN&fuAkN%r;<%Auob0R%oRcp&Ec^tv z3Ojrc@=8`5K$Mya@|=aIggme5D;mMm6NK@wLe2!Jd^h|` zU(nMwfe`N~rKVmZ3>V$Z2P#FPw*LJxm4I?ptyP$PND%ny0O}eD`SS63_>2m*_WJUz zh)-<^;IQh01D=|!YUxJfUpo+Kc1-tBTmeY~Pr=?n$lfOp-zGqxWeeo!{V7&JE()Z8 zb+{I~*pk)CS~-teJ0i9RJ(X>F^*$e09#7q8240y|O)#NxnJu^QMKQJ54iS^D zJd1DAZ;c>F+`^9hw+V*Gbwq1EGvqaZJ#d}-O{9{WIQ}HVh{JrQt>!bg-I=wiC%)$m zEml}AMx@Tn?Z-&!aJ60}3g<=V!VUv`0AUMYp?Hb4_XhE_Ty4?C?)fO#bNt{5%@PFT zYFTvee!R}H`&bkdT3N=N7P*qy1v9Ae3=lANntoc zT)q;DpDpEzbE-PrjF1mSe;i=DmOY&!Zcnh(`@l0H1~}@0;pDe3vA@Aq>n=+@q_WQ{ z{G~jPoq`?eid8(WG#feeGYpX8O*fIOj+XbqlLtPwoC@1)29ADUH3{MHz4Iyr{&E8C z6H0kvu>_xR?JKA%_L{FF)hG0~FmT{JSP$xhJg4_?lIvEu z*{0~IN^C(kllOK>!`@Lzr}aIcxe3j!3@$+*TEKAIy(GPs6b3ahIlMdYyot)o zm+^3PoSYMuBU$gW)e&VeAIM_D7y91cQR*`3%i0mh66*s5yWQ_&T?l17()2C$qqo9U zMKuIzfGt|gIX7Mhr`)!ADyYO`Iyawn%nNF6WERNbK#(b zL`m;w`>ak2nBdaWvan3+jMz5ao3e6gq*i&JX&g zBHlL=iFjP3WYP`sV%@4uJr9=J9+uj5iPWQ?hifktgV9Xhc;myaRWSF`GL`+DdC$yc z_JgN0hy|PCPa=y6c0P#&Pd8WqdBm1}?^0E#q#B_U^8S%Xx#O^}w*|!V>nR6bPH5-X z{8Jp@^*A@|n*I10%9*i29-1q{R>xU;sy{B?#3YA|Rm@)^!q^_&T6EVw=F|GMO zY<*LFUD5V+nqp7GViYnkkH$AV@X*tq+8Y zcIm>vG`!mtS5Grv05{{x6L8eI9rvWJ(iH=Pnt7Cq85a_LAUqAfF6$Kzd{ouQep)|# zIMFVCs^5O0z8S|pV)2?vK!D_97YIE@mzhL*MWI=-x($R#Eb?9C%D(XP%MA|2(g{0? zW`c^3FPa1jIsTJ!Z^90)y$XWq42{K3fu!@Age?9y#4MfI-3o zWd~pm(!6{D`JVVAhy9Vs0W=nKOfNx}5E4uY74Lxabk6KHo+{N3*SRDOOzQf!hZFdQ zPZ5bv&@d9M8rq}apeP>a9L@;esd#xrcwVj*7YLocf0tjn@anyS8Rq5zv6iR$_R!^L zutx$EC$cTkpqeziwCDIJoF>k_CtC1j%dx&xK3&ZebN>!Ct$xw}WGt_}lcZrQb!+$N z2v8=HG_n|++V+~#2 zV2uJrAUX|Awat@`eowWEg)+JDy7K)CKYre9Zv3Wh3Rga4`S|0ppOn#L^lJsK>!m81 z`-Y3s?&yzadmPZ+t$h-p+mgY*#bOAdNqUp0_vX_T?+Pf!E^0&raFU$5_UzPt572B2|{yhY;fS=#iPho?m}}w!U4C z27`L=fOCqF97aw-1!@2wBUr6U>)_Yl{fOuR1jBr+TnWT0U2=KPkXx7+0>atc1{pnw zbz|Hif@9BwsDlq1J-tAW71oi=rQEbY$xf$~e{^_gq(IjV$_LNjX`T)tMBSG*L#`# zs~F1WgQn~%J5|VS>6zQVB7U%YrsDP~ZEmCT_I)GNA^#3)QR~WH74&=Jgl|ITyyo7_ z)ikE%72&Zi6J%YDS5V#f$s*=~Ux{E6sAcrGs<1W=Ui6zHq0Z1M47UCZd*v5|lSp(Q zK)-}inQ<;f5y+9_$Tg(n4R6jLXW)MvJL^7|<3P-N0=Qsqrm&t*0uwo^l{$O=g=ZfO z%LTrZxVb^^$tf~6KM-P|z;D^%RTRNHl5Jx7DTFr9Ul0lLF_gir1t6BY6%j#T^XDU6 zqv|j-YJFToVUttI%b{b&ur3s&tdSWj)I*2#jiDY+Nbm&ZpNPYAp z0L=I`G&|WWtYtBG7nn2QujO?VbgX0Zfa#2aAKmN%IYIq>n%Qn*dqO%ZOS90Y(+5B@~ZGj#YOLYPcV1=FJ(gX*bf%HhM)i4`|tP zfWKfvk4w%;Z+k_rw+U!#gS~gNt}2^o6_jvK`oU_q7UcFLwR9+!=BrFGM{hoISVz<4;$iT{R$# zJ|vF9IXqmm^puG3c;wjk90RZk=>d-YVkrP@H;DtULTKm5ut{=X!tc3J0r{nS9H3jJ z1S@xmUl?@H;(XQI?q@WgCP`MFXs1q~Y;xXI=#fMqex1y!l6tT#d~yD4j3I%{W@y9A z#bw9Eb#Yl$FM83`2%jV!=@Gcxqf&RVG00M;y{79!e(cRvcy-fA7k71~s%mjM_|O9z zWbE0~&I)n^_GPz!1oe*k55K-Y=U9hMV7_NorzdCoOk3k7TEI69e4?r<3z(Q@lV@+S zZjPKlY*6CV93ed)-scNe5j%EuZF?!M#;+}KcO(_Uks%%}pWe8{yv2wq!A6qrhQ^+BCGMeG4QAjY{kd z*WXsJF+qH8B!?rB>H}$3m%}-}E{TOkrpXIHm<*7Mf+8F4pn$BM#s;sQ_OOVoA4pq^ z`19LwS+#-_d4j*S;7@oe!5vMaSFP*a^X;2rAqWM4^@gV6?R>2;0$n&#Z9C)@%l7PT z9oO^DNdi%%dxZ3kd|J^C$Zy@hu^|(|wHVEyg8m$vt!od9x>I@cz2nl2@aU&3$n92&nio_?njI%GhR_8`)a1OsTD9u&AABI_VjonvVbFRAie{rvFBhDeVmFC`($GO%srU8-MzP8`66%EgDthbY@U39~ zazFU62tS}BuJqgC7od_tn|($2d?wrEN&lfI8a+QQQG^z!(!|aqv~H2k3ia>xUBX}d z*u-Ekx;EnLNH8EiKx1zWAtVR0$z;f}&@jhZ+y9qE<8MT2cmGR%u$Z!kU93P#J}#d2 zyWMwsI-szMMas9Jk9CT{@(pyF%D=2dCY`%bmSo?rM$Iz%Z(AXz{|6ck%p7!gxc^aI z%;P>7*7Y1EQR?rSnEaQ>;xC!@K|3_r53|cWMht&`8PuKB2V+cry%QZ>l(L zBDp$Gny;&_94CxNtiD+Z2XB=|J}>8C#}GH!$McDk#;_^}Skb~SXy(he))~YFjfNfz z6`W2f-?gL7&Gq+-r+ z?rVJbpOcAJqlwGfKVoBTe}po;;#eh}{@@Z|T58np)p@->5GvP!RFb7chQhR#xSTp1 zZ%snBe6ag78ja`j=|8%x1k&$-mA-44uCGSN)A)n2$DbW`^Hq*yKnn9V8DY_bjeq=* zNGcJkZ>VZHV%Sgfn^?D%xnpGCM)y;``3z2ro$HbnK5hQN0lD@DXCv)}CqWO@xBrlO z7?6OXGve*u^8*=J5*3;Ic9cMigSU^iC*?TBV*PoSztCa>4PovOF0Zc7ucjx5_F9R@ zQAeA>uHdKql`iJE-LXW6-S73&zTG^^C2>g&Tuw)2_Q)#u)oXR8hZTw@oH%#ei?t~` zo1wb#HO7S{H#Xr3)DN`)f5Xt>Oe^6SCkA(f=x+V3f>!r3G=Y{|x zJ%+<|I%~Sd7@yspVT}wvX$Ob*wQ9w^@SH;jp1V%FqTMAhojc9R24LjHOC-(@J5G>h zeXDldKW$lC{1W40{o77tHQRSY*Ei$(x9@^B`%bcM^w3&+IM2R6Zp(bd;azX&1V@1* z*}oTI0N^Da6Wtmo({knuh0i_qaHFeGvEcI%!^^CFS&k-=Xl|?iGk*SLG?N1w!@;BB zz?XS%UPw~XeDq4$92IbLU8h}=aLV|w#ULuapy8e`oKI4!>0dTwg5AkQ*aDYhW;~uw z4iu+rDw>^Jy`C>e7~JeE0UeFKiiWoPE#wy31!V^B$Da%~E42(pVwN#3XG@8)iKQJD zFHho0G_A73$?HVxKAu0}V|?GZ%OghAn;$LK!>PgHsB@3KNn23X_cNEvB{UQGHl=CZ zH~hI634Niw4d$tC9c{H;PGJt@TtP=$=;y5D%|v?J3K`IXud3a zrB=H;#sw(mg`Jb80gO@QEs28`U2GB-7Ks>lIW)QsDFcIX&3P|M7)R077IkIF$oOhQ z_z}hllK!wLTu-1Ibof{ic_lO$%6Q(-S9W)SjM(*G`iY9Co?KCba^ar^%u- z^^mgj^{Dz{Znf(7xEEE=+0MH@b@n0&)y(@HNViZh%Y#Xk8hb-}c4x{V<)x-eFUL{s z);}Yc+v*8p9IDtQDmOSdRW{n<&R6BE5@?`rD#bPPiR-TbZH1&!&#SE#u|Xt?8ju+V zo6~O6TvimK?c}Ojp5OmKc*1vr8gQ9oMx)^*Fki__Y`p{s=Y+cusZ5AdszlBl7uF~u z{ijXU=$&}jb1-Q#<{)vpByN+7@55H2lF)16O<`jZ79-1ZEME16v>1no-lj#NOhev^ zJ@$KoV3f+AQRWWf>y?_%oJw8SwjJi=+@3u7!v^r?zwL62#ylhpXFP`2eaR@&<>kds z!@}9|OLtybDMj(lz1Nn1CG+}D58-hYB{c_(C!Yd#hDn85VsDI-ForvcPH&l5+-jYt z@%pG#Nrv)%|o}G@JBsM$b$Lh8{WMQb|)nt{_xhbs7)O#o$Lj7(s93)DVM$C**zM^S zbSQ-}erIe%a}trr^&K?i`%JiT5;{ac=2|Udc(un^5|*K~pdOIUZkW>9E?SUd9mCt9 zIGmskP*m7je>~TB8s}?)w4co6gG+Zk;chpCS?>y@kiS%Vwv$NCJ7?J>v<}*`bTZ(V z#TGL8I*12DVFRuAf3Wg02bI`;p34~=O=wha>^=e9lLbJu(HiV0{N?dNW#HiEa{bd< z6E2ywh1soardK?*u1~Wlp4HAMCe6)}F+jlg`ZStM$D5y2ugzegM34LfZb2U~qjhVH zJU1&|SHQ~}C!+C45>oYv|1$71?$;NOr_&B%0{Q+j^+9FKNEP7`Ry+JdJNDdlUEfvr zIx~ndJ+yzj?)2Wb9E3a9U~Qvbwb6Dn(4Lyf?V6VS5QYeDd^`_&!u=oC*A&`e^CRqePv>Sf9pt~gbj4tDWN^Et{!`GB)d-_jI`Ztbu6 zAk<;8!>g|Qa_Hs>OtzVds|n+c@GB9xoO|?+7Jq4*K%1I1h&%^x?0(w8^d92QNhzqf zse=TP-?UQxFh_j^DhqOQ2$h_NWRe!mx|gB?YTvIf@xhW+ZkN7u4yq!e3MNMzr17EFQUMoCI-2^bzj~J z|2J&sr7$&T-6T2HD#vH#&m6{?{~`J_yf=!Y2?0X%n+E?4*_!$wC!O8Gbw6-<|1XG3 zAb$TuDkr?r;e^(X6|vP}O;AiT2^1Arcgd;%?%|8-y6nFG2s&gOT#6Pf?z z4-cdzkaJp|1TH72cGLIi8I{rUx@$x6@z$htoT5YPJKLo z8s~EQ`S4^j!?`%Ap@mBt9+UO#&aOmn?JHy?{#%~fypXogKJT;YD#5?20^nU z??&&;dNjSENg4}_<-BV0Is2&tqe|&~uWl^L;@iW1EvartbdC4ZN7WZxPPmy}FP*qr z(_T%LgZoQRZNI0is*%kuRjtbeo#f1gv%;*`l9A8c)R2(j6b@%)3Gcf_s|?T==8$E_ zu6VregDCMkzG1i?{;8%kz5*(Lu_bKxv~uaG)@_r9?H4_40MVJH1c93ZoAqo8{Z?IZ z_hNI#8Sw!BdR_DyOjpY1(L4pf*IR2GjCm=vTe62{3dRVOStyIPCZ%OEB;+JFpv0x5 zD1CEI1s)o#kJIqALamqK;tU6Q;9Hjf-N9cuf`zyv+>d6c+Hr;74O5@S0VsW>&fGVJ zt>$pXbEt8hHcmVqZMeqxT-?d&$otCaV3JQHCNuw-g~gHCcJv3kg?qb# zmXv!>j;D9g>V3vop@0@u*EH`mmebf|-HwHPH$dfYRGFvz`Enp`QbfW+!L*STD4C)>$S-gCz}z8v&*w7xsl)LpHt=3iE* zGt)7jEsFS)X^m1|ZRopj+?c0Ssq@85-Q=G&gLtPfeRX5DRME6ppf!~MU|b|zX*Si4 z`z`xgT>SHg06~}qZPy$5QeiUl={ye%QSmHw$KBzh#!YCnHVWsXdrAWre;)6p;#}vz z=jSU2YRb;Qj@=Cd)-H@px-pxNw2v z&l&qqFz5U8yS?dVlFd8`z&Bc~kp=+rvT9H_Xc0W3uU;1$(H8YZ`xz%oA^rRJmx>BC zMm|FT+t$G8nT0$B86jDvNRC!H^%b(nY_kgEtKWLO>{>y^;2DGS5rZmbUnQWaG(6Wr zqj}5Tz~p)B7Ujyh_v7~Mb{vGO#4>#Q6KDWvfS{k;N_!J03IrVFQZ5b3l7KzD2F{}& zNPZJa0Gcbs8C-uxb~hnOSK3?i;@FuKLo@_LITj9F9;`H9kU(44u<1rP{!(ulCygpin8UU@^!YT!Ol*gz$r^|QOGqR4e z-hEI&{+C{%?b&57)oriSA7M<2pA~o{grgN(YYN^v0>huPI-ao)QHU`$zRuefX4qh5 zKF1B(-y0XigfkC?g5`SGXTl4Kq;~hFnoZ4O$&%9ITq=!9xud_Jt?HcW&>s;vPupvA zs^@?6hjOgiK(ff=2BZvutbX}7!*>J(5COomAR`DMk zc0bI2^Ll&9Dv&npV-8pkg}=+2`ngb42O^y&qa-1hM7hLb)J)zXUldl9sd0X!x(nM}Xqz!AB@7{JW3&}2r~WLhrwc1H3pG0} zZTTYX%EAd7qBC-Ya~uhz*{|qNHtp#=g}8?C{JG(n@z+0OCvVIi(LqUdZ9FynSS`aaKeQ0(^AD~S)UGpgcBdP&5^RKfd27e-8bf^Y}UAg zbq@y`dznsqM(sL!?QaphzrPp6*N6OF&yAB24hCii>8o(D=bi1)++48BInOYuZixLj zk*dW?Q%i~=S< z+kBElvn*6^>ADwrBn)UOIMn&P*m#cs-qj9wF!e}%DmPPW=do&vd@v@f&|z1D%2M&q zPJXdQJ>2!hCbDD3&Ih%+92i24xT1G_mk4(rlJsEV-7x;$OTM?LH8lu5<^J8H`G-rlc*e zCveU>yespY%1=HoUmT;l9%t5LEO+fN0eiIdXX2!}LevrisLyiJUGmM`ows82)Z6u9^0~{YLJu z=vAM6hYxa)x^AuEVDf2KA|>UvQB%y`pg#vT`A+V%Kf`>>A55u4obX7Yt=Yo&dX3%V z;t}#i<4BV-B}cD`b*{ngQ>wga)zRDKjOdN~Ev;It_9lC*))r~r?tbH>uWZFkE1z$~ z`Ht0$*I4S@bgMr!%CF+{8DL%XEf%e75?-CHL{P^JSkqzr|F>&&V0>LQ2R8i3MsYU-u3~ zqo!RH)}_R->D!!at)TYUkt`jGu?sT#%}E#A`G2)Emd55 z3e~E;oNkV+u^5arAcRLw_~fq(*h?qV{2DD-)#Hq_zqngqoKhS2P+l|CrP54INBK|tdKAZIX z>64jW@HvJ2O3=~c?$d(P>Us2(i!&6GR&)r+2=Y#I=iV#hsp;%Jv`(HxV!EP7o?iap z4`E?w$82n1F$D;T~ujlA%`Z;gBMMk4o`3< zDynsJWJuuu8N4ncV0Wpvs$Vt}D83~s1jmO!BC(a?udzF-#5cO`BLo7jQKA3*{^7I` zPyr5d!<)0+*=#D;SR5&BDocO6%AP(8PUs2pyT*@T$p3Ag{IQcQxc+uGB2VF6{{w@+&?~1X9t4I?(#hiCQzqPMPaipoVZL? z*shBQDMkW}SJLLcP608@$0Q-l(0K0G z;Y?uxv!#pOiL@&~tL7d{YYY*IeopcATh^T8ZDH1!H^O;Qh*PAilC(lC^kVaQ13{He zPA)N}v96oz*j-Y@yz6l~9Fl%~nIzJ%vncn7{_x0q1e13iKJy(OyvlQ@=%6YL;hD$F zO%4qD(p6@jsJw<`n|XS^iQOE74<-WA=Bm5;BPlb1 zs8u3L^?=67ll9zipv4^JpmirPYh#2J>eE7z_U(yoadm9(SbshnD&3i}3r;h|zhj3e zNr1L*SA%nnRDC5{JvAX*eKYLvFV1+69q7dQs5NxsI!0;!T@Dy7Aww?G1wIlaAhbi$A5o$Y z{9vK!a>AAhha0g7X~SyEBgE|?6^==#?eh15A%gv8Ibgt~5<0o7K&DE`mRQoY!&Yh# z=XN~4u}4X+3mxzz!;-c*{3yDktjI@JGCkij&mE!(=NKaz?gDFh0xKu8@yboW43C&6 zEV-wVKVe&%#s@?&7EJg8EpK)#*1hZe*@{VHeZE|j+(wIx7A1pqr^RqG z!(uLCswX(@aHHE%sm5s31Tj#j)k+yyXyo^e*sm0FS22@dCG~iUGh6V=I@rU^jZ$PbF77AaMrGOd zIaH^g5yuIvJUg@&a%@G`?~Hmve&{6RXo)8GlMNS`3P357xrRM|d(q4_e6~_i%{}wQ zj!|#GlphiqxGMOi=VjdjSGXZC-V#EkazX?j!!Yn-u%UeNySh2s*Y@bWFCONnlls(bzs;lF>60aqIBe>)u7m#R^Jt#@Pc*qootY*K zXI+a08URttuaC(L zL#s>1FX0*1>DgkcBL2V*3!vYf6(dT%#mR@m=NXCHb6~gw^WkFi6Sv2XXr*o~2sP3B zYZMmp{#J$1&7CCaeT?WmV^eb{=sz-tiqbb!^Y5t0rywk5w1gUp^FImjlGt&LF(ak2 z%q)XwE}-WOUQ7$NKr((wHKVv&{&mSA{TZQO)~cbnC8d>Go?2&rS1_lM-s{i(m#O&W zwzH-A`cNeIbZzgC&%I|mR@(U38X4@4QE_Cl0u+j0>T{B9nDfm z3x#1SkQO! zDlPB~pt3%e8naG{Dp&e$a5BAo!JE5`Qw2)}U#6t`0N$cAv9Xqqg{-|IdVQyt>srZF zD(_!X@H5nVwTf;ELS@6D8Y!`|X7@I1H-Pl$fq2{H&C=Q{Bb3G#x1butO}90RseIQS z-ah^JCbQMYr|kMb(h@eRw2+!u_l=+NZjfOj`1Cf0EpQE>^38jXEOJ`eX2N+WUr^hSK{d1 z;2m|eR;hM`Z)dPnGl1-Ze7p<`9@bJj26xqtj!RXBj+m!+dFaATCTzTH$`q_)qaw$} zbuYX4HAIt&3hjRZF)2#WN*azT=?IC{KpzzqK!pG_oiZzAL*akEk3FE!+}mJacBV0a znZck@K@tc%*eU&`^yaTzA!k#2J_RXmF7;US_-!|UwpF$?SV*h^W%AO zldK`>zt-G|KyNRZgu_bmHXDM^p6}v`AbJtAUKbjTRY)ClmV(jbJF>~*SPa9jwVSe8qs;$B$5vwJGfQ)L|mErvo5CmO521R2UO{T>K{{t?b!tgNn3>fADY*w#g3DoFV zMg-8z7@TH5jfOHA=i?-K-%j6&^+1U4`xCnM)W4~QGgntn4zlh!1N82b>DGUE)3W}3 zAo&cx6y5cr4pQ~bInXVKuWtI$mFm3eBb8v^qI2$Hj7Jo(XqW}98$R&c^-O3ic19kY z3G=xBqSrv^6HYXwN`udW2nr~S--i!B(;`@EObFZyt?Jp&X{Ha zvR)EXU9f6Dkq3jz$(CMK?GPR)@zBXWY$lanH&P{9}y}%Q5oG$ewrSm&J%Z zj~Magxk0q3WKnQ!`mbsim zgDtPf!<8xX%+{4{w?J?LB!q{nqa^>@+9|ny&%MZ}UhObs|AP548cjGyi_ojG>>DL0 zEFSNthFpB#^y~x%PcpRN;~JAOBnv#p06j0UquBvkHAm~+%)iEm5-F^|z>@_!Oii$B zeSO`4(R7~*EsJwUkmN~gRAO{8LkPkR{lN<4?wnU8V&r7eJCS)XGk{CO4jB+)nB1!J zyvr=8HbkFMkSX=|yQ3qtGj};ONcrU8CY|1tfJTi`1o9yKH{P3y@_Tqo!(4Ttc!Crs z@S}l@VSk%NYxv!jN-CiuCbHSKy_Xc%;$l6@Uh>?-E^yy^XwV?{_<6pJF5gHWJO+<@ zTO)Lgnlg<{R!60|OK1q{jWanDh#J{gKHB4Q+&+aj&7Gzf=KK6oNhnfT2umXiiPkDs zh^4E!+8uG7dVWZQXl|&xFQu!2VK(m$NT(%%Ib39916Rk8-mU(-5)yzz%7Lw`JXFlSFG3|9IaCR@^PFONtQ7A)ZBVHWU*IV=!K9<)HZ)~knJxg>8G1SOS3Yy*cP zV&-Vh^P0VRw4Bs({+g1$_4BQE{F z;5+v#5PauejKt^rVuHZq^W=*}J(lK<5=BdH%+u&ryxf0#y<)#X8;n%%BO_*IKQl2l zeqRIXkB}j*zg4eNpt05XOR2*2<@P6|KE+$Q%p=A7>`kh;iVkJ!GYha(-s7qYo^^Mq zjelMX9qDayP?+UU3pW7w#trrT+3!k7%)K)snw#Mm4rez`#ap2q)>lHx-u}_SL7Ri` zT2!u_$Eb5q?Sazb<12aVCw`*Y-E@*UB`EuvdD}nxYDCcY5LhIhFJzF3hZ$2=}ri!fj@IZDP`f2e5|89wpT@c$xjn6}zRVhV8x zlO2HFXbB>Zxw_zxKQ|ECzXNS#>oL}NbX0LO@@^E_>LUdVj~77|%cgCw)MZmvXu0jL zVH+7e)aBZ+W0Kn)m@^%Xw`b)rEO;iHi~!5S{9UP&hu=i&z7q%1CdMB$I6@SQh~1&%>Y>#9ZaND(ThioV3^TJtlhC znaKXcRV)4Xsh{Z68{O&VS;i|({nu9$@dw+Y*5^g~zbjVB0lAlQN+l}CGKk&tCbeL|VyY+2V(uuX!Wl#o8L7&{?Rv|BFYv(kO;hH44U7zI zKtoRC*NA1+5ulIRYks=q{rJj8xL7bI_LI5Z_6@sSt9kjzxxt28ioVn3G?P*=&yo3T z*_ERYBVTggl=R6h?5PF2Xd$d6D((}9lF%PMh(0kOW$I5@zzdB^)(VUP3Gn$P=doWj z6mvf1tYun-)@TSu;cTK}|BF$^+dkUYtQy=gxUL+tS0Ihiw1Q$pL}<9$@Xg*m!-vV$h8Ix2k0`N(0Q@h7 z_lvX4Qp0JHj|cgeUm&{e98~?lR2rsSz~jzr^=r;CC*YdyyPa?tx|X3LeLLm@MrMBp z%FpJrkL$j&^8FFP6+3Kf8K6~Ri!TVvc4&R7UCAnPX z#H7_*hWG>3obv0(P>_Ong{bfvyRETKaAcYzS*5V;aww&G-fA~1@g9%c7hDIc&L^SE z=16I;XvSKSQ>^>>4<9hp#fA8l1l_Gy9|ot=7F;vR;aZ4(xA0`~AUKbhJ}(ZPi3m1( zP2Rh|)exEhs2l~UY$i#cx96JKd|8P?v|LA_PJ4|N;uc#S0Y7Z7cB{-mxz^KN7AZCr%Tr?k(6%?8Wj#DMSE)zx zxzgqukd`JABT^acPt52U>03oGk7sA*YVKoVr^}v`Ni?F@XoR}DRWhT@#I@w4wQ@Np zR`yVjuS-pEo=k&S1ov=Ze_@O=QoR>n?{ZxEC3ZM}X~p`Jb2$cJ0#Y{oARCxE=|;zh zyh5!-^FAp@gTJ;d&etp;P=DkOtxAhcBuF?o#5lRFe)_VBmi|k<=lI!xtFAtu&37YU zI;>eB)(o`UkV}vr{ipF!4^z}s#26YMGJklQ+a=MhZrb&s;-P8(Iz0X(ML#7_NNA|r zF&znq*9;%0Dus?v1tyF2iNylJT?P%6@?mA?{FE#&a}EsVz4kJ|GcYA0ky~v%hiN^L zn9AfPgAA9j#syadq;^B(%$AilN&=4&D&!O|uRMzr)X9y3Xi~xdBp7;M4hlViPZOA9 zR?WMwaksYK3D3ki>i2QInrilh_`TM|2lEA?x)XyQqfF4M-HMS&Eng2-V)#^Tg!#-+yXmCo6I;9J_s zpgfKiGhX7c+ms+uMHph6Uux(s(A!jgLxr|R(T-Fxn|SCbEro9PK&`IGfyY;)hl}g# z+UT^0o~I(mw!;Tn(u1S%+Qk$Lf|OLovpRE7Y90409AQuf=N8GUbh@3vg`sWwE}CHJ z^Vvxy8xwMc9jA)qd4pOQK(l}TX3{Dy3THDOmQZ<=nVQTwRmN{(mnJ3J{;AWpaZd29 zv^4lU=P+Og0JHotWC2WV>Ve8{#@l8w+|FTKX{e{;%5j4YWZji!a_eVDg|*Z+?qDwU z1{+PDI{C=Qy{m`34B7!IqNZ*r<0GC?`3Tc3{Ml{n@-#%=4&^1J^0d6uu@)VbMnFgT z)8L_+J4V-$d=xfOg*e$gw!M^Gs#Cx$QYc)8sy~28cLx}gLm+i}J-~q{yKcMe|2{&+ znK3?%%T~+aat{XP!J}z6JCLnJsF+d;1!k=>;P877bHAX^%UrpIj`~Gb14R&ew#ew}< z`PFJn-co7cN=nanm_mVnl9t+T5I< z9SqGH8m8Xja!UL1u&*IdZPN6#!SVcZw`*isBSUWc!|>#HKQK5#1l7=3mcJ~e)=>*C zAWIaTZY>W;{_GGQ_2E)=bF)6%Yo)Q`=W@E@QL=7VXb(a{X`K9C@@=pq-yDkHcY3@= zMO~=3n8PFPmt>WzF$zzla_SBFRMI}qVBCZcDBGkHdV_=9I2W?OE~oQ&zZVa3vepv@ z+uX60zjDn}{>?Nd-xsjJ7aigGHP%ojBDHBT`r@$C(KAfr;?iBM*Dj(GwG|WbQ?R1< zJOoOK{Ygt7xV?d4XAxdN~fv7 z(W3_R@cKOGZPhw~?ArE)J(T@OBx%|uNTfu~i6gz5{uK*t)lk>v9j55A2^j!itrw3C zaf{DbPzx0{c-T9hG3clf!Y#o*3t1-m_FS9H;fytiRt|Z+7sU&UbD#0b9&oT+S1i1? z=2&DrlH;{>jtf2?8Yvq1ty;AbZ5S&=36a~=;UqV_&PrWdy-lM5N2K2JF7Fd~;#%uR0q2rY^JXCc+HaAkJHVWQ2w5!K*o0g#j1pS z^WlUu{f7Gdk7a6dD5n_GFe$LiFt;!0QiM>q!t8=MJZ-N0( zu`FQ?2c#t-lW(JVEBUe}XRH2Yx6agks-utUThXC$i zAnFNFG2}_O17~$As5W{31=Lqwg?VI&Z4v?MD z$NM$GS-YLbZgB{;fJtr*Q@~^b<6l82pe$Zc9vwZoA6)4dP}@)+&BXHkt`X_^qH&t} z?t{Lt-I%y>jB9c3L<+QogrR_%SE^rC_EWUAm1s!)vWTIlrseyGm*-;s#rksoe@P0v z^TF?PQI-EI{R`>87ppgu$t2O=pYX4L#7Xl5?d80Eik*Y>Hxt)8U23O{r36hmN2l59 z*(G|v%HK>3zyzi)@`sax{C*-|T#%2XY>>$}+Gs5e1+;nGgw(FTu!6g{ZLzDn$5%{$=nE~|Jrjmctm-At<#{3!s| z+sj2R6z(S^=3@10fi$>-%RToas&r~y>tgUE||+&TUrC#;Qz;fx-fPBqC547AqQKbv}Z!n9Q#9Sa968nY;tW_~lZ5 zQ!G`1f4p7Z6!8=$o`uHY@ydMU$);P%WOr0vUtjiQ@eh~D^jZ<$wtVrTCK@jqs{g7`&fbBk`eodiEi|9aiG@nAu>Q112wliYTV zTIr{V{W#8Rs|J+?p+Fa-WRlb4qxZ{WeQt#|nS3~sdy}lm<7s1Xx?3G6DqZGa2~G9| zy~W4eJ)Li8bISPST!k{)IrHVsY)w%CaST8eOCPM17Yr11`sVm<4q~Bl{bzUJ8^EkD zDzx~r*75RP@`ZYj`kq#)I#)&0emwmKRM_G9o;HXJAV>o-Gx{lMu>z@Lg)q6YTtk7( zsSHrDW#Z)O7!13^V#uROq)_a#!u|)H-75f<_(z2uVtpMWc+qV7Jol@T>z6yj#5kZL z3e^CE@{`?>rRX>MvtPG_^W(Mht39j?g9b2YJ2A|$Cn&!}k2V52nFhN9zh*eGbDm0A zy#^E=4i38aU@AjEd3d@&8c)Twl}D7&$S4TVr(xM0&j?4&TLIV^AM?#NIlLbe>I?_z z0U3~*RM)}iZL4Z1S^+?Zg+qnqm^{BxD&@Is#Y>M$RXc57t=d3;srjl&XKJbSu1+B@ zr)iNAlOZoxr91$YDFdY4?NX&gqn=!|WVnQIbw!~@KqZOAmZwa$&dWOsP;n4*5I_ha!*WsM#OsGQm4bI55xZdRf zmR0`wew@ibDAcZYDC4Qr?#9_wR=Nln$B}&@RFEfiC;^OKo0Eks00^L_m*p*pF!Vfs6C(Xv=Ub)%)YoBTSph;t3(4UbEnlBD#OgWrS)RMOs zC`)JyB6Q7$9n7{zyx(t_&xsHKL!I2?O>T3(HFp!Qt^*MEY-7-q#UCSljcNnd)uv7C zp2t6rH<}F^)Y(Zir86VRwzKPs1x~e#P=+SvD(em;3!6VhCw}x=N~E*JYViKR`ZKEz zxoHsSd0zPcdb;v>sJAzavP;O$*s@L(b?uB@vW#gkmZ_WlrV(Q=Tb8Uf))~q!Th!$MF-Kl{h zC&t0m<9Zs{`6?E#*B+|GKsFbTd4@{s@>-4D_=05+D8zp?l8YfoU?B~JFb@pDhWj6r z%9bQfh~v3VMr7~W)sqchO>unH2CV#r$u_iUswX{j+xkzrqH+%I@TzSECEk>Co9e|Ai8k?SS6fK0Tr}$FO#Etz_deS3jE7>0M)2v9ip$9< z(kw()ozP$Womn8K*6Bc8+Y%%nBS>?#@*8W@rgHr8N{I<{X5bW@_!tdR-2dB*Znum8 z3%r$75jC2kfnm$*y8i2pXeG0fce7Q3+St#nY8_EiJMS~_9J0Z8O}v=w^(|gu6u;7w zA|#7ufZ%2A5IDtvlW&aKH1|~!}$Z^k1P`f1D5kdFmvpQL}`+iLqXdzLFsIJ(3n_!O74R*Dsw8M}+Ma|S1?!(D|KW2;IcxcaO7-@+J2+@74@>TkBj0{kX4$8h-16z&)cd*z9-ookJ}q{4 z^lFznuB*BjUtp~^vB$oA8T+(>vDEZYchb7z%*m9f#a@DD6?uz^Q;7GQUP;lR8ruOs zDQVmX7b}`lJUF1LllSNqt`Y6+pG&@={FoJ^qvUIclpx*bi;HFL^WThS%wo}jtJXL0 z^5}vqXi#4gI6xIrW2fq*9oj>JZZML($xwz%`9jtBdk=L@CJ?Pe6@iEIVqegJlMs?(Bp7tQ8OhTh!y&U32cYhyTQZI?7=mwvQ&5%C+F$a#>B$$yg z^l$3y_UNmGKBgYSbSX1)oRDoXAk-BSZOvR702r-*A`YuwCjg$xJ zbBTTQn+sg}SJvwDEwb6meGSRZ?62+4))USqfHXY9hM!nIuW65@IscsVuzQQVDZ^3O zGSS8XtE&QIsL_w5+Y3+HY=JqRN}#I?T!R(a=yM9GVKbm`t+3$JHY^N{iv?z-IX0lD4=rP4qp3V&e%dm`A#*w#G5b2-b_ z`s+=CKrC^${zn;KKJZAtliyp;Z$(kvKQxC~*E*`88lVU;k}kKY%+f*$DhIi;-dZT+ z6_GdETqRgF^8Ea&UY?Pv86*v-%Sr?0o6;NONsu>V;Q>*dgXC8*r%t5cCHK0D-?lG2 zX^ekA%c}7a9MXp9UbOh%-Q%cv(HxBwzp)6P$=D1|y zBahxyrk4fIyCHIO+3%HzOX%QUY#8T+t1RLsTZu!X^P%hkEFsBY)@IuMVA=f((z!?l zcp{#tdhgj(Wj>=?g(z6)uvgiG(?d_d=I{H}*1H$qgZuO4+p24U`Q{8q&QZWpWJ?j>u(uc_y;3Hh$@h-n;lHe z#PI#La98e@*zR5|HW{O!$P(A{f2_#jyiHme}bld(pyrVJg0mzm>4~@I)7a0qv zDAVaE`s@`TEAK32lb>1MIiAhxHF@jIN9-#@jaftf;6B-1lp$JmUCQiM8Ee3hjOEr< z89Ht4%9CSv_g`Z5 zGwm;xG#r;|+>%rD^7Su?WI(zRa%~xp)XnDGSEkbYMO;Bl}!Sry0N2Qw2 zlAW*56=34GDb7;H5hP_Ds?igRqG6&XlKiDXx+&Yw+Jix&AR6x*-M|UNWE5#LRUy9V z(5eH>9eO2?zk^Wts^C~;sX1*Ub)CZcS$$reyWb#g@P|k+l>StHTNrvcSqF|#e`Zd& zXcl77%n-^(-;3Ve5?KYpB{UEfky#SSyfgQm=#zK1TjxhJ4b(zH_yRiL zUZq!mDGk3?;hRUI$o^cPR(#ZW$25Npm{7e{-fTulk*shhQt6&ND}@ zn-tPu_^M2js$88eAre1!@<^TQw>@|2Xn80dcGc{^vN+=!upfB|AGDqH{~ElLcUg+^ zFv0g@r^V{{TF_t#A+b8RT3AGHC~p7~j;q)Gx3shl4-iCdX8J;=YG0XP5vooUfx!sL>5P({XzqBOOAORmSXodbIZNC}hQ6Ag#!X*2%d rq5E%-GT~O(<~mXW{$0aSOnaha{IlV&KZkD9fX@{pxM7XHW7Piu=+!JK literal 0 HcmV?d00001