Merge pull request #16 from nicholasjng/testing

Add dynamic option sourcing for builder, runner and reporter classes

Author: nicholasjng

.github/workflows/ci.yaml

@@ -41,10 +41,14 @@ jobs:
         with:
           python-version: 3.9
 
+      - name: Install dependencies
+        run: python -m pip install wheel pyyaml
+
       - name: Run sum example test
         run: |
-          python -m pip install .
-          pybm init
-          pybm run -m benchmarks master linear-time constant-time --checkouts
+          python -m pip install ../
+          python -m venv venv --system-site-packages
+          pybm init -v
+          pybm run -m benchmarks master linear-time constant-time --checkout
           pybm compare latest master linear-time constant-time
         working-directory: sum-example

examples/sum-example.md

@@ -1,10 +1,10 @@
 # pybm example #1: "Hello, math world!"
 
-This markdown contains a walkthrough for the inaugural pybm example, `sum`. It
-is meant to display the usefulness of pybm in the context of Python library
-development, where usually only a single implementation of a function is
-maintained; therefore, especially in performance-critical sections, usually the
-most optimized algorithm and implementation should be used.
+This markdown contains a walkthrough for the inaugural pybm example, `sum`. It is meant
+to display the usefulness of pybm in the context of Python library development, where
+usually only a single implementation of a function is maintained; therefore, especially
+in performance-critical sections, usually the most optimized algorithm and
+implementation should be used.
 
 ## Prerequisites
 
@@ -17,8 +17,8 @@ cd sum-example
 git checkout master
 ```
 
-You need to run any pybm commands from a virtual environment with pybm
-installed. The easiest way to do this is with the following series of commands:
+You need to run any pybm commands from a virtual environment with pybm installed. The
+easiest way to do this is with the following series of commands:
 
 ```
 # you should be in the root of the pybm-sum-example repository now
@@ -33,10 +33,10 @@ pybm init
 
 ## Setting the stage
 
-We put ourselves in the perspective of the author of a fictitous Python math
-library. As any good math package would require, there also has to be a
-`sum` function, calculating the sum of the first `n` natural numbers for a given
-integer input `n`. Currently, our author solved it like this:
+We put ourselves in the perspective of the author of a fictitous Python math library. As
+any good math package would require, there also has to be a `sum` function, calculating
+the sum of the first `n` natural numbers for a given integer input `n`. Currently, our
+author solved it like this:
 
 ```python
 def my_sum(n: int):
@@ -49,21 +49,19 @@ def my_sum(n: int):
 ```
 
 The code speaks volumes: The sum of the first `n` numbers is just the number 1
-repeated `n` times. Not terribly clever, yet of course correct. But as you
-notice, the computation is really tedious: A nested loop, with constant
-increments of 1, each time.
+repeated `n` times. Not terribly clever, yet of course correct. But as you notice, the
+computation is really tedious: A nested loop, with constant increments of 1, each time.
 
 In fact, this code is pretty much a complete catastrophe: Our function has
-_quadratic_ complexity, meaning that the computational workload scales with the
-square of the input. Without even running it, we can assume that this will not
-behave very well when users want to compute sums of large numbers. Can we do
-better?
+_quadratic_ complexity, meaning that the computational workload scales with the square
+of the input. Without even running it, we can assume that this will not behave very well
+when users want to compute sums of large numbers. Can we do better?
 
 ## Reducing it to linear time
 
-Alright, maybe the improvement here is already obvious. Of course, we can easily
-cut the complexity by summing the actual numbers instead of ones. The new
-function then looks like this:
+Alright, maybe the improvement here is already obvious. Of course, we can easily cut the
+complexity by summing the actual numbers instead of ones. The new function then looks
+like this:
 
 ```python
 def my_sum(n: int):
@@ -74,12 +72,12 @@ def my_sum(n: int):
     return result
 ```
 
-But we need to adhere to a normal development workflow here! So instead of just
-hacking the new algorithm and pushing the changes, we should create a feature
-branch (we're calling it "linear-time") containing our improved algorithm. The
-branch is already present in the example repository that you previously checked
-out. You can create a pybm benchmark environment for it with the following
-command, run from the repository root folder on `master`:
+But we need to adhere to a normal development workflow here! So instead of just hacking
+the new algorithm and pushing the changes, we should create a feature branch (we're
+calling it "linear-time") containing our improved algorithm. The branch is already
+present in the example repository that you previously checked out. You can create a pybm
+benchmark environment for it with the following command, run from the repository root
+folder on `master`:
 
 ```shell
 pybm env create linear-time
@@ -92,40 +90,40 @@ Successfully installed packages git+https://github.com/nicholasjng/pybm into vir
 Successfully created benchmark environment for ref 'linear-time'.
 ```
 
-This checks out the HEAD of the branch "linear-time" into a separate git
-worktree located in the parent folder of the repository, and creates a fresh
-Python virtual environment for it.
+This checks out the branch "linear-time" at HEAD into a separate git worktree
+located in the parent folder of the repository, and creates a fresh Python virtual
+environment for it.
 
 But everything changes once we pick up an analysis textbook!
 
 ## The super speedy sum, after C. F. Gauss
 
-At first glance, calculating a sum of `n` numbers looks like an inherently
-linear problem. Yet, the mathematical problem contains so much hidden structure
-that we can actually do it for any number `n` on a sheet of paper. The proof is
-standard for any first-semester analysis course in university mathematics, and
-sometimes finds its way into school curricula as well.
+At first glance, calculating a sum of `n` numbers looks like an inherently linear
+problem. Yet, the mathematical problem contains so much hidden structure that we can
+actually do it for any number `n` on a sheet of paper. The proof is standard for any
+first-semester analysis course in university mathematics, and sometimes finds its way
+into school curricula as well.
 
-In Germany specifically, it floats around as a nice little anecdote from the
-early childhood of
-[Carl Friedrich Gauss](https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss),
-commonly viewed as one of the greatest mathematicians of all time, who,
-according to legend, used it to solve the summation of the first 100 numbers in
-a matter of seconds, much faster than his fellow pupils. There is a nice
-[article](https://de.wikipedia.org/wiki/Gau%C3%9Fsche_Summenformel) on German
-Wikipedia on it as well.
+In Germany specifically, it floats around as a nice little anecdote from the early
+childhood of
+[Carl Friedrich Gauss](https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss), commonly
+viewed as one of the greatest mathematicians of all time, who, according to legend, used
+it to solve his detention exercise of calculating the sum of the first 100 
+numbers in a matter of seconds, much faster than his fellow pupils. There is a nice
+[article](https://de.wikipedia.org/wiki/Gau%C3%9Fsche_Summenformel) on German Wikipedia
+on it as well.
 
-The implementation is quite literally a one-liner, and looks like this:
+The implementation is a one-liner, and looks like this:
 
 ```python
 def my_sum(n: int):
     return n * (n + 1) // 2
 ```
 
 No more loops, no `if`s, no buts: We have reduced the summation to a
-_constant time_ problem! This looks very promising. Again, this algorithm is
-already implemented on another branch called `constant-time`, for which we can
-also create a benchmark environment:
+_constant time_ problem! This looks very promising. Again, this algorithm is already
+implemented on another branch called `constant-time`, for which we can also create a
+benchmark environment:
 
 ```shell
 pybm env create constant-time
@@ -138,28 +136,22 @@ Successfully installed packages git+https://github.com/nicholasjng/pybm into vir
 Successfully created benchmark environment for ref 'constant-time'.
 ```
 
-Now we are left with a high-noon situation: Three implementation candidates,
-three different algorithms, only one can be added to our math library. But what
-are the numbers? We want to make an **informed decision** and find our best
-performer in a scientific manner. That's where a benchmark helps!
+Now we are left with a high-noon situation: Three implementation candidates, three
+different algorithms, only one can be added to our math library. But what are the
+numbers? We want to make an **informed decision** and find our best performer in a
+scientific manner. That's where a benchmark helps!
 
 ## Running the benchmark
 
-This is the perfect situation for pybm! We have environments for all of our
-algorithms (our master branch is also contained in a benchmark environment
-called "root", created during `pybm init`), so we can directly compare them. We
-do this by writing a very basic benchmark test:
+This is the perfect situation for pybm! We have environments for all of our algorithms (
+our master branch is also contained in a benchmark environment called "root", created
+during `pybm init`), so we can directly compare them. We do this by writing a very basic
+benchmark test:
 
 ```python
 # benchmarks/sum.py
-import os
-import sys
-
 import pybm
 
-SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
-sys.path.append(os.path.dirname(SCRIPT_DIR))
-
 from main import my_sum
 
 
@@ -171,20 +163,18 @@ if __name__ == "__main__":
     pybm.run(context=globals())
 ```
 
-Aside from some sys-path-hacking to get the python path set up correctly
-(which you should just ignore right now), the test file is very simple: We
-import our function `my_sum`, sum up all numbers from 1 to 10000, and run the
-benchmark when executing the module as `__main__`. Everything else is set up by
-pybm's default configuration, so we do not need to tweak more options and spend
-more time to get up and running.
+The test file is very simple: We import our function `my_sum`, sum up all numbers from 1
+to 10000, and run the benchmark when executing the module as `__main__`. Everything else
+is set up by pybm's default configuration, so we do not need to tweak more options and
+spend more time to get up and running.
 
-NOTE: The above benchmark file is the same on all three branches, and there is a
-good reason for it! When comparing the different implementations, we do need the
+NOTE: The above benchmark file is the same on all three branches, and there is a good
+reason for it! When comparing the different implementations, we do need the
 benchmarking _procedure_ itself to stay the same to yield comparable results.
 
 ```shell
 # Tells pybm to run the benchmarks in the benchmarks directory in all environments.
-pybm run benchmarks/ --all
+pybm run benchmarks --all
 
 Starting benchmarking run in environment 'root'.
 Discovering benchmark targets in environment 'root'.....done.
@@ -204,17 +194,17 @@ Finished benchmarking run in environment 'env_3'.
 Finished benchmarking in all specified environments.
 ```
 
-And there we have it! Instead of the manual rinse-and-repeat in a checkout
-branch->benchmark->save-results kind of workflow, we obtained all the results we
-need in one single command. Very nice!
+And there we have it! Instead of the manual rinse-and-repeat in a checkout branch->
+benchmark->save-results kind of workflow, we obtained all the results we need in one
+single command. Very nice!
 
 ## And finally... the numbers
 
-Lastly, we need to check how big our improvements actually are (or rather,
-if we have achieved any in the first place!). This is handled by the
-`pybm compare` command, which compares all measured results to a "frame of
-reference" branch, which is taken to be the baseline for performance
-comparisons. In our case, that is our fictitious math library's current
+Lastly, we need to check how big our improvements actually are (or rather, if we have
+achieved any in the first place!). This is handled by the
+`pybm compare` command, which compares all measured results to a "frame of reference"
+branch, which is taken to be the baseline for performance comparisons. In our case, that
+is our fictitious math library's current
 `master`.
 
 ```shell
@@ -227,13 +217,13 @@ pybm compare latest master linear-time constant-time
  benchmarks/sum.py:f | constant-time | 0.13             | 0.12            | -100.00%        | 10759575.02x | 2000000   
 ```
 
-And look here, instead of 10x-ing our previous algorithm like a normal engineer,
-we actually... 10-million-x-ed it. Great work! Our constant time algorithm is
-definitely ready for a pull request :-)
+And look here, instead of 10x-ing our previous algorithm like a normal engineer, we
+actually... 10-million-x-ed it. Great work! Our constant time algorithm is definitely
+ready for a pull request :-)
 
-These are of course video game numbers, obtained by algorithmic improvements.
-More common real-world examples would see improvements in the one-to-three digit
-percentage range, but the example you see above does happen from time to time.
+These are of course video game numbers, obtained by algorithmic improvements. More
+common real-world examples would see improvements in the one-to-three digit percentage
+range, but the example you see above does happen from time to time.
 
-And with that, the first pybm tutorial is finished. I hope you enjoyed it, and
-catch you on the next one!
+And with that, the first pybm tutorial is finished. I hope you enjoyed it, and catch you
+on the next one!

pybm/__init__.py

@@ -4,26 +4,27 @@
 # flake8: noqa: F401
 from typing import Optional, List, Any, Dict
 
-from .config import PybmConfig, get_runner_class
-from .exceptions import PybmError
-from .runners.base import BenchmarkRunner
-from .status_codes import SUCCESS
+from pybm.config import PybmConfig, get_runner_class
+from pybm.exceptions import PybmError
+from pybm.runners.base import BenchmarkRunner
+from pybm.status_codes import SUCCESS
 
 
-def run(argv: Optional[List[str]] = None,
-        context: Dict[str, Any] = None) -> int:
+def run(argv: Optional[List[str]] = None, context: Dict[str, Any] = None) -> int:
     """
     Syntactic sugar for dispatching a run inside a benchmark target file.
     The argv argument should realistically never need to be specified,
     while the context should almost always be the `globals()` object.
     """
     if context is None:
-        raise PybmError("Context is missing. When running a benchmark on a "
-                        "Python source file target, you need to pass the "
-                        "target's __main__ context in order for the "
-                        "benchmarks to be discovered correctly. You can do "
-                        "this e.g. by passing \"globals()\" as a value for "
-                        "the context object.")
+        raise PybmError(
+            "Context is missing. When running a benchmark on a "
+            "Python source file target, you need to pass the "
+            "target's __main__ context in order for the "
+            "benchmarks to be discovered correctly. You can do "
+            'this e.g. by passing "globals()" as a value for '
+            "the context object."
+        )
     config_file = PybmConfig.load(".pybm/config.yaml")
     runner: BenchmarkRunner = get_runner_class(config_file)
     runner.run_benchmark(argv, context=context)

pybm/builders/base.py

@@ -13,43 +13,47 @@ def __init__(self, config: PybmConfig):
         super().__init__()
         self.ex_type = BuilderError
         self.wheel_caches = []
-        wheel_cache_string: str = config.get_value(
-            "builder.localWheelCaches")
+
+        wheel_cache_string: str = config.get_value("builder.localWheelCaches")
         if wheel_cache_string != "":
             self.wheel_caches = wheel_cache_string.split(":")
 
     def add_arguments(self, command: str):
         raise NotImplementedError
 
-    def create(self,
-               executable: Union[str, Path],
-               destination: Union[str, Path],
-               options: Optional[List[str]] = None,
-               verbose: bool = False) -> PythonSpec:
+    def create(
+        self,
+        executable: Union[str, Path],
+        destination: Union[str, Path],
+        options: Optional[List[str]] = None,
+        verbose: bool = False,
+    ) -> PythonSpec:
         raise NotImplementedError
 
     def delete(self, env_dir: Union[str, Path], verbose: bool = False) -> None:
         raise NotImplementedError
 
-    def link(self, env_dir: Union[str, Path], verbose: bool = False) \
-            -> PythonSpec:
+    def link(self, env_dir: Union[str, Path], verbose: bool = False) -> PythonSpec:
         raise NotImplementedError
 
-    def install_packages(self,
-                         spec: PythonSpec,
-                         packages: Optional[List[str]] = None,
-                         requirements_file: Optional[str] = None,
-                         options: Optional[List[str]] = None,
-                         verbose: bool = False) -> None:
+    def install_packages(
+        self,
+        spec: PythonSpec,
+        packages: Optional[List[str]] = None,
+        requirements_file: Optional[str] = None,
+        options: Optional[List[str]] = None,
+        verbose: bool = False,
+    ) -> None:
         raise NotImplementedError
 
-    def uninstall_packages(self,
-                           spec: PythonSpec,
-                           packages: List[str],
-                           options: Optional[List[str]] = None,
-                           verbose: bool = False) -> None:
+    def uninstall_packages(
+        self,
+        spec: PythonSpec,
+        packages: List[str],
+        options: Optional[List[str]] = None,
+        verbose: bool = False,
+    ) -> None:
         raise NotImplementedError
 
-    def list_packages(self, executable: Union[str, Path],
-                      verbose: bool = False):
+    def list_packages(self, executable: Union[str, Path], verbose: bool = False):
         raise NotImplementedError