updated documentation.

Author: fresh2dev

CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+
+## v0.2.0
+
+### Added
+
+- Added `map()` function to `ezpq.Queue`.
+- Integration with tqdm package for progress bars.
+- Added `lane` property to `ezpq.Job`. When set, jobs in the same lane are processing sequentially.
+- Added `exception` property to `ezpq.Job`. Stores the exception of a failed job.
+- Added `suppress_errors` parameter to `ezpq.Job`.
+- Added `facet_by`, `theme`, and `color_pal` parameters to `ezpq.Plot.build()`.
+- Added `qid` (queue id) to job data output. This can be used to facet an `ezpq.Plot`.
+- Unit tests.
+
+### Changed
+
+- Removed parameter `n` from the `wait()` function of `ezpq.Queue`; `wait()` will halt execution until all jobs in the queue are processed.
+- Moved `ezpq.Plot()` plotting parameters to `ezpq.Plot.build()`.
+- Reordered `ezpq.Queue()` init parameters.
+
+## v0.1.0
+
+- Initial release

README.Rmd

@@ -0,0 +1,358 @@
+---
+title: '`ezpq`: an easy parallel queueing system.'
+output:
+  github_document:
+    toc: true
+    toc_depth: 3
+---
+
+```{r setup, include=FALSE}
+library(knitr)
+library(reticulate)
+use_python('~/envs/mypy36/bin/python', required = TRUE)
+knitr::opts_chunk$set(echo=FALSE, warning=FALSE)
+py_config()
+```
+
+```{python}
+import os
+import sys
+import pprint
+
+print = pprint.pprint
+
+if os.path.exists('./ezpq/__init__.py') and sys.path[0] != os.getcwd():
+    sys.path.insert(0, os.getcwd())
+import ezpq
+```
+
+Read this on [GitHub](https://github.com/dm3ll3n/ezpq) or [my site](https://www.donaldmellenbruch.com/project/ezpq/).
+
+## Overview
+
+`ezpq` implements a parallel queueing system consisting of:
+
+1. a priority "waiting" queue in.
+2. a lookup table of "working" jobs.
+3. a priority "completed" queue out.
+
+The queueing system uses `multiprocessing.Process` by default and can also run jobs with `threading.Thread`.
+
+![](docs/imgs/ezpq.png)
+
+## Features
+
+* Simple interface; pure Python.
+* No required dependencies outside of standard library.
+* Optional integration with [`tqdm`](https://github.com/tqdm/tqdm) progress bars.
+* Compatible with Python 2 & 3.
+* Cross platform with MacOS, Linux, and Windows.
+* Data remains in-memory.
+* Priority Queueing, both in and out and within lanes.
+* Synchronous lanes allow dependent jobs to execute in the desired order.
+* Easily switch from processes to threads.
+* Automatic handling of output.
+* Rich job details, easily viewed as pandas dataframe.
+* Built-in logging to CSV.
+* Customizable visualizations of queue operations.
+
+## How to get it
+
+Install from [PyPI](https://pypi.org/project/ezpq/) with:
+
+```python
+pip install ezpq
+```
+
+Optional packages:
+
+```python
+pip install pandas    # required for plots
+pip install plotnine  # required for plots
+pip install tqdm      # required for progress bars
+```
+
+##  Quickstart
+
+Suppose you wanted to speed up the following code, which runs 60 operations that take anywhere from 0s to 2s. With an average job time of ~1s, this operation should take ~60s.
+
+```{python, echo=TRUE}
+import time
+import random
+
+def random_sleep(x):
+  random.seed(x)
+  n = random.uniform(0.5, 1.5)
+  time.sleep(n)
+  return n
+```
+
+```{python, echo=TRUE}
+start = time.time()
+
+output = [random_sleep(x) for x in range(60)]
+
+end = time.time()
+
+print('> Runtime: ' + str(end - start))
+```
+
+Here is the function ran in parallel with an `ezpq` Queue of 6 workers. Thus, the runtime of the above operation will be reduced from ~60s to ~10s.
+
+```{python, echo=TRUE}
+start = time.time()
+```
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+  output = Q.map(random_sleep, range(60))
+```
+
+```{python, echo=TRUE}
+end = time.time()
+print('> Runtime: ' + str(end - start))
+```
+
+Here is the same scenario, using the `@ezpq.Queue` decorator.
+
+```{python, echo=TRUE}
+@ezpq.Queue(6)
+def random_sleep(x):
+    random.seed(x)
+    n = random.uniform(0.5, 1.5)
+    time.sleep(n)
+    return n
+
+output = random_sleep(iterable=range(60))
+```
+
+```{python}
+# redefine for future functions
+def random_sleep(x):
+  random.seed(x)
+  n = random.uniform(0.5, 1.5)
+  time.sleep(n)
+  return n
+```
+
+While `map()` and the decorator are useful for quick-n-simple parallization, the essential functions of an `ezpq` Queue include `put()`, `wait()`, and `get()` (or `collect()`).
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+    for x in range(60):
+        Q.put(random_sleep, args=x)
+    Q.wait()
+    output = Q.collect()
+```
+
+The output is a list of dicts containing verbose information about each job, along with its output, and exit code.
+
+```{python, echo=TRUE}
+print( output[0] )
+```
+
+Easily convert output to a `pandas` dataframe:
+
+```{python, echo=TRUE}
+import pandas as pd
+
+df = pd.DataFrame(output)
+
+print( df.head()[['id', 'output', 'runtime', 'exitcode']] )
+```
+
+Use `ezpq.Plot` to generate a Gannt chart of the job timings.
+
+```{python, echo=TRUE}
+plt = ezpq.Plot(output).build(show_legend=False)
+plt.save('docs/imgs/quickstart.png')
+```
+
+![](docs/imgs/quickstart.png)
+
+## `ezpq.Queue`
+
+The `Queue` class implements the queueing system, which is itself a 3-part system composed of the:
+
+1. waiting queue
+2. working table
+3. completed queue
+
+```{python}
+print(help(ezpq.Queue.__init__))
+```
+
+## `ezpq.Job`
+
+A `ezpq` job defines the function to run. It is passed to an `ezpq` queue with a call to `submit()`.
+
+```{python}
+print(help(ezpq.Job.__init__))
+```
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+  for x in range(60):
+    priority = x % 2 # give even numbers higher priority.
+    job = ezpq.Job(random_sleep, args=x, priority=priority)
+    Q.submit(job)
+  Q.wait()
+  output = Q.collect()
+```
+
+```{python}
+plt = ezpq.Plot(output).build(color_by='priority',
+                              color_pal=['blue', 'green'])
+plt.save('docs/imgs/submit.png')
+```
+
+![](docs/imgs/submit.png)
+
+
+### `put`
+
+The `put` method creates a job and submits it to an `ezpq` queue. All of its arguments are passed to `ezpq.Job()`.
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+    for x in range(60):
+        Q.put(random_sleep, args=x)
+    Q.wait()
+    output = Q.collect()
+```
+
+### `size`
+
+`size()` returns a count of all items across all three queue components. It accepts three boolean parameters, `waiting`, `working`, and `completed`. If all of these are `False` (default), all jobs are counted. If any combination of these is `True`, only the corresponding queue(s) will be counted. For example:
+
+```{python, echo=TRUE}
+def print_sizes(Q):
+    msg = 'Total: {0}; Waiting: {1}; Working: {2}; Completed: {3}'.format(
+        Q.size(),
+        Q.size(waiting=True),
+        Q.size(working=True),
+        Q.size(completed=True)
+    )
+    print(msg)
+```
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+    # enqueue jobs
+    for x in range(60):
+        Q.put(random_sleep, x)
+
+    # repeatedly print sizes until complete.
+    while Q.has_work():
+        print_sizes(Q)
+        time.sleep(1)
+
+    print_sizes(Q)
+```
+
+### `wait`
+
+The `wait()` method will block execution until all jobs complete. It also accepts a `timeout` parameter, given in seconds. The return value is the count of jobs that did not complete. Thus, a return value greater than 0 indicates the timeout was exceeded. The parameter `poll` can be used to adjust how frequently (in seconds) the operation checks for completed jobs (default=0.1).
+
+New in v0.2.0, include `show_progress=True` to show a progress bar while waiting. This is equivalent to a call to `waitpb()`.
+
+![](docs/imgs/tqdm.gif)
+
+
+### `get`
+
+`get()` retrieves and deletes ("pop") the highest priority job from the completed queue, if one is available. If the completed queue is empty, `get()` returns `None`. However, `get()` will wait for a completed job if the `poll` frequency is greater than 0. If the timeout is exceeded, `None` is returned.
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+    n_inputs = 60
+    output = [None] * n_inputs
+    # enqueue jobs
+    for x in range(n_inputs):
+        Q.put(random_sleep, args=x)
+        
+    # repeatedly `get()` queue is empty.
+    for i in range(n_inputs):
+        output[i] = Q.get(poll=0.1)
+```
+
+### `collect`
+
+`collect()` is similar to `get()`, but it will return a list of *all* completed jobs and clear the completed queue. It does not support the `poll` or `timeout` parameters, but you can call `wait()` before `collect()` if desired.
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+    # enqueue jobs
+    for x in range(60):
+        Q.put(random_sleep, x)
+
+    # wait and collect all jobs
+    print('Queue size before: {0}'.format(Q.size()))
+
+    Q.wait()
+    output = Q.collect()
+
+    print('Queue size after: {0}'.format(Q.size()))
+    print('Output size: {0}'.format(len(output)))
+```
+
+### `map`
+
+`map` encapsulates the logic of `put`, `wait`, and `collect` in one call. Include `show_progress=True` to get output `tqdm` progress bar.
+
+![](docs/imgs/tqdm_map.gif)
+
+### `dispose`
+
+The queueing operations performed by `ezpq.Queue` are performed on a periodic basis. By default, the `poll` parameter for a Queue is `0.1` seconds. This "pulse" thread will continue firing until the Queue is disposed of.
+
+In the previous examples, use of the context manager (`with ezpq.Queue() as Q:`) results in automatic disposal. If not using the context manager (or decorator), clean up after yourself with `dispose()`.
+
+```{python}
+Q = ezpq.Queue(6)
+
+Q.map(random_sleep, range(60))
+
+Q.dispose()
+```
+
+
+## Synchronous Lanes
+
+When you have jobs that are dependent upon another, you can use "lanes" to execute them in sequence. All that is required is an arbitrary lane name/id passed to the `lane` parameter of `put`. Empty lanes are automatically removed.
+
+![](docs/imgs/lanes.gif)
+
+In the above graphic, notice how same-colored bars never overlap. These bars represent jobs that are in the same lane, which executed synchronously.
+
+## `ezpq.Plot`
+
+The `Plot` class is used to visualize the wait, start, and end times for each job that entered the queueing system. The class is initialized with a list of dicts; exactly what is returned from a call to `collect()` or `map()`.
+
+Arguments given to `build()` control various aspects of the plot, from coloring, to faceting, 
+
+```{python}
+print(help(ezpq.Plot.build))
+```
+
+```{python, echo=TRUE}
+with ezpq.Queue(6) as Q:
+  for x in range(60):
+    lane = x % 5
+    Q.put(random_sleep, x, timeout=1, lane=lane)
+  Q.wait()
+  output = Q.collect()
+```
+
+```{python, echo=TRUE}
+plt = ezpq.Plot(output).build(facet_by='lane', show_legend=False)
+plt.save('docs/imgs/lanes2.png')
+```
+
+![](docs/imgs/lanes2.png)
+
+Each horizontal bar represents an independent job id. The start of the gray bar indicates when the job entered the queuing system. The start of the colored bar indicates when the job started running, and when it ended. The gray bar that follows (if any) reflects how long it took for the queue operations to recognize the finished job, join the job data with its output, remove it from the working table, and place it in the completed queue.
+
+## More Examples
+
+Many more examples can be found in [docs/examples.ipynb](//github.com/dm3ll3n/ezpq/blob/master/docs/examples.ipynb).