ENH: Redesign of the Extension [Major Change]

.gitignore

@@ -12,6 +12,7 @@ tests/pdf/_build/
 tests/pdf/_static/
 tests/pdf/theme/
 tests/no_inline_exercises/_build
+tests/execute/_build/
 tests/ipynb/*checkpoints
 *.pyc
 .DS_Store

.travis.yml

@@ -3,18 +3,12 @@
 sudo: false        #Use new Container Infrastructure
 language: python
 python:
-  - 2.7
   - 3.7
 
 env:
-  - SPHINX_VERSION=1.7.9
   - SPHINX_VERSION=1.8.5
   - SPHINX_VERSION=2.1.2
-
-matrix:
-  exclude:
-  - python: 2.7
-    env: SPHINX_VERSION=2.1.2
+  - SPHINX_VERSION=2.4.1
 
 directories:
   - "/tmp/texlive"
@@ -33,15 +27,13 @@ before_install:
   - travis_wait 45 bash ./util/setup_latex.sh
   - export PATH="/tmp/texlive/bin/x86_64-linux:$PATH"
 
+addons:
+  apt_packages:
+    - pandoc
 
 install:
-  - if [ $TRAVIS_PYTHON_VERSION == 2.7 ]; then
-      pip install sphinx==$SPHINX_VERSION nbconvert nbformat dask distributed nbdime;
-      pip install sphinxcontrib-bibtex==0.4.2;
-    else
-      pip install sphinx==$SPHINX_VERSION nbconvert nbformat dask==2.5.2 distributed==2.5.2 nbdime;
-      pip install sphinxcontrib-bibtex;
-    fi
+  - pip install sphinx==$SPHINX_VERSION nbconvert nbformat dask==2.5.2 distributed==2.5.2 nbdime;
+  - pip install sphinxcontrib-bibtex;
   - python setup.py install
 
 script:

docs/builders.rst

@@ -3,36 +3,38 @@
 Builders
 =========
 
-This extension has the following Builders
+.. contents:: Options
+    :depth: 1
+    :local:
 
-jupyter
+execute
 -------
 
-This builder currently handles `jupyter`, `html`, and `coverage` output
-
-.. code-block:: bash
-
-    @$(SPHINXBUILD) -M jupyter "$(SOURCEDIR)" "$(BUILDCOVERAGE)" $(FILES) $(SPHINXOPTS) $(O)
+This builder extracts code-blocks from the `RST` files and manages the 
+execution of each block. It saves the results in a `json` type codetree
+object for use by other builders that construct notebooks.
 
+jupyter
+-------
 
-.. warning::
+This builder build IPYNB notebooks.
 
-    If your project needs to build `jupyter` and `html` then configuration for `html` 
-    and `coverage` is currently handled through Makefile overrides. 
-    The project is working on separate builders for html and coverage
+if `jupyter_execute_notebooks=True` is set in the `conf.py` file then 
+the `execute` builder is automatically called to run all code.
 
-Example Configuration for `HTML` production
 
-.. code-block:: bash
+jupyterhtml 
+-----------
 
-    @$(SPHINXBUILD) -M jupyter "$(SOURCEDIR)" "$(BUILDWEBSITE)" $(FILES) $(SPHINXOPTS) $(O) -D jupyter_make_site=1 -D jupyter_generate_html=1 -D jupyter_download_nb=1 -D jupyter_execute_notebooks=1 -D jupyter_target_html=1 -D jupyter_download_nb_image_urlpath="https://s3-ap-southeast-2.amazonaws.com/lectures.quantecon.org/py/_static/" -D jupyter_images_markdown=0 -D jupyter_html_template="python-html.tpl" -D jupyter_download_nb_urlpath="https://lectures.quantecon.org/" -D jupyter_coverage_dir=$(BUILDCOVERAGE)
+This builder manages the construction of project websites
 
+if `jupyter_execute_notebooks=True` is set in the `conf.py` file then 
+the `execute` builder is automatically called to run all code.
 
 jupyterpdf
 ----------
 
-This builder handles production of `pdf`
-
-.. code-block:: bash
+This builder handles construction of `pdf` versions of the RST documents.
 
-    @$(SPHINXBUILD) -M jupyterpdf "$(SOURCEDIR)" "$(BUILDCOVERAGE)" $(FILES) $(SPHINXOPTS) $(O)
+if `jupyter_execute_notebooks=True` is set in the `conf.py` file then 
+the `execute` builder is automatically called to run all code.

docs/config-execution.rst

@@ -0,0 +1,182 @@
+.. _config_execution:
+
+Execution Configuration
+=======================
+
+.. contents:: Options
+    :depth: 1
+    :local:
+
+Code execution is handled by a specialised `execute` builder.
+
+jupyter_file_dependencies
+--------------------------
+
+Specify support (dependencies) for notebook collection at the `file` or 
+the `directory` level.
+
+``conf.py`` usage:
+
+.. code-block:: python
+
+   jupyter_dependencies = {
+       <dir> : ['file1', 'file2'],
+       {<dir>}/<file.rst> : ['file1']
+   }
+
+.. note::
+
+    to specify a support file at the root level of the source directory
+    the key should be `""`
+
+jupyter_notebook_dependencies
+-----------------------------
+
+Dependency of notebooks on other notebooks for execution can 
+be added to the configuration file above in the form of a dictionary. 
+The key/value pairs will contain the names of the notebook files.
+
+``conf.py`` usage:
+
+.. code-block:: python
+
+   # add your dependency lists here
+   jupyter_notebook_dependencies = {
+      'python_advanced_features' : ['python_essentials','python_oop'],
+      'discrete_dp' : ['dp_essentials'],
+   }
+
+jupyter_number_workers
+-------------------------
+
+Specify the number cores to use with dask
+
+.. list-table:: 
+   :header-rows: 1
+
+   * - Values
+   * - Integer (**default** = 1)
+
+``conf.py`` usage:
+
+    jupyter_number_workers = 4
+
+
+jupyter_execution_threads_per_worker
+------------------------------------
+
+Specify the number of threads per worker for dask
+
+.. list-table:: 
+   :header-rows: 1
+
+   * - Values
+   * - Integer (**default** = 1)
+
+``conf.py`` usage:
+
+    jupyter_threads_per_worker = 1
+
+jupyter_coverage_report_template
+-----------------------------------
+
+Provide path to template coverage report file
+
+.. todo::
+
+    Document format for template
+
+``conf.py`` usage:
+
+.. code-block:: python
+
+    jupyter_template_coverage_file_path = "theme/templates/<file>.json"
+
+.. warning::
+
+    ``coverage`` will currently produce a ``json`` report
+    that can be used to support an execution status page. But adding
+    badges and a status page needs to be made into an option and 
+    specified via the default theme. 
+    See `#237 <https://github.com/QuantEcon/sphinxcontrib-jupyter/issues/237>`__
+
+
+jupyter_options
+---------------
+
+An dict-type object that is used by dask to control execution
+
+
+.. TODO:: 
+
+    @aakash is this used?
+
+jupyter_drop_tests
+------------------
+
+**Note:** Future Feature?
+
+Drop ``code-blocks` that include ``:class: test``
+
+.. list-table:: 
+   :header-rows: 1
+
+   * - Values
+   * - False (**default**)
+   * - True 
+
+.. TODO::
+
+    This option should be added to `execute` builder so tests can 
+    be switched on/off for execution testing
+
+jupyter_ignore_no_execute:
+--------------------------
+
+**Note:** Deprecate?
+
+.. list-table:: 
+   :header-rows: 1
+
+   * - Values
+   * - False (**default**)
+   * - True 
+
+When constructing notebooks this option can be enabled to ignore `:class: no-execute`
+for `code-blocks`. This is useful for `html` writer for pages that are meant to fail 
+but shouldn't be included in `coverage` tests. 
+
+``conf.py`` usage:
+
+.. code-block:: python
+
+    jupyter_ignore_no_execute = True
+
+.. todo::
+
+    @aakash this is no longer required as all code-blocks are run correct?
+
+jupyter_ignore_skip_test
+------------------------
+
+**Note:** Deprecate?
+
+When constructing notebooks this option can be enabled to ignore `:class: skip-test`
+for `code-blocks`.
+
+.. list-table:: 
+   :header-rows: 1
+
+   * - Values
+   * - False (**default**)
+   * - True
+
+``conf.py`` usage:
+
+.. code-block:: python
+
+    jupyter_ignore_skip_test = True
+
+.. todo::
+
+    @aakash is this required?