Remove opinions and introduce a test loader. (ipython#39)

* Refactor the compile notebook and expose an export function to format our development code.
* Add docstring information to the readme
* Add a docstring test.
* Explicit names for the compiler and exporter classes giving a nicer graph structure
* End the markdown quotes on a new lines
* Explicit class names for the parameterize functions.
* Add an ability to load python files as modules with from_filename
* Format source code.
* Doctest a module in the tests.
* Remove the tox requirements and use pytest syntax for the tests.
* Add more tests
* Skip the docstring test in 3.7 until the docstring is fixed
* modify how the changelog is read and remove the markdown version
    * Include the changelog.ipynb in the manifest
* Make Notebook classes callable and make from_filename an regular method.
* Use the default IPython transformer
* Update the benchmark with the new from file code

Author: tonyfast

MANIFEST.in

@@ -1,2 +1,2 @@
-include LICENSE readme.md changelog.md
+include LICENSE readme.md changelog.ipynb src/notebooks/block_string.tpl
 recursive-include src/importnb *.ipynb

benchmarks/benchmark.ipynb

@@ -114,7 +114,7 @@
     "if repo and __name__ != '__main__':\n",
     "    modules = {}\n",
     "    for path in notebooks.glob('*.ipynb'):\n",
-    "        modules[path] = module = Timed.from_filename(path, stdout=True, stderr=True, display=display)\n",
+    "        modules[path] = module = Timed(stdout=True, stderr=True, display=display).from_filename(path)\n",
     "        print(f\"{'🏆😩'[bool(module.__exception__)]} {path}\")\n",
     "        "
    ]

benchmarks/readme.ipynb

@@ -55,15 +55,15 @@ Benchmarks for `importnb` explore the ability to import existing notebooks as mo
     🏆 ThinkDSP/code/chap09soln.ipynb
     😩 ThinkDSP/code/chap10.ipynb
     🏆 ThinkDSP/code/chap10preview.ipynb
-    🏆 ThinkDSP/code/chap10soln.ipynb
+    😩 ThinkDSP/code/chap10soln.ipynb
     😩 ThinkDSP/code/chap11.ipynb
     😩 ThinkDSP/code/chap11soln.ipynb
     😩 ThinkDSP/code/diff_int.ipynb
     🏆 ThinkDSP/code/phase.ipynb
     🏆 ThinkDSP/code/saxophone.ipynb
     🏆 ThinkDSP/code/scipy2015_demo.ipynb
     😩 ThinkDSP/code/voss.ipynb
-    70.0% of the ThinkDSP notebooks loaded with a import threshold of 10.000000 seconds
+    66.7% of the ThinkDSP notebooks loaded with a import threshold of 10.000000 seconds
     {'repo': 'https://www.github.com/jakevdp/PythonDataScienceHandbook', 'notebooks': 'PythonDataScienceHandbook/notebooks/'}
     ==========
     🏆 PythonDataScienceHandbook/notebooks/00.00-Preface.ipynb
@@ -257,7 +257,7 @@ Benchmarks for `importnb` explore the ability to import existing notebooks as mo
 
 
     😩 PythonDataScienceHandbook/notebooks/05.07-Support-Vector-Machines.ipynb
-    🏆 PythonDataScienceHandbook/notebooks/05.08-Random-Forests.ipynb
+    😩 PythonDataScienceHandbook/notebooks/05.08-Random-Forests.ipynb
     🏆 PythonDataScienceHandbook/notebooks/05.09-Principal-Component-Analysis.ipynb
     😩 PythonDataScienceHandbook/notebooks/05.10-Manifold-Learning.ipynb
     😩 PythonDataScienceHandbook/notebooks/05.11-K-Means.ipynb
@@ -267,19 +267,14 @@ Benchmarks for `importnb` explore the ability to import existing notebooks as mo
     🏆 PythonDataScienceHandbook/notebooks/05.15-Learning-More.ipynb
     🏆 PythonDataScienceHandbook/notebooks/06.00-Figure-Code.ipynb
     🏆 PythonDataScienceHandbook/notebooks/Index.ipynb
-    60.3% of the PythonDataScienceHandbook notebooks loaded with a import threshold of 10.000000 seconds
+    58.8% of the PythonDataScienceHandbook notebooks loaded with a import threshold of 10.000000 seconds
 
 
 
 ![png](readme_files/readme_4_3.png)
 
 
 
-```python
-    
-```
-
-
 ```python
     !jupyter nbconvert --to markdown readme.ipynb
 ```

benchmarks/readme.md

@@ -7,6 +7,24 @@
     "## CHANGELOG"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 0.3.0\n",
+    "\n",
+    "* Include `Partial`, `Lazy`, and `NotebookTest` loaders.\n",
+    "* Transform markdown cells to literate block strings so they are included in the ast.\n",
+    "    * `__doc__`'s are extracted from the first markdown cell or normal source code from a code cell.\n",
+    "* Export the python source code with `black`.\n",
+    "* `Notebook.from_filename` is a loader for paths and strings.\n",
+    "* Add `importnb.nbtest` for notebook testing tools..\n",
+    "* Benchmark `importnb` against existing notebooks.\n",
+    "* Include a `watchdog` trick to watch tests..\n",
+    "* Extend the project to >= 3.4\n",
+    "* Use nbviewer/github hierachy for the docs."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

changelog.ipynb

@@ -12,7 +12,11 @@
     "\n",
     "\n",
     "\n",
-    "    pip install importnb"
+    "    pip install importnb\n",
+    "    \n",
+    "---\n",
+    "\n",
+    "    conda install -c conda-forge importnb"
    ]
   },
   {
@@ -30,7 +34,7 @@
    "source": [
     "### For brevity\n",
     "\n",
-    "[`importnb.loader`](src/notebooks/loader.ipynb) will find notebooks avaiable anywhere along the [`sys.path`](https://docs.python.org/2/library/sys.html#sys.path)."
+    "[`importnb.loader`](src/notebooks/loader.ipynb) will find notebooks available anywhere along the [`sys.path`](https://docs.python.org/2/library/sys.html#sys.path)."
    ]
   },
   {
@@ -151,6 +155,33 @@
     "        import readme"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Docstring\n",
+    "\n",
+    "The first cell is the module docstring."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "__importnb__ imports notebooks as modules & packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "    if __name__ == '__main__':\n",
+    "        print(readme.__doc__.splitlines()[0])"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -179,7 +210,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -197,7 +228,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
@@ -206,7 +237,7 @@
        "<Signature (*, foo=42)>"
       ]
      },
-     "execution_count": 9,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -333,45 +364,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "src/notebooks/capture.ipynb\n",
-      "src/notebooks/compiler_ipython.ipynb\n",
-      "src/notebooks/compiler_python.ipynb\n",
-      "src/notebooks/decoder.ipynb\n",
-      "src/notebooks/exporter.ipynb\n",
-      "src/notebooks/loader.ipynb\n",
-      "src/notebooks/parameterize.ipynb\n",
-      "src/notebooks/utils/__init__.ipynb\n",
-      "src/notebooks/utils/ipython.ipynb\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "test_import (src.importnb.tests.test_unittests.TestContext) ... "
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "src/notebooks/utils/pytest_plugin.ipynb\n",
-      "src/notebooks/utils/setup.ipynb\n",
-      "src/notebooks/utils/watch.ipynb\n"
-     ]
-    },
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "ok\n",
+      "test_import (src.importnb.tests.test_unittests.TestContext) ... ok\n",
       "test_reload_with_context (src.importnb.tests.test_unittests.TestContext) ... ok\n",
       "test_failure (src.importnb.tests.test_unittests.TestExtension) ... expected failure\n",
       "test_import (src.importnb.tests.test_unittests.TestExtension) ... ok\n",
@@ -380,7 +380,7 @@
       "test_imports (src.importnb.tests.test_unittests.TestRemote) ... skipped 'requires IP'\n",
       "\n",
       "----------------------------------------------------------------------\n",
-      "Ran 7 tests in 2.023s\n",
+      "Ran 7 tests in 2.021s\n",
       "\n",
       "OK (skipped=1, expected failures=1)\n"
      ]
@@ -389,38 +389,50 @@
    "source": [
     "    if __name__ == '__main__':\n",
     "        from pathlib import Path\n",
-    "        from importnb.compiler_python import ScriptExporter\n",
+    "        from nbconvert.exporters.python import PythonExporter\n",
+    "        from importnb.compile import export\n",
     "        for path in Path('src/notebooks/').rglob(\"\"\"*.ipynb\"\"\"):                \n",
     "            if 'checkpoint' not in str(path):\n",
-    "                src = ScriptExporter().from_filename(path)[0]\n",
-    "                try:\n",
-    "                    import black\n",
-    "                    src = black.format_str(src, 100)\n",
-    "                except: ...\n",
-    "\n",
-    "                print(path)\n",
-    "                (Path('src/importnb') / path.with_suffix('.py').relative_to('src/notebooks')).write_text(src)\n",
+    "                export(path, Path('src/importnb') / path.with_suffix('.py').relative_to('src/notebooks'))\n",
     "            \n",
     "        __import__('unittest').main(module='src.importnb.tests.test_unittests', argv=\"discover --verbose\".split(), exit=False) \n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "[NbConvertApp] Converting notebook readme.ipynb to markdown\n",
-      "[NbConvertApp] Writing 7132 bytes to readme.md\n"
+      "[NbConvertApp] Writing 7121 bytes to readme.md\n"
      ]
     }
    ],
    "source": [
-    "    !jupyter nbconvert --to markdown readme.ipynb"
+    "    if __name__ == '__main__':\n",
+    "        !jupyter nbconvert --to markdown readme.ipynb"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "    if __name__ == '__main__':\n",
+    "        from IPython.display import display, Image\n",
+    "        !pyreverse importnb -opng -pimportnb\n",
+    "        display(*map(Image, ('classes_importnb.png', 'packages_importnb.png')))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {