Merge pull request #146 from TomAugspurger/cleanup-2

Cleanup 2

Author: TomAugspurger

00_overview.ipynb

@@ -103,13 +103,12 @@
    "metadata": {},
    "source": [
     "*  Reference\n",
-    "    *  [Documentation](https://dask.pydata.org/en/latest/)\n",
+    "    *  [Documentation](https://docs.dask.org/)\n",
     "    *  [Code](https://github.com/dask/dask/)\n",
-    "    *  [Blog](http://matthewrocklin.com/blog/)\n",
+    "    *  [Blog](https://blog.dask.org)\n",
     "*  Ask for help\n",
     "    *   [dask](http://stackoverflow.com/questions/tagged/dask) tag on Stack Overflow\n",
     "    *   [github issues](https://github.com/dask/dask/issues/new) for bug reports and feature requests\n",
-    "    *   [gitter](https://gitter.im/dask/dask) for quasi-realtime conversation\n",
     "    "
    ]
   },

01_dask.delayed.ipynb

@@ -13,7 +13,15 @@
     "\n",
     "In this section we parallelize simple for-loop style code with Dask and `dask.delayed`. Often, this is the only function that you will need to convert functions for use with Dask.\n",
     "\n",
-    "This is a simple way to use `dask` to parallelize existing codebases or build [complex systems](http://matthewrocklin.com/blog/work/2018/02/09/credit-models-with-dask).  This will also help us to develop an understanding for later sections."
+    "This is a simple way to use `dask` to parallelize existing codebases or build [complex systems](https://blog.dask.org/2018/02/09/credit-models-with-dask).  This will also help us to develop an understanding for later sections.\n",
+    "\n",
+    "**Related Documentation**\n",
+    "\n",
+    "* [Delayed documentation](https://docs.dask.org/en/latest/delayed.html)\n",
+    "* [Delayed screencast](https://www.youtube.com/watch?v=SHqFmynRxVU)\n",
+    "* [Delayed API](https://docs.dask.org/en/latest/delayed-api.html)\n",
+    "* [Delayed examples](https://examples.dask.org/delayed.html)\n",
+    "* [Delayed best practices](https://docs.dask.org/en/latest/delayed-best-practices.html)"
    ]
   },
   {
@@ -31,7 +39,7 @@
    "source": [
     "from dask.distributed import Client\n",
     "\n",
-    "client = Client()"
+    "client = Client(n_workers=4)"
    ]
   },
   {
@@ -698,7 +706,11 @@
     "- How much speedup did you get? Is this how much speedup you'd expect?\n",
     "- Experiment with where to call `compute`. What happens when you call it on `sums` and `counts`? What happens if you wait and call it on `mean`?\n",
     "- Experiment with delaying the call to `sum`. What does the graph look like if `sum` is delayed? What does the graph look like if it isn't?\n",
-    "- Can you think of any reason why you'd want to do the reduction one way over the other?"
+    "- Can you think of any reason why you'd want to do the reduction one way over the other?\n",
+    "\n",
+    "### Learn More\n",
+    "\n",
+    "Visit the [Delayed documentation](https://docs.dask.org/en/latest/delayed.html). In particular, this [delayed screencast](https://www.youtube.com/watch?v=SHqFmynRxVU) will reinforce the concepts you learned here and the [delayed best practices](https://docs.dask.org/en/latest/delayed-best-practices.html) document collects advice on using `dask.delayed` well."
    ]
   },
   {

02_bag.ipynb

@@ -31,8 +31,10 @@
     "    \n",
     "**Related Documentation**\n",
     "\n",
-    "*  [Bag Documenation](http://dask.pydata.org/en/latest/bag.html)\n",
-    "*  [Bag API](http://dask.pydata.org/en/latest/bag-api.html)"
+    "* [Bag documentation](https://docs.dask.org/en/latest/bag.html)\n",
+    "* [Bag screencast](https://youtu.be/-qIiJ1XtSv0)\n",
+    "* [Bag API](https://docs.dask.org/en/latest/bag-api.html)\n",
+    "* [Bag examples](https://examples.dask.org/bag.html)"
    ]
   },
   {
@@ -50,7 +52,7 @@
    "source": [
     "from dask.distributed import Client\n",
     "\n",
-    "client = Client()"
+    "client = Client(n_workers=4)"
    ]
   },
   {
@@ -626,6 +628,18 @@
     "    a normalised dataframe."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Learn More\n",
+    "\n",
+    "* [Bag documentation](https://docs.dask.org/en/latest/bag.html)\n",
+    "* [Bag screencast](https://youtu.be/-qIiJ1XtSv0)\n",
+    "* [Bag API](https://docs.dask.org/en/latest/bag-api.html)\n",
+    "* [Bag examples](https://examples.dask.org/bag.html)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

03_array.ipynb

@@ -25,10 +25,15 @@
     "*  **Larger-than-memory**:  Lets you work on datasets that are larger than your available memory by breaking up your array into many small pieces, operating on those pieces in an order that minimizes the memory footprint of your computation, and effectively streaming data from disk.\n",
     "*  **Blocked Algorithms**:  Perform large computations by performing many smaller computations\n",
     "\n",
+    "In this notebook, we'll build some understanding by implementing some blocked algorithms from scratch.\n",
+    "We'll then use Dask Array to analyze large datasets, in parallel, using a familiar NumPy-like API.\n",
+    "\n",
     "**Related Documentation**\n",
     "\n",
-    "* [Documentation](http://dask.readthedocs.io/en/latest/array.html)\n",
-    "* [API reference](http://dask.readthedocs.io/en/latest/array-api.html)"
+    "* [Array documentation](https://docs.dask.org/en/latest/array.html)\n",
+    "* [Array screencast](https://youtu.be/9h_61hXCDuI)\n",
+    "* [Array API](https://docs.dask.org/en/latest/array-api.html)\n",
+    "* [Array examples](https://examples.dask.org/array.html)"
    ]
   },
   {
@@ -39,7 +44,7 @@
    "source": [
     "from dask.distributed import Client\n",
     "\n",
-    "client = Client(processes=False)"
+    "client = Client(n_workers=4, processes=False)"
    ]
   },
   {
@@ -298,7 +303,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And the variance, std, etc..  This should be a trivial change to the example above.\n",
+    "And the variance, std, etc..  This should be a small change to the example above.\n",
     "\n",
     "Look at what other operations you can do with the Jupyter notebook's tab-completion."
    ]
@@ -639,11 +644,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "jupyter": {
-     "source_hidden": true
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "result = x.mean(axis=0)\n",
@@ -668,11 +669,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "jupyter": {
-     "source_hidden": true
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "result = x[0] - x.mean(axis=0)\n",
@@ -732,11 +729,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "jupyter": {
-     "source_hidden": true
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import h5py\n",
@@ -919,24 +912,29 @@
     "Limitations\n",
     "-----------\n",
     "\n",
-    "Dask.array does not implement the entire numpy interface.  Users expecting this\n",
-    "will be disappointed.  Notably dask.array has the following failings:\n",
+    "Dask Array does not implement the entire numpy interface.  Users expecting this\n",
+    "will be disappointed.  Notably Dask Array has the following failings:\n",
     "\n",
     "1.  Dask does not implement all of ``np.linalg``.  This has been done by a\n",
     "    number of excellent BLAS/LAPACK implementations and is the focus of\n",
     "    numerous ongoing academic research projects.\n",
-    "2.  Dask.array does not support any operation where the resulting shape\n",
-    "    depends on the values of the array.  In order to form the Dask graph we\n",
-    "    must be able to infer the shape of the array before actually executing the\n",
-    "    operation.  This precludes operations like indexing one Dask array with\n",
-    "    another or operations like ``np.where``.\n",
-    "3.  Dask.array does not attempt operations like ``sort`` which are notoriously\n",
+    "2.  Dask Array does not support some operations where the resulting shape\n",
+    "    depends on the values of the array. For those that it does support\n",
+    "    (for example, masking one Dask Array with another boolean mask),\n",
+    "    the chunk sizes will be unknown, which may cause issues with other\n",
+    "    operations that need to know the chunk sizes.\n",
+    "3.  Dask Array does not attempt operations like ``sort`` which are notoriously\n",
     "    difficult to do in parallel and are of somewhat diminished value on very\n",
     "    large data (you rarely actually need a full sort).\n",
     "    Often we include parallel-friendly alternatives like ``topk``.\n",
     "4.  Dask development is driven by immediate need, and so many lesser used\n",
-    "    functions, like ``np.full_like`` have not been implemented purely out of\n",
-    "    laziness.  These would make excellent community contributions."
+    "    functions, like ``np.sometrue`` have not been implemented purely out of\n",
+    "    laziness.  These would make excellent community contributions.\n",
+    "    \n",
+    "* [Array documentation](https://docs.dask.org/en/latest/array.html)\n",
+    "* [Array screencast](https://youtu.be/9h_61hXCDuI)\n",
+    "* [Array API](https://docs.dask.org/en/latest/array-api.html)\n",
+    "* [Array examples](https://examples.dask.org/array.html)"
    ]
   },
   {