First pass at reading over notebooks (#253)

Author: jsignell

01_dataframe.ipynb

@@ -12,17 +12,11 @@
     "     alt=\"Dask logo\\\">\n",
     "\n",
     "\n",
-    "# Dask DataFrame API\n",
+    "# Dask DataFrame - parallelized pandas\n",
     "\n",
-    "`dask.dataframe` API is the high-level collection that looks and feels like the pandas API, but for parallel and distributed workflows. At it's core, the `dask.dataframe` module implements a \"blocked parallel\" `DataFrame` object that. One Dask `DataFrame` is comprised of many in-memory pandas `DataFrame`s separated along the index. One operation on a Dask `DataFrame` triggers many pandas operations on the constituent pandas `DataFrame`s in a way that is mindful of potential parallelism and memory constraints.\n",
+    "Looks and feels like the pandas API, but for parallel and distributed workflows. \n",
     "\n",
-    "<img src=\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\"\n",
-    "     width=\"30%\"\n",
-    "     alt=\"Dask DataFrame is compsed of pandas DataFrames\"/>\n",
-    "\n",
-    "The term \"Dask DataFrame\" is slightly overloaded. Depending on the context, it can refer to the API or the DataFrame object (data structure composed on multiple pandas DataFrames shown above). To avoid confusion, throughout this notebook:\n",
-    "- `dask.dataframe` (note the all lowercase) refers to the API, and\n",
-    "- `dask.dataframe.DataFrame` or simply `DataFrame` (note the CamelCase) refers to the object."
+    "At its core, the `dask.dataframe` module implements a \"blocked parallel\" `DataFrame` object that. One Dask `DataFrame` is comprised of many in-memory pandas `DataFrame`s separated along the index. One operation on a Dask `DataFrame` triggers many pandas operations on the constituent pandas `DataFrame`s in a way that is mindful of potential parallelism and memory constraints.\n"
    ]
   },
   {
@@ -31,6 +25,19 @@
     "tags": []
    },
    "source": [
+    "<img src=\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\"\n",
+    "     align=\"right\"\n",
+    "     width=\"30%\"\n",
+    "     alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
+    "\n",
+    "**Related Documentation**\n",
+    "\n",
+    "* [DataFrame documentation](https://docs.dask.org/en/latest/dataframe.html)\n",
+    "* [DataFrame screencast](https://youtu.be/AT2XtFehFSQ)\n",
+    "* [DataFrame API](https://docs.dask.org/en/latest/dataframe-api.html)\n",
+    "* [DataFrame examples](https://examples.dask.org/dataframe.html)\n",
+    "* [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)\n",
+    "\n",
     "## When to use `dask.dataframe`\n",
     "\n",
     "pandas is great for tabular datasets that fit in memory. A general rule of thumb for pandas is:\n",
@@ -43,20 +50,8 @@
     "\n",
     "Dask becomes useful when the datasets exceed the above rule.\n",
     "\n",
-    "In this notebook, you will be working with the New York City Airline data. This dataset is only ~200MB, so that you can download it in a reasonable time, but `dask.dataframe` will scale to  datasets **much** larger than memory."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "**Related Documentation**\n",
-    "\n",
-    "* [DataFrame documentation](https://docs.dask.org/en/latest/dataframe.html)\n",
-    "* [DataFrame screencast](https://youtu.be/AT2XtFehFSQ)\n",
-    "* [DataFrame API](https://docs.dask.org/en/latest/dataframe-api.html)\n",
-    "* [DataFrame examples](https://examples.dask.org/dataframe.html)\n",
-    "* [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)"
+    "In this notebook, you will be working with the New York City Airline data. This dataset is only ~200MB, so that you can download it in a reasonable time, but `dask.dataframe` will scale to  datasets **much** larger than memory.\n",
+    "\n"
    ]
   },
   {
@@ -136,13 +131,6 @@
     "Let's read an extract of flights in the USA across several years. This data is specific to flights out of the three airports in the New York City area."
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The following filename includes a glob pattern `*`, so all files in the path matching that pattern will be read into the same Dask `DataFrame`."
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -157,16 +145,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "By convention, we import `dask.dataframe` as `dd`, and call the corresponding Dask DataFrame `ddf`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import dask.dataframe as dd"
+    "By convention, we import the module `dask.dataframe` as `dd`, and call the corresponding `DataFrame` object `ddf`.\n",
+    "\n",
+    "**Note**: The term \"Dask DataFrame\" is slightly overloaded. Depending on the context, it can refer to the module or the DataFrame object. To avoid confusion, throughout this notebook:\n",
+    "- `dask.dataframe` (note the all lowercase) refers to the API, and\n",
+    "- `DataFrame` (note the CamelCase) refers to the object.\n",
+    "\n",
+    "The following filename includes a glob pattern `*`, so all files in the path matching that pattern will be read into the same `DataFrame`."
    ]
   },
   {
@@ -175,16 +160,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import dask.dataframe as dd\n",
+    "\n",
     "ddf = dd.read_csv(os.path.join('data', 'nycflights', '*.csv'),\n",
-    "                  parse_dates={'Date': [0, 1, 2]})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "                  parse_dates={'Date': [0, 1, 2]})\n",
     "ddf"
    ]
   },
@@ -201,7 +180,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Notice that the representation of the DataFrame object contains no data - Dask has just done enough to read the start of the first file, and infer the column names and dtypes."
+    "Notice that the representation of the `DataFrame` object contains no data - Dask has just done enough to read the start of the first file, and infer the column names and dtypes."
    ]
   },
   {
@@ -360,7 +339,7 @@
    "source": [
     "## Computations with `dask.dataframe`\n",
     "\n",
-    "Let's compute the maximum of the `DepDelay` column.\n",
+    "Let's compute the maximum of the flight delay.\n",
     "\n",
     "With just pandas, we would loop over each file to find the individual maximums, then find the final maximum over all the individual maximums.\n",
     "\n",
@@ -388,7 +367,8 @@
    "outputs": [],
    "source": [
     "%%time\n",
-    "ddf.DepDelay.max().compute()"
+    "result = ddf.DepDelay.max()\n",
+    "result.compute()"
    ]
   },
   {
@@ -409,7 +389,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can view the underlying task graph using the `.visualize` method:"
+    "You can view the underlying task graph using `.visualize()`:"
    ]
   },
   {
@@ -419,7 +399,7 @@
    "outputs": [],
    "source": [
     "# notice the parallelism\n",
-    "ddf.DepDelay.max().visualize()"
+    "result.visualize()"
    ]
   },
   {
@@ -428,7 +408,7 @@
    "source": [
     "## Exercises\n",
     "\n",
-    "In this section you will do a few `dask.dataframe` computations. If you are comfortable with pandas then these should be familiar. You will have to think about when to call `compute`."
+    "In this section you will do a few `dask.dataframe` computations. If you are comfortable with pandas then these should be familiar. You will have to think about when to call `.compute()`."
    ]
   },
   {
@@ -437,7 +417,7 @@
    "source": [
     "### 1. How many rows are in our dataset?\n",
     "\n",
-    "If you aren't familiar with pandas, how would you check how many records are in a list of tuples?"
+    "_Hint_: how would you check how many items are in a list?"
    ]
   },
   {
@@ -466,7 +446,7 @@
    "source": [
     "### 2. In total, how many non-canceled flights were taken?\n",
     "\n",
-    "With pandas, you would use [boolean indexing](https://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing)."
+    "_Hint_: use [boolean indexing](https://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing)."
    ]
   },
   {
@@ -493,9 +473,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### 3. In total, how many non-cancelled flights were taken from each airport?\n",
+    "### 3. In total, how many non-canceled flights were taken from each airport?\n",
     "\n",
-    "*Hint*: use [`groupby`](https://pandas.pydata.org/pandas-docs/stable/groupby.html)."
+    "*Hint*: use [groupby](https://pandas.pydata.org/pandas-docs/stable/groupby.html)."
    ]
   },
   {
@@ -596,7 +576,11 @@
    },
    "outputs": [],
    "source": [
-    "ddf[\"Distance\"].apply(lambda x: x+1).compute() # don't worry about the warning, we'll discuss in the next sections"
+    "ddf[\"Distance\"].apply(lambda x: x+1).compute() # don't worry about the warning, we'll discuss in the next sections\n",
+    "\n",
+    "# OR\n",
+    "\n",
+    "(ddf[\"Distance\"] + 1).compute()"
    ]
   },
   {
@@ -605,9 +589,9 @@
    "source": [
     "## Sharing Intermediate Results\n",
     "\n",
-    "When computing all of the above, we sometimes did the same operation more than once. For most operations, `dask.dataframe` hashes the arguments, allowing duplicate computations to be shared, and only computed once.\n",
+    "When computing all of the above, we sometimes did the same operation more than once. For most operations, `dask.dataframe` stores the arguments, allowing duplicate computations to be shared and only computed once.\n",
     "\n",
-    "For example, lets compute the mean and standard deviation for departure delay of all non-canceled flights. Since Dask operations are lazy, those values aren't the final results yet. They're just the recipe required to get the result.\n",
+    "For example, let's compute the mean and standard deviation for departure delay of all non-canceled flights. Since Dask operations are lazy, those values aren't the final results yet. They're just the steps required to get the result.\n",
     "\n",
     "If you compute them with two calls to compute, there is no sharing of intermediate computations."
    ]
@@ -697,11 +681,11 @@
     "tags": []
    },
    "source": [
-    "### `persit`\n",
+    "### `.persist()`\n",
     "\n",
-    "While using a distributed scheduler (you will learn more about schedulers in the upcoming notebooks), which is the case in this notebook, you can keep some _data that you want to use often_ in the _distributed memory_. \n",
+    "While using a distributed scheduler (you will learn more about schedulers in the upcoming notebooks), you can keep some _data that you want to use often_ in the _distributed memory_. \n",
     "\n",
-    "`persist` computes the \"Futures\" ( and stores it in the same structure as your output. You can use persist with any data or computation that fits in memory."
+    "`persist` generates \"Futures\" (more on this later as well) and stores them in the same structure as your output. You can use `persist` with any data or computation that fits in memory."
    ]
   },
   {
@@ -783,14 +767,14 @@
     "\n",
     "Additionally, some important operations like `set_index` work, but are slower than in pandas because they include substantial shuffling of data, and may write out to disk.\n",
     "\n",
-    "**So, what if you want to use some custom functions that aren't (or can't be) implemented for Dask DataFrame yet?**\n",
+    "**What if you want to use some custom functions that aren't (or can't be) implemented for Dask DataFrame yet?**\n",
     "\n",
     "You can open an issue on the [Dask issue tracker](https://github.com/dask/dask/issues) to check how feasible the function could be to implement, and you can consider contributing this function to Dask.\n",
     "\n",
     "If it's a custom function or tricky to implement, `dask.dataframe` provides a few methods to make applying custom functions to Dask DataFrames easier:\n",
     "\n",
-    "- [`map_partitions`](https://docs.dask.org/en/latest/generated/dask.dataframe.DataFrame.map_partitions.html): to use a function on each partition (each pandas DataFrame) of the Dask DataFrame\n",
-    "- [`map_overlap`](https://docs.dask.org/en/latest/generated/dask.dataframe.rolling.map_overlap.html): to use a function on each partition (each pandas DataFrame) of the Dask DataFrame, with some overlap between partitions\n",
+    "- [`map_partitions`](https://docs.dask.org/en/latest/generated/dask.dataframe.DataFrame.map_partitions.html): to run a function on each partition (each pandas DataFrame) of the Dask DataFrame\n",
+    "- [`map_overlap`](https://docs.dask.org/en/latest/generated/dask.dataframe.rolling.map_overlap.html): to run a function on each partition (each pandas DataFrame) of the Dask DataFrame, with some rows shared between neighboring partitions\n",
     "- [`reduction`](https://docs.dask.org/en/latest/generated/dask.dataframe.Series.reduction.html): for custom row-wise reduction operations."
    ]
   },
@@ -829,7 +813,7 @@
     "\n",
     "meta = ddf.head()\n",
     "\n",
-    "ddf = ddf.map_partitions(my_custom_function, True, meta=meta)"
+    "ddf = ddf.map_partitions(my_custom_function, parameter_a=True, meta=meta)"
    ]
   },
   {
@@ -839,8 +823,8 @@
     "We suggest using Dask's `apply` and `map` functions when you can because they already use `map_partitions` internally.\n",
     "\n",
     "Using the correct `meta` is important here, because your output will depend on it. A few notes about `meta`:\n",
-    "* Think of `meta` as a suggestion that Dask will while it is operating lazily and can not infer the output structure.\n",
-    "* The best way to specify `meta` is using a small pandas DataFrame and Series that match the structure of your final output."
+    "* Think of `meta` as a suggestion that Dask uses while it is operating lazily. Importantly `meta` _never infers with the output structure_.\n",
+    "* The best way to specify `meta` is using a small pandas DataFrame or Series that match the structure of your final output."
    ]
   },
   {
@@ -890,13 +874,6 @@
     "    \n",
     "Furthermore, the distributed scheduler (you will learn about it later) allows the same `dask.dataframe` expressions to be executed across a cluster. To enable massive \"big data\" processing, one could execute data ingestion functions such as `read_parquet`, where the data is held on storage accessible to every worker node (e.g., amazon's S3), and because most operations begin by selecting only some columns, transforming and filtering the data, only relatively small amounts of data need to be communicated between the machines."
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -916,7 +893,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.10"
+   "version": "3.10.5"
   }
  },
  "nbformat": 4,

02_array.ipynb

@@ -9,14 +9,20 @@
     "     width=\"30%\"\n",
     "     alt=\"Dask logo\\\">\n",
     "\n",
-    "# Dask Arrays\n",
-    "\n",
-    "<img src=\"https://docs.dask.org/en/stable/_images/dask-array.svg\" width=\"40%\" align=\"right\">\n",
-    "Dask array provides a parallel, larger-than-memory, n-dimensional array using blocked algorithms. \n",
+    "# Dask Arrays - parallelized numpy\n",
+    "Parallel, larger-than-memory, n-dimensional array using blocked algorithms. \n",
     "\n",
     "*  **Parallel**: Uses all of the cores on your computer\n",
     "*  **Larger-than-memory**:  Lets you work on datasets that are larger than your available memory by breaking up your array into many small pieces, operating on those pieces in an order that minimizes the memory footprint of your computation, and effectively streaming data from disk.\n",
-    "*  **Blocked Algorithms**:  Perform large computations by performing many smaller computations.\n",
+    "*  **Blocked Algorithms**:  Perform large computations by performing many smaller computations.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<img src=\"https://docs.dask.org/en/stable/_images/dask-array.svg\" width=\"40%\" align=\"right\">\n",
+    "\n",
     "\n",
     "In other words, Dask Array implements a subset of the NumPy ndarray interface using blocked algorithms, cutting up the large array into many small arrays. This lets us compute on arrays larger than memory using all of our cores. We coordinate these blocked algorithms using Dask graphs.\n",
     "\n",
@@ -715,26 +721,33 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "client.shutdown()"
+    "### Learn More \n",
+    "\n",
+    "Both xarray and zarr have their own tutorials that go into greater depth:\n",
+    "\n",
+    "* [Zarr tutorial](https://zarr.readthedocs.io/en/stable/tutorial.html)\n",
+    "* [Xarray tutorial](https://tutorial.xarray.dev/intro.html)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Useful links\n",
-    "    \n",
-    "* [Array documentation](https://docs.dask.org/en/latest/array.html)\n",
-    "* [Array screencast](https://youtu.be/9h_61hXCDuI)\n",
-    "* [Array API](https://docs.dask.org/en/latest/array-api.html)\n",
-    "* [Array examples](https://examples.dask.org/array.html)\n",
-    "* [Zarr tutorial](https://zarr.readthedocs.io/en/stable/tutorial.html)\n",
-    "* [Xarray tutorial](https://tutorial.xarray.dev/intro.html)"
+    "## Close your cluster\n",
+    "\n",
+    "It's good practice to close any Dask cluster you create:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client.shutdown()"
    ]
   }
  ],
@@ -755,7 +768,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.10"
+   "version": "3.10.5"
   }
  },
  "nbformat": 4,