Add Overview (#254)

Author: ncclementi

00_overview.ipynb

@@ -4,100 +4,207 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "<img src=\"http://dask.readthedocs.io/en/latest/_images/dask_horizontal.svg\"\n",
-    "     align=\"right\"\n",
-    "     width=\"30%\"\n",
-    "     alt=\"Dask logo\\\">\n",
+    "# Welcome to the Dask Tutorial\n",
     "\n",
-    "# Introduction\n",
+    "<img src=\"https://docs.dask.org/en/latest/_images/dask_horizontal.svg\" align=\"right\" width=\"30%\" alt=\"Dask logo\">\n",
     "\n",
-    "Welcome to the Dask Tutorial.\n",
+    "Dask is a parallel and distributed computing library that scales the existing Python and PyData ecosystem.\n",
     "\n",
-    "Dask is a parallel computing library that scales the existing Python ecosystem. This tutorial will introduce Dask and parallel data analysis more generally.\n",
+    "Dask can scale up to your full laptop capacity and out to a cloud cluster."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## An example Dask computation\n",
+    "\n",
+    "In the following lines of code, we're reading the NYC taxi cab data from 2015 and finding the mean tip amount. Don't worry about the code, this is just for a quick demonstration. We'll go over all of this in the next notebook. :)\n",
+    "\n",
+    "**Note for learners:** This might be heavy for Binder.\n",
     "\n",
-    "Dask can scale down to your laptop and up to a cluster. Here, we'll use an environment you setup on your laptop to analyze medium sized datasets in parallel locally."
+    "**Note for instructors:** Don't forget to open the Dask Dashboard!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import dask.dataframe as dd\n",
+    "from dask.distributed import Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = Client()\n",
+    "client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ddf = dd.read_parquet(\n",
+    "    \"s3://dask-data/nyc-taxi/nyc-2015.parquet/part.*.parquet\",\n",
+    "    columns=[\"passenger_count\", \"tip_amount\"],\n",
+    "    storage_options={\"anon\": True},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = ddf.groupby(\"passenger_count\").tip_amount.mean().compute()\n",
+    "result"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Overview"
+    "## What is [Dask](\"https://www.dask.org/\")?\n",
+    "\n",
+    "There are many parts to the \"Dask\" the project:\n",
+    "* Collections/API also known as \"core-library\".\n",
+    "* Distributed -- to create clusters\n",
+    "* Intergrations and broader ecosystem\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Dask provides multi-core and distributed parallel execution on larger-than-memory datasets.\n",
+    "### Dask Collections\n",
     "\n",
-    "We can think of Dask at a high and a low level\n",
+    "Dask provides **multi-core** and **distributed+parallel** execution on **larger-than-memory** datasets\n",
     "\n",
-    "*  **High-level collections:**  Dask provides high-level Array, Bag, and DataFrame\n",
-    "   collections that mimic NumPy, lists, and Pandas but can operate in parallel on\n",
-    "   datasets that don't fit into memory.  Dask's high-level collections are\n",
-    "   alternatives to NumPy and Pandas for large datasets.\n",
-    "*  **Low-level schedulers:** Dask provides dynamic task schedulers that\n",
-    "   execute task graphs in parallel.  These execution engines power the\n",
-    "   high-level collections mentioned above but can also power custom,\n",
-    "   user-defined workloads.  These schedulers are low-latency (around 1ms) and\n",
-    "   work hard to run computations in a small memory footprint.  Dask's\n",
-    "   schedulers are an alternative to direct use of `threading` or\n",
-    "   `multiprocessing` libraries in complex cases or other task scheduling\n",
-    "   systems like `Luigi` or `IPython parallel`.\n",
+    "We can think of Dask's APIs (also called collections)  at a high and a low level:\n",
     "\n",
-    "Different users operate at different levels but it is useful to understand\n",
-    "both.\n",
+    "<center>\n",
+    "<img src=\"images/high_vs_low_level_coll_analogy.png\" width=\"75%\" alt=\"High vs Low level clothes analogy\">\n",
+    "</center>\n",
     "\n",
-    "The Dask [use cases](https://stories.dask.org/en/latest/) provides a number of sample workflows where Dask should be a good fit."
+    "*  **High-level collections:**  Dask provides high-level Array, Bag, and DataFrame\n",
+    "   collections that mimic NumPy, lists, and pandas but can operate in parallel on\n",
+    "   datasets that don't fit into memory.\n",
+    "* **Low-level collections:**  Dask also provides low-level Delayed and Futures\n",
+    "   collections that give you finer control to build custom parallel and distributed computations."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Prepare"
+    "### Dask Cluster\n",
+    "\n",
+    "Most of the times when you are using Dask, you will be using a distributed scheduler, which exists in the context of a Dask cluster. The Dask cluster is structured as:\n",
+    "\n",
+    "<center>\n",
+    "<img src=\"images/distributed-overview.png\" width=\"75%\" alt=\"Distributed overview\">\n",
+    "</center>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You should clone this repository: \n",
+    "### Dask Ecosystem\n",
     "\n",
-    "    git clone http://github.com/dask/dask-tutorial\n",
+    "In addition to the core Dask library and its distributed scheduler, the Dask ecosystem connects several additional initiatives, including:\n",
     "\n",
-    "The included file `environment.yml` in the `binder` subdirectory contains a list of all of the packages needed to run this tutorial. To install them using `conda`, you can do\n",
+    "- Dask-ML (parallel scikit-learn-style API)\n",
+    "- Dask-image\n",
+    "- Dask-cuDF\n",
+    "- Dask-sql\n",
+    "- Dask-snowflake\n",
+    "- Dask-mongo\n",
+    "- Dask-bigquery\n",
     "\n",
-    "    conda env create -f binder/environment.yml\n",
-    "    conda activate dask-tutorial\n",
-    "    jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
-    "    jupyter labextension install @bokeh/jupyter_bokeh\n",
-    "    \n",
-    "Do this *before* running this notebook."
+    "Community libraries that have built-in dask integrations like:\n",
+    "\n",
+    "- Xarray\n",
+    "- XGBoost\n",
+    "- Prefect\n",
+    "- Airflow\n",
+    "\n",
+    "Dask deployment libraries\n",
+    "- Dask-kubernetes\n",
+    "- Dask-YARN\n",
+    "- Dask-gateway\n",
+    "- Dask-cloudprovider\n",
+    "- jobqueue\n",
+    "\n",
+    "... When we talk about the Dask project we include all these efforts as part of the community. "
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Links"
+    "## Dask Use Cases\n",
+    "\n",
+    "Dask is used in multiple fields such as:\n",
+    "\n",
+    "* Geospatial\n",
+    "* Finance\n",
+    "* Astrophysics\n",
+    "* Microbiology\n",
+    "* Environmental science\n",
+    "\n",
+    "Check out the Dask [use cases](https://stories.dask.org/en/latest/) page that provides a number of sample workflows."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "*  Reference\n",
-    "    *  [Docs](https://dask.org/)\n",
-    "    *  [Examples](https://examples.dask.org/)\n",
-    "    *  [Code](https://github.com/dask/dask/)\n",
-    "    *  [Blog](https://blog.dask.org/)\n",
-    "*  Ask for help\n",
-    "    *   [`dask`](http://stackoverflow.com/questions/tagged/dask) tag on Stack Overflow, for usage questions\n",
-    "    *   [github issues](https://github.com/dask/dask/issues/new) for bug reports and feature requests\n",
-    "    *   [discourse forum](https://dask.discourse.group/) for general, non-bug, questions and discussion\n",
-    "    *   Attend a live tutorial"
+    "## Prepare"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "#### 1. You should clone this repository\n",
+    "\n",
+    "\n",
+    "    git clone http://github.com/dask/dask-tutorial\n",
+    "\n",
+    "and then install necessary packages.\n",
+    "There are three different ways to achieve this, pick the one that best suits you, and ***only pick one option***.\n",
+    "They are, in order of preference:\n",
+    "\n",
+    "#### 2a) Create a conda environment (preferred)\n",
+    "\n",
+    "In the main repo directory\n",
+    "\n",
+    "\n",
+    "    conda env create -f binder/environment.yml\n",
+    "    conda activate dask-tutorial\n",
+    "\n",
+    "\n",
+    "#### 2b) Install into an existing environment\n",
+    "\n",
+    "You will need the following core libraries\n",
+    "\n",
+    "\n",
+    "    conda install -c conda-forge ipycytoscape jupyterlab python-graphviz matplotlib zarr xarray pooch scipy dask distributed dask-labextension\n",
+    "\n",
+    "Note that these options will alter your existing environment, potentially changing the versions of packages you already\n",
+    "have installed."
    ]
   },
   {
@@ -108,26 +215,36 @@
     "\n",
     "Each section is a Jupyter notebook. There's a mixture of text, code, and exercises.\n",
     "\n",
+    "0. [Overview](00_overview.ipynb) - dask's place in the universe.\n",
+    "\n",
+    "1. [Dataframe](01_dataframe.ipynb) - parallelized operations on many pandas dataframes spread across your cluster.\n",
+    "\n",
+    "2. [Array](02_array.ipynb) - blocked numpy-like functionality with a collection of numpy arrays spread across your cluster.\n",
+    "\n",
+    "3. [Delayed](03_dask.delayed.ipynb) - the single-function way to parallelize general python code.\n",
+    "\n",
+    "4. [Deployment/Distributed](04_distributed.ipynb) - Dask's scheduler for clusters, with details of how to view the UI.\n",
+    "\n",
+    "5. [Distributed Futures](05_futures.ipynb) - non-blocking results that compute asynchronously.\n",
+    "\n",
+    "6. Conclusion\n",
+    "\n",
+    "\n",
     "If you haven't used Jupyterlab, it's similar to the Jupyter Notebook. If you haven't used the Notebook, the quick intro is\n",
     "\n",
     "1. There are two modes: command and edit\n",
     "2. From command mode, press `Enter` to edit a cell (like this markdown cell)\n",
     "3. From edit mode, press `Esc` to change to command mode\n",
     "4. Press `shift+enter` to execute a cell and move to the next cell.\n",
     "\n",
-    "The toolbar has commands for executing, converting, and creating cells.\n",
-    "\n",
-    "The layout of the tutorial will be as follows:\n",
-    "- Foundations: an explanation of what Dask is, how it works, and how to use lower-level primitives to set up computations. Casual users may wish to skip this section, although we consider it useful knowledge for all users.\n",
-    "- Distributed: information on running Dask on the distributed scheduler, which enables scale-up to distributed settings and enhanced monitoring of task operations. The distributed scheduler is now generally the recommended engine for executing task work, even on single workstations or laptops.\n",
-    "- Collections: convenient abstractions giving a familiar feel to big data\n",
-    "    - bag: Python iterators with a functional paradigm, such as found in func/iter-tools and toolz - generalize lists/generators to big data; this will seem very familiar to users of PySpark's [RDD](http://spark.apache.org/docs/2.1.0/api/python/pyspark.html#pyspark.RDD)\n",
-    "    - array: massive multi-dimensional numerical data, with Numpy functionality\n",
-    "    - dataframes: massive tabular data, with Pandas functionality\n",
-    "    \n",
-    "Whereas there is a wealth of information in the documentation, linked above, here we aim to give practical advice to aid your understanding and application of Dask in everyday situations. This means that you should not expect every feature of Dask to be covered, but the examples hopefully are similar to the kinds of work-flows that you have in mind.\n",
-    "\n",
-    "## Exercise: Print `Hello, world!`\n",
+    "The toolbar has commands for executing, converting, and creating cells."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Exercise: Print `Hello, world!`\n",
     "Each notebook will have exercises for you to solve. You'll be given a blank or partially completed cell, followed by a hidden cell with a solution. For example.\n",
     "\n",
     "\n",
@@ -157,12 +274,38 @@
    "metadata": {
     "jupyter": {
      "source_hidden": true
-    }
+    },
+    "tags": []
    },
    "outputs": [],
    "source": [
     "print(\"Hello, world!\")"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "## Useful Links"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "*  Reference\n",
+    "    *  [Docs](https://dask.org/)\n",
+    "    *  [Examples](https://examples.dask.org/)\n",
+    "    *  [Code](https://github.com/dask/dask/)\n",
+    "    *  [Blog](https://blog.dask.org/)\n",
+    "*  Ask for help\n",
+    "    *   [`dask`](http://stackoverflow.com/questions/tagged/dask) tag on Stack Overflow, for usage questions\n",
+    "    *   [github issues](https://github.com/dask/dask/issues/new) for bug reports and feature requests\n",
+    "    *   [discourse forum](https://dask.discourse.group/) for general, non-bug, questions and discussion\n",
+    "    *   Attend a live tutorial"
+   ]
   }
  ],
  "metadata": {
@@ -182,7 +325,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.10"
+   "version": "3.9.13"
   }
  },
  "nbformat": 4,

README.md

@@ -6,27 +6,7 @@ This tutorial was last given at SciPy 2020 which was a virtual conference.
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/dask/dask-tutorial/main?urlpath=lab)
 [![Build Status](https://github.com/dask/dask-tutorial/workflows/CI/badge.svg)](https://github.com/dask/dask-tutorial/actions?query=workflow%3ACI)
 
-Dask provides multi-core execution on larger-than-memory datasets.
-
-We can think of dask at a high and a low level
-
-*  **High-level collections:**  Dask provides high-level Array, Bag, and DataFrame
-   collections that mimic NumPy, lists, and Pandas but can operate in parallel on
-   datasets that don't fit into main memory.  Dask's high-level collections are
-   alternatives to NumPy and Pandas for large datasets.
-*  **Low-level schedulers:** Dask provides dynamic task schedulers that
-   execute task graphs in parallel.  These execution engines power the
-   high-level collections mentioned above but can also power custom,
-   user-defined workloads.  These schedulers are low-latency (around 1ms) and
-   work hard to run computations in a small memory footprint.  Dask's
-   schedulers are an alternative to direct use of `threading` or
-   `multiprocessing` libraries in complex cases or other task scheduling
-   systems like `Luigi` or `IPython parallel`.
-
-Different users operate at different levels but it is useful to understand
-both.  This tutorial will interleave between high-level use of `dask.array` and
-`dask.dataframe` (even sections) and low-level use of dask graphs and
-schedulers (odd sections.)
+Dask is a parallel and distributed computing library that scales the existing Python and PyData ecosystem. Dask can scale up to your full laptop capacity and out to a cloud cluster.
 
 ## Prepare
 

binder/environment.yml

@@ -15,6 +15,8 @@ dependencies:
   - pip
   - python-graphviz
   - ipycytoscape
+  - pyarrow
+  - s3fs
   - zarr
   - pooch
   - xarray