Update notebooks

- Remove use of curl
- Sync with changes made in ibis repo

Author: jcrist

.gitignore

@@ -127,3 +127,7 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# Tutorial and examples artifacts
+geography.db
+*.log

Dockerfile

@@ -12,7 +12,7 @@ RUN adduser --disabled-password \
     ${NB_USER}
 
 RUN apt-get update && \
-    apt-get install -y git curl && \
+    apt-get install -y git && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 

tutorial/01-Introduction-to-Ibis.ipynb

@@ -6,9 +6,9 @@
    "source": [
     "# Getting started\n",
     "\n",
-    "To start using Ibis, you need a Python environment with Ibis installed. Follow the <a href='../../backends/SQLite#install'>installation instructions for SQLite</a> to setup an environment.\n",
+    "To start using Ibis, you need a Python environment with Ibis installed. If you're running through this tutorial on your own machine (rather than binder) please follow the [installation instructions for SQLite](https://ibis-project.org/docs/latest/backends/SQLite/) to setup an environment.\n",
     "\n",
-    "Once you have your environment ready, to start using Ibis simply import the `ibis` module:"
+    "You'll also need access to the `geography.db` database hosted [here](https://storage.googleapis.com/ibis-tutorial-data/geography.db). Every notebook in the tutorial starts with the following code to download the database if it doesn't already exist."
    ]
   },
   {
@@ -17,16 +17,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import ibis"
+    "from tutorial_utils import setup\n",
+    "setup()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "To make it things easier in this tutorial, we will be using _Ibis interactive mode_. For production code, that will rarely be the case. More details on Ibis non-interactive (aka lazy) mode are covered in the third tutorial, _Expressions, lazy mode and logging queries_.\n",
-    "\n",
-    "To set the interactive mode, use:"
+    "You should now have `ibis` and the tutorial data all setup. We're ready to get started. First lets import `ibis`."
    ]
   },
   {
@@ -35,20 +34,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ibis.options.interactive = True"
+    "import ibis"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next thing we need is to create a connection object. The connection defines where the data is stored and where the computations will be performed.\n",
+    "To make it things easier in this tutorial, we will be using Ibis's \"interactive mode\". This is the recommended mode to use when doing interactive/iterative work with `ibis`. When deploying production code you'll typically run in non-interactive mode. More details on Ibis non-interactive mode are covered in [a later notebook](./03-Expressions-Lazy-Mode-Logging.ipynb).\n",
     "\n",
-    "For a comparison to pandas, this is not the same as where the data is imported from (e.g. `pandas.read_sql`). pandas loads data into memory and performs the computations itself. Ibis won't load the data and perform any computation, but instead will leave the data in the backend defined in the connection, and will _ask_ the backend to perform the computations.\n",
-    "\n",
-    "In this tutorial we will be using a SQLite connection for its simplicity (no installation is needed). But Ibis can work with many different backends, including big data systems, or GPU-accelerated analytical databases. As well as most common relational databases (PostgreSQL, MySQL,...).\n",
-    "\n",
-    "Let's download the SQLite database from the `ibis-tutorial-data` GCS (Google Cloud Storage) bucket, then connect to it using `ibis`."
+    "To enable interactive mode, run:"
    ]
   },
   {
@@ -57,7 +52,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
+    "ibis.options.interactive = True"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next thing we need is to create a connection object. The connection defines where the data is stored and where the computations will be performed.\n",
+    "\n",
+    "For a comparison to pandas, this is not the same as where the data is imported from (e.g. `pandas.read_sql`). pandas loads data into memory and performs the computations itself. Ibis won't load the data and perform any computation, but instead will leave the data in the backend defined in the connection, and will _ask_ the backend to perform the computations.\n",
+    "\n",
+    "In this tutorial we will be using a SQLite connection for its simplicity (no installation is needed). But Ibis can work with many different backends, including big data systems, or GPU-accelerated analytical databases. As well as most common relational databases (PostgreSQL, MySQL,...)."
    ]
   },
   {
@@ -73,8 +79,6 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Note that if you installed Ibis with `pip` instead of `conda`, you may need to install the SQLite backend separately with `pip install 'ibis-framework[sqlite]'`.\n",
-    "\n",
     "### Exploring the data\n",
     "\n",
     "To list the tables in the `connection` object, we can use the `.list_tables()` method. If you are using Jupyter, you can see all the methods and attributes of the `connection` object by writing `connection.` and pressing the `<TAB>` key."
@@ -312,7 +316,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.10.8"
   }
  },
  "nbformat": 4,

tutorial/02-Aggregates-Joins.ipynb

@@ -6,11 +6,9 @@
    "source": [
     "# Aggregating and joining data\n",
     "\n",
-    "This is the second introductory tutorial to Ibis. If you are new to Ibis, you may want to start\n",
-    "by the first tutorial, _01-Introduction-to-Ibis_.\n",
+    "This is the second introductory tutorial to Ibis. If you are new to Ibis, you may want to start at [the beginning of this tutorial](./01-Introduction-to-Ibis.ipynb).\n",
     "\n",
-    "In the first tutorial, we saw how to operate on the data of a table. We will work again with\n",
-    "the `countries` table as we did previously."
+    "In the first notebook we saw how to load and query data using `ibis`. In this notebook we'll continue with the same dataset, building up some more complicated queries."
    ]
   },
   {
@@ -19,7 +17,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
+    "from tutorial_utils import setup\n",
+    "setup()"
    ]
   },
   {
@@ -369,7 +368,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.10.8"
   }
  },
  "nbformat": 4,

tutorial/03-Expressions-Lazy-Mode-Logging.ipynb

@@ -13,7 +13,7 @@
     "In lazy mode, Ibis won't be executing the operations automatically, but instead, will generate an\n",
     "expression to be executed at a later time.\n",
     "\n",
-    "Let's see this in practice, starting with the same example as in previous tutorials - the geography database."
+    "Let's see this in practice, starting with the same database as in previous tutorials."
    ]
   },
   {
@@ -22,7 +22,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
+    "from tutorial_utils import setup\n",
+    "setup()"
    ]
   },
   {
@@ -236,26 +237,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import datetime\n",
-    "import os\n",
-    "import tempfile\n",
     "from pathlib import Path\n",
     "\n",
     "\n",
     "def log_query_to_file(query: str) -> None:\n",
-    "    \"\"\"\n",
-    "    Log queries to `data/tutorial_queries.log`.\n",
-    "\n",
-    "    Each file is a query. Line breaks in the query are\n",
-    "    represented with the string '\\n'.\n",
-    "\n",
-    "    A timestamp of when the query is executed is added.\n",
-    "    \"\"\"\n",
-    "    dirname = Path(tempfile.gettempdir())\n",
-    "    fname = dirname / 'tutorial_queries.log'\n",
-    "    query_in_a_single_line = query.replace('\\n', r'\\n')\n",
+    "    \"\"\"Log queries to `./tutorial_queries.log`.\"\"\"\n",
+    "    fname = Path() / 'tutorial_queries.log'\n",
+    "    query = query.replace(\"\\n\", \" \")\n",
     "    with fname.open(mode='a') as f:\n",
-    "        f.write(f'{query_in_a_single_line}\\n')"
+    "        # log on a single line\n",
+    "        f.write(f\"{query}\\n\")"
    ]
   },
   {
@@ -272,20 +263,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import time\n",
-    "\n",
     "ibis.options.verbose_log = log_query_to_file\n",
     "\n",
     "countries.execute()\n",
-    "time.sleep(1.0)\n",
     "countries['name', 'continent', population_in_millions].limit(3).execute()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This has created a log file in `data/tutorial_queries.log` where the executed queries have been logged."
+    "This has created a log file in `$PWD/tutorial_queries.log` where the executed queries have been logged."
    ]
   },
   {
@@ -294,7 +282,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!cat -n data/tutorial_queries.log"
+    "!cat -n $PWD/tutorial_queries.log"
    ]
   }
  ],
@@ -314,7 +302,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.10.8"
   }
  },
  "nbformat": 4,

tutorial/04-More-Value-Expressions.ipynb

@@ -21,7 +21,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
+    "from tutorial_utils import setup\n",
+    "setup()"
    ]
   },
   {
@@ -31,6 +32,7 @@
    "outputs": [],
    "source": [
     "import ibis\n",
+    "\n",
     "ibis.options.interactive = True\n",
     "\n",
     "connection = ibis.sqlite.connect('geography.db')"
@@ -512,7 +514,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.10.8"
   }
  },
  "nbformat": 4,

tutorial/05-IO-Create-Insert-External-Data.ipynb

@@ -20,7 +20,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
+    "from tutorial_utils import setup\n",
+    "setup()"
    ]
   },
   {
@@ -30,9 +31,10 @@
    "outputs": [],
    "source": [
     "import ibis\n",
+    "\n",
     "ibis.options.interactive = True\n",
     "\n",
-    "connection = ibis.sqlite.connect('geography.db')"
+    "connection = ibis.sqlite.connect(\"geography.db\")"
    ]
   },
   {
@@ -112,106 +114,6 @@
    "source": [
     "connection.drop_table('continents')"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Creating new tables from in-memory Pandas dataframes\n",
-    "\n",
-    "Pandas and NumPy are convenient to create test data in memory as a dataframe. This can then be turned into an Ibis expression using `ibis.memtable`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pandas as pd\n",
-    "import numpy as np\n",
-    "\n",
-    "\n",
-    "def make_students_df(num_records, random_seed=None):\n",
-    "    rng = np.random.default_rng(random_seed)\n",
-    "    return pd.DataFrame(\n",
-    "        {\n",
-    "            \"firstname\": rng.choice([\"Alice\", \"Bob\", \"Jane\", \"John\"], size=num_records),\n",
-    "            \"birth_date\": (\n",
-    "                pd.to_datetime(\"2021-01-01\")\n",
-    "                + pd.to_timedelta(rng.integers(0, 365, size=num_records), unit=\"D\")\n",
-    "            ),\n",
-    "            \"math_grade\": rng.normal(55, 10, size=num_records).clip(0, 100).round(1),\n",
-    "        }\n",
-    "    )\n",
-    "\n",
-    "students_df = make_students_df(21, random_seed=42)\n",
-    "students_memtable = ibis.memtable(students_df)\n",
-    "students_memtable\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "By default `ibis.memtable` uses the `duckdb` in-memory backend to execute queries against the Pandas dataframe data efficiently.\n",
-    "\n",
-    "We can then materialize it as a physical table for a specific backend if necessary:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "connection = ibis.duckdb.connect(\"ibis_tutorial_students.duckdb\")\n",
-    "connection.create_table('students', students_memtable)\n",
-    "students = connection.table('students')\n",
-    "students.group_by(students.birth_date.month()).aggregate(\n",
-    "    count=students.count(),\n",
-    "    avg_math_grade=students.math_grade.mean(),\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Note that NumPy, Pandas and `ibis.memtable` are only suitable to generate data that fits in memory. To generate data larger than memory, we can generate data in chunks and iteratively insert the chunks using `connection.insert(tablename, pandas_dataframe)`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "connection.insert(students.get_name(), make_students_df(10_000, random_seed=43))\n",
-    "students.count()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "connection.insert(students.get_name(), make_students_df(10_000, random_seed=44))\n",
-    "students.count()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "students.group_by(students.birth_date.month()).aggregate(\n",
-    "    count=students.count(),\n",
-    "    avg_math_grade=students.math_grade.mean(),\n",
-    ")"
-   ]
   }
  ],
  "metadata": {
@@ -230,7 +132,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.10.8"
   },
   "vscode": {
    "interpreter": {