add example: read remote parquet using duckdb (#14)

Author: p-a-a-a-trick

examples/Reading-Parquet-Files-using-DuckDB.ipynb

@@ -0,0 +1,178 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "379407de-ed10-472c-ad81-228ba73c7d15",
+   "metadata": {},
+   "source": [
+    "# Reading Parquet Files using DuckDB\n",
+    "\n",
+    "In this example, we will use Ibis's DuckDB backend to analyze data from a remote parquet source using `ibis.read_parquet`.\n",
+    "`ibis.read_parquet` can also read local parquet files,\n",
+    "and there are other `ibis.read_*` functions that conveniently return a table expression from a file.\n",
+    "One such function is `ibis.read_csv`, which reads from local and remote CSV.\n",
+    "\n",
+    "We will be reading from the [**Global Biodiversity Information Facility (GBIF) Species Occurrences**](https://registry.opendata.aws/gbif/) dataset.\n",
+    "It is hosted on S3 at `s3://gbif-open-data-us-east-1/occurrence/`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4402d524-bd38-4127-a8ec-500be723711c",
+   "metadata": {},
+   "source": [
+    "## Reading One Partition\n",
+    "\n",
+    "We can read a single partition by specifying its path.\n",
+    "\n",
+    "We do this by calling [`read_parquet`](https://ibis-project.org/api/expressions/top_level/#ibis.read_parquet) on the partition we care about.\n",
+    "\n",
+    "So to read the first partition in this dataset, we'll call `read_parquet` on `00000` in that path:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "062ba84c-1f4f-4ec7-9df5-73444c491342",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import ibis\n",
+    "\n",
+    "t = ibis.read_parquet(f\"s3://gbif-open-data-us-east-1/occurrence/2023-04-01/occurrence.parquet/000000\")\n",
+    "t"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5440fa0f-2aca-40da-b4ed-4fde06051e10",
+   "metadata": {},
+   "source": [
+    "Note that we're calling `read_parquet` and receiving a table expression without establishing a connection first.\n",
+    "Ibis spins up a DuckDB connection (or whichever default backend you have) when you call `ibis.read_parquet` (or even `ibis.read_csv`).\n",
+    "\n",
+    "Since our result, `t`, is a table expression, we can now run queries against the file using Ibis expressions.\n",
+    "For example, we can select columns, filter the file, and then view the first five rows of the result:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "035e845c-761a-4728-9361-ae33f3205c45",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "cols = ['gbifid', 'datasetkey', 'occurrenceid', 'kingdom',\n",
+    "        'phylum', 'class', 'order', 'family', 'genus',\n",
+    "        'species', 'day', 'month', 'year']\n",
+    "\n",
+    "t.select(cols).filter(t['family'].isin(['Corvidae'])).limit(5).execute()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4595a5ae-0007-4b8a-8e31-803d92e7e52c",
+   "metadata": {},
+   "source": [
+    "or count the rows in the table (partition):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bd6d8cc6-ce49-44dd-9507-bd26176127f8",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "t.count().execute()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4286d9f0-8e06-498b-a561-e75193126adc",
+   "metadata": {},
+   "source": [
+    "## Reading All Partitions: Filter, Aggregate, Export\n",
+    "We can use `read_parquet` to read an entire parquet file by globbing all partitions:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3d2246c9-57b0-4b6c-8849-e8d2d85b29bb",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "t = ibis.read_parquet(f\"s3://gbif-open-data-us-east-1/occurrence/2023-04-01/occurrence.parquet/*\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9bd746c0-d414-4212-ab76-c5d585bafc82",
+   "metadata": {},
+   "source": [
+    "and since the function returns a table expression, we can perform valid selections, filters, aggregations, and exports just as we could with any other table expression:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0f92c38b-1487-464c-86a2-4b922831207e",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "df = (\n",
+    "    t.select(['gbifid', 'family', 'species'])\n",
+    "    .filter(t['family'].isin(['Corvidae']))\n",
+    "    # Here we limit by 10,000 to fetch a quick batch of results\n",
+    "    .limit(10000)\n",
+    "    .group_by('species')\n",
+    "    .count()\n",
+    "    .execute()\n",
+    ")\n",
+    "\n",
+    "print(df.shape)\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "aecbd689-d632-42e1-80ed-28a7f0a22d17",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}