edit and add links

Author: JohnMount

Examples/data_schema/df_types.ipynb

@@ -343,7 +343,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Any and all of the above inconsistencies can be fairly hazardous to any insufficiently careful system that tries to export Pandas to other type sensitive systems (such as databases, JSON, arrow and so on)."
+    "Any and all of the above inconsistencies can be fairly hazardous to any insufficiently careful system that tries to export Pandas to other type sensitive systems (such as databases, JSON, arrow and so on).\n",
+    "\n",
+    "Additional advanced data type schema documentation tools can be found [here](https://github.com/WinVector/data_algebra/blob/main/Examples/data_schema/schema_check.ipynb)."
    ]
   },
   {

Examples/data_schema/schema_check.ipynb

@@ -4,16 +4,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The Pandas data frame is one tool used to model tabular data in Python. It serves the role an relational database might server for in-memory data, using methods instead of a relational query language.\n",
+    "The Pandas data frame is the most popular tool used to model tabular data in Python. It serves the role an relational database might server for in-memory data, using methods instead of a relational query language.\n",
     "\n",
-    "The difference of Pandas from relational tables is not fundamental, and can even extend Pandas to accept or different operator algebras, as we have done in with [the data algebra](https://github.com/WinVector/data_algebra). However, a common missing component is: [data schema](https://en.wikipedia.org/wiki/Database_schema) definition and invariant enforcement.\n",
+    "The differences of Pandas from relational tables are not fundamental. One can even extend Pandas to accept or different operator algebras, as we have done in with [the data algebra](https://github.com/WinVector/data_algebra). However, a common missing component is: [data schema](https://en.wikipedia.org/wiki/Database_schema) definition, documentation, and invariant enforcement.\n",
     "\n",
-    "It turns out it is quite simple to add such functionality using Python decorators. This isn't particularly useful for general functions (such as `pd.merge()`), where the function is supposed to support arbitrary data schema. However, it can be *very* useful in adding checks and safety to specific applications and analysis workflows built on top such generic functions. In fact, it is a good way to copy schema details from external data sources such as databases or CSV into enforced application invariants. Application code that transforms fixed tables into expected exported results can benefit greatly from schema documentation and enforcement.\n",
+    "It turns out it is quite simple to add such functionality using Python decorators. This isn't particularly useful for general functions (such as `pd.merge()`), where the function is supposed to support arbitrary data schemas. However, it can be *very* useful in adding checks and safety to specific applications and analysis workflows built on top such generic functions. In fact, it is a good way to copy schema details from external data sources such as databases or CSV into enforced application invariants. Application code that transforms fixed tables into expected exported results can benefit greatly from schema documentation and enforcement.\n",
     "\n",
     "I propose a simple check criteria for both function signatures and data frames that applies to both inputs and outputs:\n",
     "\n",
     "  * Data must have *at least* the set of argument names or column names specified.\n",
-    "  * Each column must *no more* types (for non-null values) than the types specified.\n",
+    "  * Each column must have *no more* types (for non-null values) than the types specified.\n",
     "\n",
     "In this note I will demonstrate the how to add schema documentation and enforcement to Python functions working over data frames using Python decorators.\n",
     "\n",
@@ -39,16 +39,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We supply a lightweight implementation of a schema enforcement system.\n",
+    "I will supply a lightweight implementation of a schema enforcement system.\n",
     "\n",
-    "This is a bit different that having a type system. We are not interested what is and what is not a data frame. But instead interested in documenting that the data frames we work with have:\n",
+    "This is a bit different that having a type system. We are interested in documenting that the data frames we work with have:\n",
     "\n",
     "  * At least the columns we expect.\n",
     "  * No types we don't expect in those columns.\n",
     "\n",
-    "These two covariant constraints are what we want to ensure we can write the operations over columns (which we need to know exist) and not get unexpected results (from unexpected types). The idea is: can we document and enforce (at least partial) schemas both on function signatures and data frames? This can be particularly useful for data science code near external data sources such as databases or CSV (comma separated value) files. Many of these sources themselves have data schemas and schema documentation.\n",
+    "These two covariant constraints are what we want to ensure we can write the operations over columns (which we need to know exist) and not get unexpected results (from unexpected types). Instead of getting down-stream signalling nor non-signalling errors during column operations, we get useful assertions on columns and values relative to our documented data model. This can be particularly useful for data science code near external data sources such as databases or CSV (comma separated value) files. Many of these sources themselves have data schemas and schema documentation.\n",
     "\n",
-    "I've started experimenting with a Python module to automate this task as debugging feature. As it is a debugging feature we want to be able to turn the feature on or off in an entire code base easily. To do this we define a indirect importer called [`schema_check.py`](https://github.com/WinVector/data_algebra/blob/main/Examples/data_schema/schema_check.py).  It's code looks like the following:\n",
+    "I've started experimenting with a Python module to automate this task as debugging feature. As it is a debugging feature, we want to be able to turn the feature on or off in an entire code base easily. To do this we define a indirect importer called [`schema_check.py`](https://github.com/WinVector/data_algebra/blob/main/Examples/data_schema/schema_check.py).  It's code looks like the following:\n",
     "\n",
     "```\n",
     "   from data_schema import SchemaCheckSwitch\n",
@@ -77,7 +77,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "What we have imported is a decorator that shows the types schemas of at least a subset of positional and named arguments. Declarations are either Python types, or sets of types. A special case is dictoinaries, which specify a subset of the column structure of data frames. \"return_spec\" is reserved to name the return schema of the function.\n"
+    "\n",
+    "The usual way to define a function in Python is as follows."
    ]
   },
   {
@@ -97,7 +98,7 @@
    "metadata": {},
    "source": [
     "\n",
-    "Let's decorate a function with `SchemaCheck`. The details of this decorator are documented [here](https://github.com/WinVector/Examples/tree/main/arg_types#readme)."
+    "Let's decorate the same function with `SchemaCheck`. The details of this decorator are documented [here](https://github.com/WinVector/Examples/tree/main/arg_types#readme)."
    ]
   },
   {
@@ -122,16 +123,18 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This declaring that `fn()` expects at least:\n",
+    "The decorator defines the types schemas of at least a subset of positional and named arguments. Declarations are either Python types, or sets of types. A special case is dictionaries, which specify a subset of the column structure of function signatures or data frames. \"return_spec\" is reserved to name the return schema of the function.\n",
+    "\n",
+    "Our `fn()` expects at least:\n",
     "\n",
     "  * an argument `a` of type `int`.\n",
     "  * an argument `b` of type `int` or `float`.\n",
     "  * an argument `c` that is a data frame (implied by the dictionary argument), and that data frame contains a column `x` that has no non-null elements of type other than `int`.\n",
-    "  * The functions returns a data frame (indicated by the dictionary argument) that has at least a column `z` that contains no non-null elements of type other than `float`.\n",
+    "  * The function returns a data frame (indicated by the dictionary argument) that has at least a column `z` that contains no non-null elements of type other than `float`.\n",
     "\n",
     "This gives us some enforceable invariants that can improve our code.\n",
     "\n",
-    "We can somewhat see this repeated back in the decorator altered `help()`."
+    "We can see this repeated back in the decorator altered `help()`."
    ]
   },
   {
@@ -168,7 +171,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "It is a learnable schema specification convention.\n",
+    "This is a learnable schema specification convention.\n",
     "\n",
     "Let's see it catch an error. We show what happens if we call `fn()` with none of the expected arguments."
    ]
@@ -201,38 +204,6 @@
     "assert threw"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can try with just one argument missing."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "function fn(), issues:\n",
-      "expected arg c missing\n"
-     ]
-    }
-   ],
-   "source": [
-    "# catch schema mismatch\n",
-    "try:\n",
-    "    fn(1, 2)\n",
-    "except TypeError as e:\n",
-    "    print(e)\n",
-    "    threw = True\n",
-    "assert threw"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -242,7 +213,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [
     {
@@ -270,16 +241,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And we show that this checking pushes down into the structure of data frame arguments! \n",
-    "\n",
-    "These sort of checks are not for generic utility methods (such as `pd.merge()`), which are designed to work over a larger variety of schema. However, they are very useful near client interfaces, APIs, and database tables. This is a place where there is fixed schema information, and one can benefit from preserving it for just a bit longer. This technique and [the data algebra](https://github.com/WinVector/data_algebra) may naturally live near data sources.\n",
-    "\n",
-    "In data science the natural types are data frame schemas, knowing the type of the outer variables just isn't and interesting invariant."
+    "And we show that this checking pushes down into the structure of data frame arguments! In our next example we see the argument is missing a required column.\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [
     {
@@ -312,7 +279,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [
     {
@@ -340,12 +307,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And we check return types."
+    "And we can check return types."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
@@ -393,7 +360,7 @@
        "0  7.0"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -424,12 +391,14 @@
    "source": [
     "Notice the rejected return value is attached to the `TypeError` to help with diagnosis and debugging.\n",
     "\n",
+    "Again, sort of checks are not for generic utility methods (such as `pd.merge()`), which are designed to work over a larger variety of schemas. However, they are very useful near client interfaces, APIs, and database tables. This is a place where there is fixed schema information, and one can benefit from preserving it for just a bit longer. This technique and [data algebra](https://github.com/WinVector/data_algebra) processing may naturally live near data sources. In data science the natural types are data frame schemas, knowing the type of the outer variables just isn't and interesting invariant.\n",
+    "\n",
     "Now, let's show a successful call."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [
     {
@@ -470,7 +439,7 @@
        "0  7.0"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -492,7 +461,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -509,7 +478,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 13,
    "metadata": {},
    "outputs": [
     {
@@ -550,7 +519,7 @@
        "0  7.0"
       ]
      },
-     "execution_count": 14,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -570,14 +539,14 @@
    "source": [
     "The return value has is missing the required `z` column, but with checks off the function is not interfered with.\n",
     "\n",
-    "The idea is: when checks are on failures are detected much closer to causes, making debugging and diagnosis much easier. Also the decorations are a easy way to document in human readable form some basics of the expected input and output schemas.\n",
+    "When checks are on: failures are detected much closer to causes, making debugging and diagnosis much easier. Also, the decorations are a easy way to document in human readable form some basics of the expected input and output schemas.\n",
     "\n",
-    "And the input and output schema are attached to the function as objects."
+    "And, the input and output schema are attached to the function as objects."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 14,
    "metadata": {},
    "outputs": [
     {
@@ -597,7 +566,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 15,
    "metadata": {},
    "outputs": [
     {
@@ -617,10 +586,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This makes the schema data available for other use, even some automatic checking of function composition conditions!\n",
+    "This makes the schema data available for other uses.\n",
     "\n",
-    "\n",
-    "The technique can run into what I call \"the first rule of meta-programming\". Meta-programming only works as long as it doesn't run into other meta-programming. That being said, I feel these decorators can be very valuable in Python data science projects.\n",
+    "A downside is, the technique *can* run into what I call \"the first rule of meta-programming\". Meta-programming only works as long as it doesn't run into other meta-programming (also called the \"its only funny when I do it\" theorem). That being said, I feel these decorators can be very valuable in Python data science projects.\n",
     "\n",
     "The implementation, documentation, and demo of this methodology can be found [here](https://github.com/WinVector/data_algebra/tree/main/Examples/data_schema).\n"
    ]
@@ -629,12 +597,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Note, the system works about the same with Polars instead of Pandas as the data frame realization."
+    "Note, the system also works with Polars data frames instead of Pandas as the data frame realization."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 16,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -644,7 +612,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 17,
    "metadata": {},
    "outputs": [
     {
@@ -670,7 +638,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 18,
    "metadata": {},
    "outputs": [
     {
@@ -703,7 +671,7 @@
        "└─────┘"
       ]
      },
-     "execution_count": 19,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -730,7 +698,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 19,
    "metadata": {},
    "outputs": [
     {
@@ -756,7 +724,7 @@
        "└─────┘"
       ]
      },
-     "execution_count": 20,
+     "execution_count": 19,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -774,12 +742,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The `SchemaCheck` decoration is simple and effective tool to add schema documentation and enforcement to your analytics projects."
+    "And we also have simple \"types in data frame\" inspection tools [here](https://github.com/WinVector/data_algebra/blob/main/Examples/data_schema/df_types.ipynb)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In conclusion: the `SchemaCheck` decoration is simple and effective tool to add schema documentation and enforcement to your analytics projects."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 20,
    "metadata": {},
    "outputs": [
     {