edit, rebuild, and recheck

Author: JohnMount

Examples/data_schema/df_types.ipynb

@@ -12,7 +12,9 @@
     "  * Separate types for atomic columns (such as `int`, `bool`, and `float`) and columns of objects (such as `str`).\n",
     "  * No out-of-band representation of missing values. Instead, missingness must be signaled by the insertion of a value representing missingness. This causes problems for types that don't have such a representation such as `int` and `bool`.\n",
     "\n",
-    "To work around the above the Pandas data frame have a number of non-avoidable column type promotion rules and cell type promotion rules. Let's take a look at a problem data frame."
+    "To work around the above the Pandas data frame have a number of non-avoidable column type promotion rules and cell type promotion rules. These promotion rules can introduce their own complexity.\n",
+    "\n",
+    "Let's take a look at a Pandas data frame."
    ]
   },
   {
@@ -198,25 +200,22 @@
    "metadata": {},
    "outputs": [
     {
-     "data": {
-      "text/plain": [
-       "{'b': {int},\n",
-       " 'q': None,\n",
-       " 'r': {float},\n",
-       " 's': {float},\n",
-       " 'x': {float},\n",
-       " 'y': {str},\n",
-       " 'z': {bool, float, int}}"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'b': {<class 'int'>},\n",
+      " 'q': None,\n",
+      " 'r': {<class 'float'>},\n",
+      " 's': {<class 'float'>},\n",
+      " 'x': {<class 'float'>},\n",
+      " 'y': {<class 'str'>},\n",
+      " 'z': {<class 'bool'>, <class 'float'>, <class 'int'>}}\n"
+     ]
     }
    ],
    "source": [
     "# report non-null (not None, NaN, or NaT) found in cells\n",
-    "non_null_types_in_frame(d)"
+    "pprint(non_null_types_in_frame(d))"
    ]
   },
   {
@@ -299,7 +298,9 @@
     "If you are not sure all of your code base (and its dependencies) are consistently only using columns or only using the values attribute, you may experience incompatible mixed types even on uniform data. We know one is not supposed to use \"`.values`\" [from the Pandas documention](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.values.html):\n",
     "\n",
     "\n",
-    "\n",
+    "<dd>\n",
+    "pandas.DataFrame.values\n",
+    "property DataFrame.values\n",
     "<dd><p>Return a Numpy representation of the DataFrame.</p>\n",
     "<div class=\"admonition warning\">\n",
     "<p class=\"admonition-title\">Warning</p>\n",
@@ -311,11 +312,11 @@
     "<dt class=\"field-odd\">Returns<span class=\"colon\">:</span></dt>\n",
     "<dd class=\"field-odd\"><dl class=\"simple\">\n",
     "<dt>numpy.ndarray</dt><dd><p>The values of the DataFrame.</p>\n",
-    "</dd></dd></dd>\n",
+    "</dd></dd></dd></dd>\n",
     "\n",
     "So, presumably, Pandas `.values` is not in fact the attribute it syntactically presents as, but in fact a method interface.\n",
     "\n",
-    "The type recommended `.to_numpy()` seems to return the same `numpy.float64`, which presumably is *not* what is inside the Pandas data frame columns are Series representations."
+    "The type the recommended method `.to_numpy()` seems to return the same `numpy.float64`, which presumably is *not* what is inside the Pandas data frame columns or Series representations. In any case, what types you see in a cell is dependent on what types are in related cells, and what path you use to access the value."
    ]
   },
   {
@@ -342,7 +343,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Any and all of the above inconsistencies can be fairly hazardous to any system that tries to export Pandas to other type sensitive systems (such as databases, JSON, arrow and so on)."
+    "Any and all of the above inconsistencies can be fairly hazardous to any insufficiently careful system that tries to export Pandas to other type sensitive systems (such as databases, JSON, arrow and so on)."
    ]
   },
   {
@@ -351,38 +352,15 @@
    "metadata": {},
    "outputs": [
     {
-     "data": {
-      "text/plain": [
-       "'2.0.3'"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "pd.__version__"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'1.25.2'"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'np': '1.25.2', 'pd': '2.0.3'}\n"
+     ]
     }
    ],
    "source": [
-    "np.__version__"
+    "pprint({'np': np.__version__, 'pd': pd.__version__})"
    ]
   }
  ],

Examples/data_schema/schema_check.ipynb

@@ -6,9 +6,9 @@
    "source": [
     "The Pandas data frame is one tool used to model tabular data in Python. It serves the role an relational database might server for in-memory data, using methods instead of a relational query language.\n",
     "\n",
-    "The difference of Pandas from relational tables is not fundamental, and can even extend Pandas to accept or different operator algebras, as we have done in with [the data algebra](https://github.com/WinVector/data_algebra). However, a missing component is: [data schema](https://en.wikipedia.org/wiki/Database_schema) definition and invariant enforcement.\n",
+    "The difference of Pandas from relational tables is not fundamental, and can even extend Pandas to accept or different operator algebras, as we have done in with [the data algebra](https://github.com/WinVector/data_algebra). However, a common missing component is: [data schema](https://en.wikipedia.org/wiki/Database_schema) definition and invariant enforcement.\n",
     "\n",
-    "It turns out it is quite simple to add such functionality using Python decorators. This isn't particularly useful for general functions (such as `pd.merge()`), where the function is supposed to support arbitrary data schema. However, it can be very useful in adding checks and safety to specific applications and analysis workflows built on top such generic functions. In fact, it is a good way to copy schema details from external data sources such as databases or CSV into enforced application invariants.\n",
+    "It turns out it is quite simple to add such functionality using Python decorators. This isn't particularly useful for general functions (such as `pd.merge()`), where the function is supposed to support arbitrary data schema. However, it can be *very* useful in adding checks and safety to specific applications and analysis workflows built on top such generic functions. In fact, it is a good way to copy schema details from external data sources such as databases or CSV into enforced application invariants. Application code that transforms fixed tables into expected exported results can benefit greatly from schema documentation and enforcement.\n",
     "\n",
     "In this note I will demonstrate the how to add schema documentation and enforcement to Python functions working over data frames using Python decorators.\n",
     "\n",
@@ -27,7 +27,7 @@
     "import pandas as pd\n",
     "import polars as pl\n",
     "import data_algebra as da\n",
-    "from data_algebra.data_schema import non_null_types_in_frame, SchemaCheckSwitch"
+    "from data_algebra.data_schema import SchemaCheckSwitch"
    ]
   },
   {
@@ -769,7 +769,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And that is a simple tool to add schemas to some of you data analysis functions."
+    "The `SchemaCheck` decoration is simple and effective tool to add schema documentation and enforcement to your analytics projects."
    ]
   },
   {
@@ -787,7 +787,11 @@
    ],
    "source": [
     "# show some relevant versions\n",
-    "pprint({'pd': pd.__version__, 'pl': pl.__version__, 'np': np.__version__, 'da': da.__version__})"
+    "pprint({\n",
+    "    'pd': pd.__version__, \n",
+    "    'pl': pl.__version__, \n",
+    "    'np': np.__version__, \n",
+    "    'da': da.__version__})"
    ]
   }
  ],

data_algebra.egg-info/requires.txt

@@ -1,5 +1,5 @@
-numpy>=1.25.2
-pandas>=2.0.3
+numpy>=1.25.0
+pandas>=2.0.0
 lark
 
 [BigQuery]

dist/data_algebra-1.6.9-py3-none-any.whl

@@ -4,7 +4,7 @@ bash ./clean.bash
 # pytest --cov data_algebra tests > coverage.txt
 pip install --no-deps -e "$(pwd)"
 pytest --cov-report term-missing --cov data_algebra tests > coverage.txt
-pdoc3 -o ./docs ./data_algebra
+pdoc3 --force -o ./docs ./data_algebra
 cat coverage.txt
 python3 setup.py sdist bdist_wheel
 # pip install dist/data_algebra-*.tar.gz

dist/data_algebra-1.6.9.tar.gz

@@ -27,8 +27,8 @@
     url='https://github.com/WinVector/data_algebra',
     packages=setuptools.find_packages(exclude=['tests', 'Examples']),
     install_requires=[
-        "numpy>=1.25.2",
-        "pandas>=2.0.3",
+        "numpy>=1.25.0",
+        "pandas>=2.0.0",
         "lark"
     ],
     extras_require={