Add PyDough metadata for masked table columns (#401)

Adds a new metadata type `masked table column`, a subclass of `table column` representing data that is stored in the underlying data in some kind of encrypted manner but where the metadata stores additional information such as how to decrypt the data. These new metadata column types have the following additional properties:
- `unprotect protocol`: a Python format string containing SQL text that, when the value is injected, indicates how to decrypt the data.
- `protect protocol`: a Python format string containing SQL text that, when a value is injected, indicates how to encrypt the value in the same manner as the column was encrypted, so the unprotect protocol will reverse it.
- `server masked`: optional boolean (default False) which, if True, indicates that information about the encryption/decryption scheme is available in a server that can be queried to rewrite/optimize predicates to avoid unmasking.

For example, suppose a string column `name` is "encrypted" by uppercasing it, and "decrypted" by lowercasing it, and does not have a server. This would be the following metadata:
```json
{
  "name": "name",
  "type": "masked table column",
  "column name": "c_name",
  "data type": "string",
  "unprotect protocol": "UPPER({})",
  "protect protocol": "LOWER({})",
  "server masked": false
}
```

So for instance, when reading the data from the table, instead of placing `c_name` in the SELECT clause we would place `LOWER(c_name)` in the SELECT clause so it is "decrypted" (since the data int he table was uppercase). Conversely, if we wish to do a filter on `name == "john smith"`, instead of doing `LOWER(c_name) == "john smith"` we could use the protect protocol to rewrite as `c_name == UPPER("john smith")`.

Author: knassre-bodo

documentation/metadata.md

@@ -9,6 +9,7 @@ This page document the exact format that the JSON files containing PyDough metad
    * [Collection Type: Simple Table](#collection-type-simple-table)
 - [Properties](#properties)
    * [Property Type: Table Column](#property-type-table-column)
+   * [Property Type: Masked Table Column](#property-type-masked-table-column)
 - [Relationships](#relationships)
    * [Relationship Type: Simple Join](#relationship-type-simple-join)
    * [Relationship Type: General Join](#relationship-type-general-join)
@@ -136,15 +137,63 @@ Example of the structure of the metadata for a table column property:
 {
     "name": "account balance",
     "type": "table column",
-    "column_name": "ba_bal",
-    "data_type": "numeric",
+    "column name": "ba_bal",
+    "data type": "numeric",
     "description": "The amount of money currently in the account",
     "sample values": [0.0, 123.45, 999864.00],
     "synonyms": ["amount", "value", "balance"],
     "extra semantic info": {...}
 }
 ```
 
+<!-- TOC --><a name="property-type-masked-table-column"></a>
+### Property Type: Masked Table Column
+
+A property with this type is the same as a regular table column, except the data in the underlying table has been masked using an encryption protocol. The metadata includes the information required to either encrypt values in SQL using the same masking protocol, or unmask previously masked data in SQL queries. 
+
+Properties of this type use the type string "masked table column" and include all the properties from [table column](#property-type-table-column), plus the following additional key-value pairs in their metadata JSON object:
+
+- `unprotect protocol` (required): a Python format string representing the SQL text that is used to unmask the data after reading it from the underlying table. The format string should expect a single placeholder value (e.g. `"SUBSTRING({0}, -1) || SUBSTRING({0}, 1, LENGTH({0}) - 1)".format("c_name")` will generate the SQL text `SUBSTRING(c_name, -1) || SUBSTRING(c_name, 1, LENGTH(c_name) - 1)`).
+- `protect protocol` (required): a Python format string, in the same format as `unprotect protocol`, used to describe how the data was originally masked. This can be used to generate masked values consistent with the encryption scheme, allowing operations such as comparisons between masked data.
+- `protected data type` (optional): same as `data type`, except referring to the type of the data when it is protected, whereas `data type` refers to the raw unprotected column. If omitted, it is assumed that the data type is the same between the unprotected vs protected data.
+- `server masked` (optional): a boolean flag indicating whether the column was masked on a server that is attached to PyDough. If `true`, PyDough can use it to optimize queries by rewriting predicates and expressions to avoid unmasking the data.
+
+Example of the structure of the metadata for a masked table column property where the string data is masked by moving the first character to the end, and unmasked by moving it back to the beginning:
+
+```json
+{
+    "name": "name",
+    "type": "masked table column",
+    "column name": "c_name",
+    "data type": "string",
+    "unprotect protocol": "SUBSTRING({0}, -1) || SUBSTRING({0}, 1, LENGTH({0}) - 1)",
+    "protect protocol": "SUBSTRING({0}, 2) || SUBSTRING({0}, 1, 1)",
+    "description": "The name of the customer",
+    "sample values": ["John Smith", "Adrien Lee", "Anna Rodriguez"],
+    "synonyms": ["full name"],
+    "extra semantic info": {...}
+}
+```
+
+Another example of the structure of the metadata for a masked table column property where the numeric is masked by converting it to a string, switching the `0` digits with asterisks, and left-padding to length 10 with asterisks, then unmasked by reversing the process:
+
+```json
+{
+    "name": "account_id",
+    "type": "masked table column",
+    "column name": "a_id",
+    "data type": "numeric",
+    "protected data type": "string",
+    "unprotect protocol": "INTEGER(REPLACE({0}, '*', '0'))",
+    "protect protocol": "LPAD(REPLACE(STRING({0}), '0', '*'), 10, '*')",
+    "server masked": false,
+    "description": "The id of the bank account",
+    "sample values": [12030061, 4000013, 560003],
+    "synonyms": ["account key", "bank account number"],
+    "extra semantic info": {...}
+}
+```
+
 <!-- TOC --><a name="relationships"></a>
 ## Relationships
 

pydough/metadata/__init__.py

@@ -7,6 +7,7 @@
     "CollectionMetadata",
     "GeneralJoinMetadata",
     "GraphMetadata",
+    "MaskedTableColumnMetadata",
     "PropertyMetadata",
     "PyDoughMetadataException",
     "SimpleJoinMetadata",
@@ -23,6 +24,7 @@
 from .properties import (
     CartesianProductMetadata,
     GeneralJoinMetadata,
+    MaskedTableColumnMetadata,
     PropertyMetadata,
     SimpleJoinMetadata,
     SubcollectionRelationshipMetadata,

pydough/metadata/abstract_metadata.py

@@ -6,6 +6,8 @@
 
 from abc import ABC, abstractmethod
 
+from .errors import extract_array, extract_object, extract_string
+
 
 class AbstractMetadata(ABC):
     """
@@ -28,6 +30,26 @@ def __init__(
         self._synonyms: list[str] | None = synonyms
         self._extra_semantic_info: dict | None = extra_semantic_info
 
+    def parse_optional_properties(self, meta_json: dict) -> None:
+        """
+        Parse the optional metadata fields from a JSON object describing
+        the metadata to fill the description / synonyms / extra semantic info
+        fields of the metadata object.
+
+        Args:
+            `meta_json`: the JSON object describing the metadata.
+        """
+        if "description" in meta_json:
+            self._description = extract_string(
+                meta_json, "description", self.error_name
+            )
+        if "synonyms" in meta_json:
+            self._synonyms = extract_array(meta_json, "synonyms", self.error_name)
+        if "extra semantic info" in meta_json:
+            self._extra_semantic_info = extract_object(
+                meta_json, "extra semantic info", self.error_name
+            )
+
     @property
     @abstractmethod
     def error_name(self) -> str:

pydough/metadata/collections/collection_metadata.py

@@ -213,36 +213,6 @@ def get_property(self, property_name: str) -> AbstractMetadata:
     def __getitem__(self, key: str):
         return self.get_property(key)
 
-    @staticmethod
-    def get_class_for_collection_type(
-        name: str, error_name: str
-    ) -> type["CollectionMetadata"]:
-        """
-        Fetches the PropertyType implementation class for a string
-        representation of the collection type.
-
-        Args:
-            `name`: the string representation of a collection type.
-            `error_name`: the string used in error messages to describe
-            the object that `name` came from.
-
-        Returns:
-            The class of the property type corresponding to `name`.
-
-        Raises:
-            `PyDoughMetadataException` if the string does not correspond
-            to a known class type.
-        """
-        from .simple_table_metadata import SimpleTableMetadata
-
-        match name:
-            case "simple_table":
-                return SimpleTableMetadata
-            case property_type:
-                raise PyDoughMetadataException(
-                    f"Unrecognized collection type for {error_name}: {repr(property_type)}"
-                )
-
     def add_properties_from_json(self, properties_json: list) -> None:
         """
         Insert the scalar properties from the JSON for collection into the
@@ -253,7 +223,7 @@ def add_properties_from_json(self, properties_json: list) -> None:
             scalar property that should be parsed and inserted into the
             collection.
         """
-        from pydough.metadata.properties import TableColumnMetadata
+        from pydough.metadata import MaskedTableColumnMetadata, TableColumnMetadata
 
         for property_json in properties_json:
             # Extract the name/type, and create the string used to identify
@@ -269,6 +239,10 @@ def add_properties_from_json(self, properties_json: list) -> None:
                     TableColumnMetadata.parse_from_json(
                         self, property_name, property_json
                     )
+                case "masked table column":
+                    MaskedTableColumnMetadata.parse_from_json(
+                        self, property_name, property_json
+                    )
                 case _:
                     raise PyDoughMetadataException(
                         f"Unrecognized property type {property_type!r} for {error_name}"

pydough/metadata/collections/simple_table_metadata.py

@@ -8,7 +8,6 @@
     NoExtraKeys,
     PyDoughMetadataException,
     extract_array,
-    extract_object,
     extract_string,
     is_string,
     unique_properties_predicate,
@@ -41,9 +40,9 @@ def __init__(
         graph,
         table_path: str,
         unique_properties: list[str | list[str]],
-        description: str | None,
-        synonyms: list[str] | None,
-        extra_semantic_info: dict | None,
+        description: str | None = None,
+        synonyms: list[str] | None = None,
+        extra_semantic_info: dict | None = None,
     ):
         super().__init__(name, graph, description, synonyms, extra_semantic_info)
         is_string.verify(table_path, f"Property 'table_path' of {self.error_name}")
@@ -151,18 +150,6 @@ def parse_from_json(
             collection_json, error_name
         )
         unique_properties: list[str | list[str]] = collection_json["unique properties"]
-        # Extract the optional fields from the JSON.
-        description: str | None = None
-        synonyms: list[str] | None = None
-        extra_semantic_info: dict | None = None
-        if "description" in collection_json:
-            description = extract_string(collection_json, "description", error_name)
-        if "synonyms" in collection_json:
-            synonyms = extract_array(collection_json, "synonyms", error_name)
-        if "extra semantic info" in collection_json:
-            extra_semantic_info = extract_object(
-                collection_json, "extra semantic info", error_name
-            )
         NoExtraKeys(SimpleTableMetadata.allowed_fields).verify(
             collection_json, error_name
         )
@@ -171,10 +158,9 @@ def parse_from_json(
             graph,
             table_path,
             unique_properties,
-            description,
-            synonyms,
-            extra_semantic_info,
         )
+        # Parse the optional common semantic properties like the description.
+        new_collection.parse_optional_properties(collection_json)
         properties: list = extract_array(
             collection_json, "properties", new_collection.error_name
         )

pydough/metadata/graphs/graph_metadata.py

@@ -48,7 +48,9 @@ def __init__(
         self._name: str = name
         self._collections: dict[str, AbstractMetadata] = {}
         self._functions: dict[str, ExpressionFunctionOperator] = {}
-        super().__init__(description, synonyms, extra_semantic_info)
+        self._description = description
+        self._synonyms = synonyms
+        self._extra_semantic_info = extra_semantic_info
 
     @property
     def name(self) -> str:

pydough/metadata/properties/README.md

@@ -20,6 +20,7 @@ The properties classes in PyDough follow a hierarchy that includes both abstract
 - [`PropertyMetadata`](property_metadata.py) (abstract): Base class for all property metadata.
     - [`ScalarAttributeMetadata`](scalar_attribute_metadata.py) (abstract): Base class for properties that are scalars within each record of a collection.
         - [`TableColumnMetadata`](table_column_metadata.py) (concrete): Represents a column of data from a relational table.
+            - [`MaskedTableColumnMetadata`](masked_table_column_metadata.py) (concrete): Represents a variant of a TableColumnMetadata where the data in the table has been encrypted by a masking protocol but the metadata stores information about that protocol, including how to unmask it when reading the data from the table.
     - [`SubcollectionRelationshipMetadata`](subcollection_relationship_metadata.py) (abstract): Base class for properties that map to a subcollection of a collection.
         - [`ReversiblePropertyMetadata`](reversible_property_metadata.py) (abstract): Base class for properties that map to a subcollection and have a corresponding reverse relationship.
             - [`CartesianProductMetadata`](cartesian_product_metadata.py) (concrete): Represents a cartesian product between a collection and its subcollection.

pydough/metadata/properties/__init__.py

@@ -5,6 +5,7 @@
 __all__ = [
     "CartesianProductMetadata",
     "GeneralJoinMetadata",
+    "MaskedTableColumnMetadata",
     "PropertyMetadata",
     "ReversiblePropertyMetadata",
     "ScalarAttributeMetadata",
@@ -15,6 +16,7 @@
 
 from .cartesian_product_metadata import CartesianProductMetadata
 from .general_join_metadata import GeneralJoinMetadata
+from .masked_table_column_metadata import MaskedTableColumnMetadata
 from .property_metadata import PropertyMetadata
 from .reversible_property_metadata import ReversiblePropertyMetadata
 from .scalar_attribute_metadata import ScalarAttributeMetadata

pydough/metadata/properties/cartesian_product_metadata.py

@@ -9,9 +9,7 @@
 from pydough.metadata.collections import CollectionMetadata
 from pydough.metadata.errors import (
     NoExtraKeys,
-    extract_array,
     extract_bool,
-    extract_object,
     extract_string,
 )
 from pydough.metadata.graphs import GraphMetadata
@@ -40,9 +38,9 @@ def __init__(
         parent_collection: CollectionMetadata,
         child_collection: CollectionMetadata,
         always_matches: bool,
-        description: str | None,
-        synonyms: list[str] | None,
-        extra_semantic_info: dict | None,
+        description: str | None = None,
+        synonyms: list[str] | None = None,
+        extra_semantic_info: dict | None = None,
     ):
         super().__init__(
             name,
@@ -56,7 +54,7 @@ def __init__(
         )
 
     @staticmethod
-    def create_error_name(name: str, collection_error_name: str):
+    def create_error_name(name: str, collection_error_name: str) -> str:
         return f"cartesian property {name!r} of {collection_error_name}"
 
     @property
@@ -108,19 +106,6 @@ def parse_from_json(
         if "always matches" in property_json:
             always_matches = extract_bool(property_json, "always matches", error_name)
 
-        # Extract the optional fields from the JSON object.
-        description: str | None = None
-        synonyms: list[str] | None = None
-        extra_semantic_info: dict | None = None
-        if "description" in property_json:
-            description = extract_string(property_json, "description", error_name)
-        if "synonyms" in property_json:
-            synonyms = extract_array(property_json, "synonyms", error_name)
-        if "extra semantic info" in property_json:
-            extra_semantic_info = extract_object(
-                property_json, "extra semantic info", error_name
-            )
-
         NoExtraKeys(CartesianProductMetadata.allowed_fields).verify(
             property_json, error_name
         )
@@ -131,10 +116,9 @@ def parse_from_json(
             parent_collection,
             child_collection,
             always_matches,
-            description,
-            synonyms,
-            extra_semantic_info,
         )
+        # Parse the optional common semantic properties like the description.
+        property.parse_optional_properties(property_json)
         parent_collection.add_property(property)
 
     def build_reverse_relationship(

pydough/metadata/properties/general_join_metadata.py

@@ -9,9 +9,7 @@
 from pydough.metadata.collections import CollectionMetadata
 from pydough.metadata.errors import (
     NoExtraKeys,
-    extract_array,
     extract_bool,
-    extract_object,
     extract_string,
 )
 from pydough.metadata.graphs import GraphMetadata
@@ -48,9 +46,9 @@ def __init__(
         condition: str,
         self_name: str,
         other_name: str,
-        description: str | None,
-        synonyms: list[str] | None,
-        extra_semantic_info: dict | None,
+        description: str | None = None,
+        synonyms: list[str] | None = None,
+        extra_semantic_info: dict | None = None,
     ):
         super().__init__(
             name,
@@ -99,7 +97,7 @@ def components(self) -> list:
         return comp
 
     @staticmethod
-    def create_error_name(name: str, collection_error_name: str):
+    def create_error_name(name: str, collection_error_name: str) -> str:
         return f"general join property {name!r} of {collection_error_name}"
 
     @staticmethod
@@ -155,18 +153,6 @@ def parse_from_json(
             always_matches = extract_bool(property_json, "always matches", error_name)
         condition = extract_string(property_json, "condition", error_name)
 
-        # Extract the optional fields from the JSON object.
-        description: str | None = None
-        synonyms: list[str] | None = None
-        extra_semantic_info: dict | None = None
-        if "description" in property_json:
-            description = extract_string(property_json, "description", error_name)
-        if "synonyms" in property_json:
-            synonyms = extract_array(property_json, "synonyms", error_name)
-        if "extra semantic info" in property_json:
-            extra_semantic_info = extract_object(
-                property_json, "extra semantic info", error_name
-            )
         NoExtraKeys(GeneralJoinMetadata.allowed_fields).verify(
             property_json, error_name
         )
@@ -182,10 +168,9 @@ def parse_from_json(
             condition,
             "self",
             "other",
-            description,
-            synonyms,
-            extra_semantic_info,
         )
+        # Parse the optional common semantic properties like the description.
+        property.parse_optional_properties(property_json)
         parent_collection.add_property(property)
 
     def build_reverse_relationship(