Create pydough from_string API (#383)

The from_string API parses a PyDough source code string and transforms it into a PyDough collection. You can then perform operations like explain(), to_sql(), or to_df() on the result.

Author: juankx-bodo

documentation/usage.md

@@ -13,11 +13,13 @@ This document describes how to set up & interact with PyDough. For instructions
 - [Evaluation APIs](#evaluation-apis)
    * [`pydough.to_sql`](#pydoughto_sql)
    * [`pydough.to_df`](#pydoughto_df)
+- [Transformation APIs](#transformation-apis)
+   * [`pydough.from_string`](#pydoughfrom_string)
 - [Exploration APIs](#exploration-apis)
    * [`pydough.explain_structure`](#pydoughexplain_structure)
    * [`pydough.explain`](#pydoughexplain)
    * [`pydough.explain_term`](#pydoughexplain_term)
-- [Logging] (#logging)
+- [Logging](#logging)
 
 <!-- TOC end -->
 
@@ -476,6 +478,187 @@ pydough.to_df(result, columns={"name": "name", "n_custs": "n"})
 
 See the [demo notebooks](../demos/notebooks/1_introduction.ipynb) for more instances of how to use the `to_df` API.
 
+<!-- TOC --><a name="evaluation-apis"></a>
+## Transformation APIs
+
+This sections describes various APIs you can use to transform PyDough source code into a result that can be used as input for other evaluation or exploration APIs.
+
+<!-- TOC --><a name="pydoughfrom_string"></a>
+### `pydough.from_string`
+
+The `from_string` API parses a PyDough source code string and transforms it into a PyDough collection. You can then perform operations like `explain()`, `to_sql()`, or `to_df()` on the result.
+
+#### Syntax
+```python
+def from_string(
+    source: str,
+    answer_variable: str | None = None,
+    metadata: GraphMetadata | None = None,
+    environment: dict[str, Any] | None = None,
+) -> UnqualifiedNode:
+```
+
+The first argument `source` is the source code string. It can be a single pydough command or a multi-line pydough code with intermediate results stored in variables. It can optionally take in the following keyword arguments:
+
+- `answer_variable`: The name of the variable that stores the final result of the PyDough code. If not provided, the API expects the final result to be in a variable named `result`. The API returns a PyDough collection holding this value. It is assumed that the PyDough code string includes a variable definition where the name of the variable is the same as `answer_variable` and the value is valid PyDough code; if not it raises an exception.
+- `metadata`: The PyDough knowledge graph to use for the transformation. If omitted, `pydough.active_session.metadata` is used.
+- `environment`: A dictionary representing additional environment context. This serves as the local namespace where the PyDough code will be executed.
+
+Below are examples of using `pydough.from_string`, and examples of the SQL that could be potentially generated from calling `pydough.to_sql` on the output. All these examples use the TPC-H dataset that can be downloaded [here](https://github.com/lovasoa/TPCH-sqlite/releases) with the [graph used in the demos directory](../demos/metadata/tpch_demo_graph.json).
+ 
+This first example is of Python code using `pydough.from_string` to generate SQL to get the count of customers in the market segment `"AUTOMOBILE"`. The result will be returned in a variable named `pydough_query` instead of the default `result`, and the market segment `"AUTOMOBILE"` is passed in an environment variable `SEG`.:
+```py
+import pydough
+
+# Setup demo metadata. Make sure you have the TPC-H dataset downloaded locally.
+graph = pydough.active_session.load_metadata_graph("demos/metadata/tpch_demo_graph.json", "TPCH")
+pydough.active_session.connect_database("sqlite", database="tpch.db")
+
+# Example of a single line pydough code snippet
+pydough_code = "pydough_query = TPCH.CALCULATE(n=COUNT(customers.WHERE(market_segment == SEG)))"
+# Transform the pydough code and get the result from pydough_query
+query = pydough.from_string(pydough_code, "pydough_query", graph, {"SEG":"AUTOMOBILE"})
+sql = pydough.to_sql(query)
+```
+
+The value of `sql` is the following SQL query text as a Python string:
+```sql
+SELECT
+  COUNT(*) AS n
+FROM main.customer
+WHERE
+  c_mktsegment = 'AUTOMOBILE'
+```
+
+This next example is of Python code to generate SQL to get the top 5 suppliers with the highest revenue. The code snippet uses variables provided in the environment context to filter by nation, ship mode and year (`TARGET_NATION`, `DESIRED_SHIP_MODE` and `REQUESTED_SHIP_YEAR`):
+```py
+# Example of a multi-line pydough code snippet with intermetiate results
+nation_name = "JAPAN"
+ship_mode = "TRUCK"
+ship_year = 1996
+env = {"TARGET_NATION" : nation_name, "DESIRED_SHIP_MODE" : ship_mode, "REQUESTED_SHIP_YEAR" : ship_year}
+
+pydough_code="""
+# The supply records for the supplier that were from a medium part
+selected_records = supply_records.WHERE(STARTSWITH(part.name, "coral")).CALCULATE(supply_cost)
+
+# The revenue generated by a specific lineitem
+line_revenue = extended_price * (1 - discount) * (1 - tax) - quantity * supply_cost
+
+# The lineitem purchases for each record that were ordered in REQUESTED_SHIP_YEAR and shipped via DESIRED_SHIP_MODE
+lines_year = selected_records.lines.WHERE((YEAR(ship_date) == REQUESTED_SHIP_YEAR) & (ship_mode == DESIRED_SHIP_MODE)).CALCULATE(rev=line_revenue)
+
+# For each supplier, list their name & selected revenue from REQUESTED_SHIP_YEAR
+selected_suppliers = suppliers.WHERE((nation.name == TARGET_NATION) & HAS(lines_year))
+supplier_info = selected_suppliers.CALCULATE(name, revenue_year=ROUND(SUM(lines_year.rev), 2))
+
+# Pick the 5 suppliers with the highest revenue from REQUESTED_SHIP_YEAR
+result = supplier_info.TOP_K(5, by=revenue_year.DESC())
+"""
+# Transform the pydough code and get the result from result
+query = pydough.from_string(pydough_code, environment=env)
+sql = pydough.to_sql(query)
+```
+
+The value of `sql` is the following SQL query text as a Python string:
+```sql
+WITH _s7 AS (
+  SELECT
+    ROUND(
+      COALESCE(
+        SUM(
+          lineitem.l_extendedprice * (
+            1 - lineitem.l_discount
+          ) * (
+            1 - lineitem.l_tax
+          ) - lineitem.l_quantity * partsupp.ps_supplycost
+        ),
+        0
+      ),
+      2
+    ) AS revenue_year,
+    partsupp.ps_suppkey
+  FROM main.partsupp AS partsupp
+  JOIN main.part AS part
+    ON part.p_name LIKE 'coral%' AND part.p_partkey = partsupp.ps_partkey
+  JOIN main.lineitem AS lineitem
+    ON CAST(STRFTIME('%Y', lineitem.l_shipdate) AS INTEGER) = 1996
+    AND lineitem.l_partkey = partsupp.ps_partkey
+    AND lineitem.l_shipmode = 'TRUCK'
+    AND lineitem.l_suppkey = partsupp.ps_suppkey
+  GROUP BY
+    partsupp.ps_suppkey
+)
+SELECT
+  supplier.s_name AS name,
+  _s7.revenue_year
+FROM main.supplier AS supplier
+JOIN main.nation AS nation
+  ON nation.n_name = 'JAPAN' AND nation.n_nationkey = supplier.s_nationkey
+JOIN _s7 AS _s7
+  ON _s7.ps_suppkey = supplier.s_suppkey
+ORDER BY
+  revenue_year DESC
+LIMIT 5
+```
+
+This final example is of Python code to generate an SQL query, using 'datetime.date' passed in through the environment.
+```py
+# For every customer, how many urgent orders have they made in year 1996 with a 
+# total price over 100000, and what is the sum of the total prices of all such 
+# orders they made? Sort the result by the sum from highest to lowest, and only
+# include customers with at least one such order
+
+# For this query we set 1 environment variable and 1 function, the year 1996
+# and date function from datetime. Optionally, we could import datetime.date
+# from inside the pydough code string
+import datetime
+env = {"date" : datetime.date, "YEAR" : 1996}
+
+pydough_code="""
+selected_orders=orders.WHERE((order_priority == '1-URGENT') & (total_price > 100000) & 
+  (order_date >= date(YEAR, 1, 1)) & (order_date < date(YEAR + 1, 1, 1)))
+
+result = (
+  customers
+  .WHERE(HAS(selected_orders))
+  .CALCULATE(name, n_orders=COUNT(selected_orders), total=SUM(selected_orders.total_price))
+  .ORDER_BY(total.DESC())
+)
+"""
+
+# Transform the pydough code and get the result from result
+query = pydough.from_string(pydough_code, environment=env)
+sql = pydough.to_sql(query)
+```
+
+The value of `sql` is the following SQL query text as a Python string:
+```sql
+WITH _s1 AS (
+  SELECT
+    COALESCE(SUM(o_totalprice), 0) AS total,
+    COUNT(*) AS n_rows,
+    o_custkey
+  FROM main.orders
+  WHERE
+    o_orderdate < '1997-01-01'
+    AND o_orderdate >= '1996-01-01'
+    AND o_orderpriority = '1-URGENT'
+    AND o_totalprice > 100000
+  GROUP BY
+    o_custkey
+)
+SELECT
+  customer.c_name AS name,
+  _s1.n_rows AS n_orders,
+  _s1.total
+FROM main.customer AS customer
+JOIN _s1 AS _s1
+  ON _s1.o_custkey = customer.c_custkey
+ORDER BY
+  total DESC
+```
+
 <!-- TOC --><a name="exploration-apis"></a>
 ## Exploration APIs
 
@@ -882,6 +1065,8 @@ This term is singular with regards to the collection, meaning it can be placed i
 For example, the following is valid:
   TPCH.nations.WHERE(region.name == 'EUROPE').CALCULATE(AVG(customers.acctbal))
 ```
+
+<!-- TOC --><a name="logging"></a>
 ## Logging
 
 Logging is enabled and set to INFO level by default. We can change the log level by setting the environment variable `PYDOUGH_LOG_LEVEL` to the standard levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.

pydough/__init__.py

@@ -8,6 +8,7 @@
     "explain",
     "explain_structure",
     "explain_term",
+    "from_string",
     "get_logger",
     "init_pydough_context",
     "parse_json_metadata_from_file",
@@ -20,7 +21,7 @@
 from .exploration import explain, explain_structure, explain_term
 from .logger import get_logger
 from .metadata import parse_json_metadata_from_file
-from .unqualified import display_raw, init_pydough_context
+from .unqualified import display_raw, from_string, init_pydough_context
 
 # Create a default session for the user to interact with.
 # In most situations users will just use this session and

pydough/unqualified/__init__.py

@@ -20,6 +20,7 @@
     "UnqualifiedWhere",
     "UnqualifiedWindow",
     "display_raw",
+    "from_string",
     "init_pydough_context",
     "qualify_node",
     "qualify_term",
@@ -44,4 +45,9 @@
     UnqualifiedWindow,
     display_raw,
 )
-from .unqualified_transform import init_pydough_context, transform_cell, transform_code
+from .unqualified_transform import (
+    from_string,
+    init_pydough_context,
+    transform_cell,
+    transform_code,
+)

pydough/unqualified/unqualified_transform.py

@@ -3,14 +3,18 @@
 variables with unqualified nodes by prepending with with `_ROOT.`.
 """
 
-__all__ = ["init_pydough_context", "transform_cell", "transform_code"]
+__all__ = ["from_string", "init_pydough_context", "transform_cell", "transform_code"]
 
 import ast
 import inspect
 import types
+from typing import Any
 
 from pydough.metadata import GraphMetadata
 
+from .errors import PyDoughUnqualifiedException
+from .unqualified_node import UnqualifiedNode
+
 
 class AddRootVisitor(ast.NodeTransformer):
     """
@@ -413,6 +417,85 @@ def transform_cell(cell: str, graph_name: str, known_names: set[str]) -> str:
     return ast.unparse(new_tree)
 
 
+def from_string(
+    source: str,
+    answer_variable: str | None = None,
+    metadata: GraphMetadata | None = None,
+    environment: dict[str, Any] | None = None,
+) -> UnqualifiedNode:
+    """
+    Parses and transforms a PyDough source string, returning an unqualified node
+    on which operations like `explain()`, `to_sql()`, or `to_df()` can be
+    called.
+
+    Args:
+        `source`: a valid PyDough code string that will be executed to define
+        the PyDough code.
+        `answer_variable`: The name of the variable that holds the result of the
+        PyDough code. If not provided, assumes the answer is `result`.
+        `metadata`: The metadata graph to use. If not provided,
+        `active_session.metadata` will be used.
+        `environment`: A dictionary of variables that will be available
+        in the environment where the PyDough code is executed. If not provided,
+        uses an empty dictionary.
+
+    Returns:
+        A PyDough UnualifiedNode object representing the result of the
+        transformed PyDough code.
+    """
+    import pydough
+
+    # Verify if graph is provided. Otherwise use pydough.active_session.metadata
+    if metadata is None:
+        metadata = pydough.active_session.metadata
+        if metadata is None:
+            raise ValueError(
+                "No active graph set in PyDough session."
+                " Please set a graph using"
+                " pydough.active_session.load_metadata_graph(...)"
+            )
+    # Verify if environment is provided
+    if environment is None:
+        environment = {}
+
+    # Verify if answer_variable is provided
+    if answer_variable is None:
+        answer_variable = "result"
+
+    # Transform PyDough code into valid Python code
+    known_names: set[str] = set(environment.keys())
+    visitor: ast.NodeTransformer = AddRootVisitor("graph", known_names)
+    try:
+        tree: ast.AST = ast.parse(source)
+    except SyntaxError as e:
+        raise ValueError(f"Syntax error in source PyDough code:\n{str(e)}") from e
+    assert isinstance(tree, ast.AST)
+    new_tree: ast.AST = ast.fix_missing_locations(visitor.visit(tree))
+    assert isinstance(new_tree, ast.AST)
+
+    # Execute the transformed PyDough code to get the UnqualifiedNode answer
+    transformed_code: str = ast.unparse(new_tree)
+    try:
+        compile_ast = compile(transformed_code, filename="<ast>", mode="exec")
+    except SyntaxError as e:
+        raise ValueError(f"Syntax error in transformed PyDough code:\n{str(e)}") from e
+    execution_context: dict[str, Any] = environment | {"graph": metadata}
+    exec(compile_ast, {}, execution_context)
+
+    # Check if answer_variable exists in execution_context after code execution
+    if answer_variable not in execution_context:
+        raise PyDoughUnqualifiedException(
+            f"PyDough code expected to store the answer in a variable named '{answer_variable}'."
+        )
+    ret_val = execution_context[answer_variable]
+    # Check if answer is an UnqualifiedNode
+    if not isinstance(ret_val, UnqualifiedNode):
+        raise PyDoughUnqualifiedException(
+            f"Expected variable {answer_variable!r} in the text to store PyDough code, instead found {ret_val.__class__.__name__!r}."
+        )
+    return ret_val
+
+
 def init_pydough_context(graph: GraphMetadata):
     """
     Decorator that wraps around a PyDough function and transforms its body into