merge

Author: hadia206

documentation/functions.md

@@ -27,6 +27,7 @@ Below is the list of every function/operator currently supported in PyDough as a
    * [STRIP](#strip)
    * [REPLACE](#replace)
    * [STRCOUNT](#strcount)
+   * [GETPART](#getpart)
 - [Datetime Functions](#datetime-functions)
    * [DATETIME](#datetime)
    * [YEAR](#year)
@@ -438,6 +439,46 @@ Customers.CALCULATE(count_substring= STRCOUNT(name, "")) # returns 0 by default
 | `'Alex Rodriguez'`| `STRCOUNT('Alex Rodriguez', 'e')`| `2`    |
 | `'Hello World'`| `STRCOUNT('Hello World', 'll')`          | `1`          |
 
+
+<!-- TOC --><a name="getpart"></a>
+
+### GETPART
+
+The `GETPART` function extracts the N-th part from a string, splitting it by a specified delimiter.
+
+- The first argument is the input string to split.
+- The second argument is the delimiter string.
+- The third argument is the index of the part to extract. This index can be positive (counting from the start, 0-based) or negative (counting from the end, -1 is the last part).
+
+If the index is out of range, `GETPART` returns `None`. If the delimiter is an empty string, the function will not split the input string and the first part will be the entire string.
+
+```py
+# Extracts the first name from a full name
+Customers.CALCULATE(first_name = GETPART(name, " ", 1))
+
+# Extracts the last name from a full name
+Customers.CALCULATE(last_name = GETPART(name, " ", -1))
+
+# Extracts the second part from a hyphen-separated string
+Parts.CALCULATE(second_code = GETPART(code, "-", 2))
+```
+
+| **Input String**      | **Delimiter** | **Index** | **GETPART Result** |
+|---------------------- |-------------- |-----------|--------------------|
+| `"Alex Rodriguez"`    | `" "`         | `1`       | `"Alex"`           |
+| `"Alex Rodriguez"`    | `" "`         | `0`       | `"Alex"`           |
+| `"Alex Rodriguez"`    | `" "`         | `2`       | `"Rodriguez"`      |
+| `"Alex Rodriguez"`    | `" "`         | `-1`      | `"Rodriguez"`      |
+| `"Alex Rodriguez"`    | `""`          | `1`       | `"Alex Rodriguez"` |
+| `"a-b-c-d"`           | `"-"`         | `3`       | `"c"`              |
+| `"a-b-c-d"`           | `"-"`         | `-2`      | `"c"`              |
+| `"a-b-c-d"`           | `"-"`         | `5`       | `None`             |
+| `"a-b-c-d"`           | `"-"`         | `-5`      | `None`             |
+
+> [!NOTE]
+> - Indexing is one-based from the start and negative indices count from the end.
+> - The 0 index will be treated as 1, returning the first part.
+
 <!-- TOC --><a name="datetime-functions"></a>
 
 ## Datetime Functions

documentation/metadata.md

@@ -13,11 +13,13 @@ This document describes how to set up & interact with PyDough. For instructions
 - [Evaluation APIs](#evaluation-apis)
    * [`pydough.to_sql`](#pydoughto_sql)
    * [`pydough.to_df`](#pydoughto_df)
+- [Transformation APIs](#transformation-apis)
+   * [`pydough.from_string`](#pydoughfrom_string)
 - [Exploration APIs](#exploration-apis)
    * [`pydough.explain_structure`](#pydoughexplain_structure)
    * [`pydough.explain`](#pydoughexplain)
    * [`pydough.explain_term`](#pydoughexplain_term)
-- [Logging] (#logging)
+- [Logging](#logging)
 
 <!-- TOC end -->
 
@@ -476,6 +478,187 @@ pydough.to_df(result, columns={"name": "name", "n_custs": "n"})
 
 See the [demo notebooks](../demos/notebooks/1_introduction.ipynb) for more instances of how to use the `to_df` API.
 
+<!-- TOC --><a name="evaluation-apis"></a>
+## Transformation APIs
+
+This sections describes various APIs you can use to transform PyDough source code into a result that can be used as input for other evaluation or exploration APIs.
+
+<!-- TOC --><a name="pydoughfrom_string"></a>
+### `pydough.from_string`
+
+The `from_string` API parses a PyDough source code string and transforms it into a PyDough collection. You can then perform operations like `explain()`, `to_sql()`, or `to_df()` on the result.
+
+#### Syntax
+```python
+def from_string(
+    source: str,
+    answer_variable: str | None = None,
+    metadata: GraphMetadata | None = None,
+    environment: dict[str, Any] | None = None,
+) -> UnqualifiedNode:
+```
+
+The first argument `source` is the source code string. It can be a single pydough command or a multi-line pydough code with intermediate results stored in variables. It can optionally take in the following keyword arguments:
+
+- `answer_variable`: The name of the variable that stores the final result of the PyDough code. If not provided, the API expects the final result to be in a variable named `result`. The API returns a PyDough collection holding this value. It is assumed that the PyDough code string includes a variable definition where the name of the variable is the same as `answer_variable` and the value is valid PyDough code; if not it raises an exception.
+- `metadata`: The PyDough knowledge graph to use for the transformation. If omitted, `pydough.active_session.metadata` is used.
+- `environment`: A dictionary representing additional environment context. This serves as the local namespace where the PyDough code will be executed.
+
+Below are examples of using `pydough.from_string`, and examples of the SQL that could be potentially generated from calling `pydough.to_sql` on the output. All these examples use the TPC-H dataset that can be downloaded [here](https://github.com/lovasoa/TPCH-sqlite/releases) with the [graph used in the demos directory](../demos/metadata/tpch_demo_graph.json).
+ 
+This first example is of Python code using `pydough.from_string` to generate SQL to get the count of customers in the market segment `"AUTOMOBILE"`. The result will be returned in a variable named `pydough_query` instead of the default `result`, and the market segment `"AUTOMOBILE"` is passed in an environment variable `SEG`.:
+```py
+import pydough
+
+# Setup demo metadata. Make sure you have the TPC-H dataset downloaded locally.
+graph = pydough.active_session.load_metadata_graph("demos/metadata/tpch_demo_graph.json", "TPCH")
+pydough.active_session.connect_database("sqlite", database="tpch.db")
+
+# Example of a single line pydough code snippet
+pydough_code = "pydough_query = TPCH.CALCULATE(n=COUNT(customers.WHERE(market_segment == SEG)))"
+# Transform the pydough code and get the result from pydough_query
+query = pydough.from_string(pydough_code, "pydough_query", graph, {"SEG":"AUTOMOBILE"})
+sql = pydough.to_sql(query)
+```
+
+The value of `sql` is the following SQL query text as a Python string:
+```sql
+SELECT
+  COUNT(*) AS n
+FROM main.customer
+WHERE
+  c_mktsegment = 'AUTOMOBILE'
+```
+
+This next example is of Python code to generate SQL to get the top 5 suppliers with the highest revenue. The code snippet uses variables provided in the environment context to filter by nation, ship mode and year (`TARGET_NATION`, `DESIRED_SHIP_MODE` and `REQUESTED_SHIP_YEAR`):
+```py
+# Example of a multi-line pydough code snippet with intermetiate results
+nation_name = "JAPAN"
+ship_mode = "TRUCK"
+ship_year = 1996
+env = {"TARGET_NATION" : nation_name, "DESIRED_SHIP_MODE" : ship_mode, "REQUESTED_SHIP_YEAR" : ship_year}
+
+pydough_code="""
+# The supply records for the supplier that were from a medium part
+selected_records = supply_records.WHERE(STARTSWITH(part.name, "coral")).CALCULATE(supply_cost)
+
+# The revenue generated by a specific lineitem
+line_revenue = extended_price * (1 - discount) * (1 - tax) - quantity * supply_cost
+
+# The lineitem purchases for each record that were ordered in REQUESTED_SHIP_YEAR and shipped via DESIRED_SHIP_MODE
+lines_year = selected_records.lines.WHERE((YEAR(ship_date) == REQUESTED_SHIP_YEAR) & (ship_mode == DESIRED_SHIP_MODE)).CALCULATE(rev=line_revenue)
+
+# For each supplier, list their name & selected revenue from REQUESTED_SHIP_YEAR
+selected_suppliers = suppliers.WHERE((nation.name == TARGET_NATION) & HAS(lines_year))
+supplier_info = selected_suppliers.CALCULATE(name, revenue_year=ROUND(SUM(lines_year.rev), 2))
+
+# Pick the 5 suppliers with the highest revenue from REQUESTED_SHIP_YEAR
+result = supplier_info.TOP_K(5, by=revenue_year.DESC())
+"""
+# Transform the pydough code and get the result from result
+query = pydough.from_string(pydough_code, environment=env)
+sql = pydough.to_sql(query)
+```
+
+The value of `sql` is the following SQL query text as a Python string:
+```sql
+WITH _s7 AS (
+  SELECT
+    ROUND(
+      COALESCE(
+        SUM(
+          lineitem.l_extendedprice * (
+            1 - lineitem.l_discount
+          ) * (
+            1 - lineitem.l_tax
+          ) - lineitem.l_quantity * partsupp.ps_supplycost
+        ),
+        0
+      ),
+      2
+    ) AS revenue_year,
+    partsupp.ps_suppkey
+  FROM main.partsupp AS partsupp
+  JOIN main.part AS part
+    ON part.p_name LIKE 'coral%' AND part.p_partkey = partsupp.ps_partkey
+  JOIN main.lineitem AS lineitem
+    ON CAST(STRFTIME('%Y', lineitem.l_shipdate) AS INTEGER) = 1996
+    AND lineitem.l_partkey = partsupp.ps_partkey
+    AND lineitem.l_shipmode = 'TRUCK'
+    AND lineitem.l_suppkey = partsupp.ps_suppkey
+  GROUP BY
+    partsupp.ps_suppkey
+)
+SELECT
+  supplier.s_name AS name,
+  _s7.revenue_year
+FROM main.supplier AS supplier
+JOIN main.nation AS nation
+  ON nation.n_name = 'JAPAN' AND nation.n_nationkey = supplier.s_nationkey
+JOIN _s7 AS _s7
+  ON _s7.ps_suppkey = supplier.s_suppkey
+ORDER BY
+  revenue_year DESC
+LIMIT 5
+```
+
+This final example is of Python code to generate an SQL query, using 'datetime.date' passed in through the environment.
+```py
+# For every customer, how many urgent orders have they made in year 1996 with a 
+# total price over 100000, and what is the sum of the total prices of all such 
+# orders they made? Sort the result by the sum from highest to lowest, and only
+# include customers with at least one such order
+
+# For this query we set 1 environment variable and 1 function, the year 1996
+# and date function from datetime. Optionally, we could import datetime.date
+# from inside the pydough code string
+import datetime
+env = {"date" : datetime.date, "YEAR" : 1996}
+
+pydough_code="""
+selected_orders=orders.WHERE((order_priority == '1-URGENT') & (total_price > 100000) & 
+  (order_date >= date(YEAR, 1, 1)) & (order_date < date(YEAR + 1, 1, 1)))
+
+result = (
+  customers
+  .WHERE(HAS(selected_orders))
+  .CALCULATE(name, n_orders=COUNT(selected_orders), total=SUM(selected_orders.total_price))
+  .ORDER_BY(total.DESC())
+)
+"""
+
+# Transform the pydough code and get the result from result
+query = pydough.from_string(pydough_code, environment=env)
+sql = pydough.to_sql(query)
+```
+
+The value of `sql` is the following SQL query text as a Python string:
+```sql
+WITH _s1 AS (
+  SELECT
+    COALESCE(SUM(o_totalprice), 0) AS total,
+    COUNT(*) AS n_rows,
+    o_custkey
+  FROM main.orders
+  WHERE
+    o_orderdate < '1997-01-01'
+    AND o_orderdate >= '1996-01-01'
+    AND o_orderpriority = '1-URGENT'
+    AND o_totalprice > 100000
+  GROUP BY
+    o_custkey
+)
+SELECT
+  customer.c_name AS name,
+  _s1.n_rows AS n_orders,
+  _s1.total
+FROM main.customer AS customer
+JOIN _s1 AS _s1
+  ON _s1.o_custkey = customer.c_custkey
+ORDER BY
+  total DESC
+```
+
 <!-- TOC --><a name="exploration-apis"></a>
 ## Exploration APIs
 
@@ -882,6 +1065,8 @@ This term is singular with regards to the collection, meaning it can be placed i
 For example, the following is valid:
   TPCH.nations.WHERE(region.name == 'EUROPE').CALCULATE(AVG(customers.acctbal))
 ```
+
+<!-- TOC --><a name="logging"></a>
 ## Logging
 
 Logging is enabled and set to INFO level by default. We can change the log level by setting the environment variable `PYDOUGH_LOG_LEVEL` to the standard levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.

documentation/usage.md

@@ -8,6 +8,7 @@
     "explain",
     "explain_structure",
     "explain_term",
+    "from_string",
     "get_logger",
     "init_pydough_context",
     "parse_json_metadata_from_file",
@@ -21,7 +22,7 @@
 from .exploration import explain, explain_structure, explain_term
 from .logger import get_logger
 from .metadata import parse_json_metadata_from_file
-from .unqualified import display_raw, init_pydough_context
+from .unqualified import display_raw, from_string, init_pydough_context
 from .user_collections import range_collection
 
 # Create a default session for the user to interact with.

pydough/__init__.py

@@ -54,7 +54,7 @@ def metadata(self) -> GraphMetadata | None:
         Get the active metadata graph.
 
         Returns:
-            GraphMetadata: The active metadata graph.
+            The active metadata graph.
         """
         return self._metadata
 
@@ -64,7 +64,7 @@ def metadata(self, graph: GraphMetadata | None) -> None:
         Set the active metadata graph.
 
         Args:
-            graph (GraphMetadata | None): The metadata graph to set.
+            graph: The metadata graph to set.
         """
         self._metadata = graph
 
@@ -74,7 +74,7 @@ def config(self) -> PyDoughConfigs:
         Get the active PyDough configuration.
 
         Returns:
-            PyDoughConfigs: The active PyDough configuration.
+            The active PyDough configuration.
         """
         return self._config
 
@@ -84,7 +84,7 @@ def config(self, config: PyDoughConfigs) -> None:
         Set the active PyDough configuration.
 
         Args:
-            config (PyDoughConfigs): The PyDough configuration to set.
+            `config`: The PyDough configuration to set.
         """
         self._config = config
 
@@ -94,7 +94,7 @@ def database(self) -> DatabaseContext:
         Get the active database context.
 
         Returns:
-            DatabaseContext: The active database context.
+            The active database context.
         """
         return self._database
 
@@ -104,7 +104,7 @@ def database(self, context: DatabaseContext) -> None:
         Set the active database context.
 
         Args:
-            context (DatabaseContext): The database context to set.
+            `context`: The database context to set.
         """
         self._database = context
 
@@ -114,13 +114,13 @@ def connect_database(self, database_name: str, **kwargs) -> DatabaseContext:
         the corresponding context in case the user wants/needs to modify it.
 
         Args:
-            database_name (str): The name of the database to connect to.
+            `database_name`: The name of the database to connect to.
             **kwargs: Additional keyword arguments to pass to the connection.
                 All arguments must be accepted using the supported connect API
                 for the dialect. Most likely the database path will be required.
 
         Returns:
-            DatabaseContext: The newly created database context.
+            The newly created database context.
         """
         context: DatabaseContext = load_database_context(database_name, **kwargs)
         self.database = context
@@ -135,13 +135,13 @@ def load_metadata_graph(self, graph_path: str, graph_name: str) -> GraphMetadata
         property directly later.
 
         Args:
-            graph_path (str): The path to load the graph. At this time this must be on
+            `graph_path`: The path to load the graph. At this time this must be on
                 the user's local file system.
-            graph_name (str): The name under which to load the graph from the file. This
+            `graph_name`: The name under which to load the graph from the file. This
                 is to allow loading multiple graphs from the same json file.
 
         Returns:
-            GraphMetadata: The loaded metadata graph.
+            The loaded metadata graph.
         """
         graph: GraphMetadata = parse_json_metadata_from_file(graph_path, graph_name)
         self.metadata = graph