SNOW-2220946: Add support for unstructured data engineering in Snowpark (#3775)

Author: sfc-gh-jdu

CHANGELOG.md

@@ -1,11 +1,34 @@
 # Release History
 
+## 1.40.0 (YYYY-MM-DD)
+
+### Snowpark Python API Updates
+
+#### New Features
+
 ## 1.39.0 (YYYY-MM-DD)
 
 ### Snowpark Python API Updates
 
 #### New Features
 
+- Added support for unstructured data engineering in Snowpark, powered by Snowflake AISQL and Cortex functions:
+  - `DataFrame.ai.complete`: Generate per-row LLM completions from prompts built over columns and files.
+  - `DataFrame.ai.filter`: Keep rows where an AI classifier returns TRUE for the given predicate.
+  - `DataFrame.ai.agg`: Reduce a text column into one result using a natural-language task description.
+  - `RelationalGroupedDataFrame.ai_agg`: Perform the same natural-language aggregation per group.
+  - `DataFrame.ai.classify`: Assign single or multiple labels from given categories to text or images.
+  - `DataFrame.ai.similarity`: Compute cosine-based similarity scores between two columns via embeddings.
+  - `DataFrame.ai.sentiment`: Extract overall and aspect-level sentiment from text into JSON.
+  - `DataFrame.ai.embed`: Generate VECTOR embeddings for text or images using configurable models.
+  - `DataFrame.ai.summarize_agg`: Aggregate and produce a single comprehensive summary over many rows.
+  - `DataFrame.ai.transcribe`: Transcribe audio files to text with optional timestamps and speaker labels.
+  - `DataFrame.ai.parse_document`: OCR/layout-parse documents or images into structured JSON.
+  - `DataFrame.ai.extract`: Pull structured fields from text or files using a response schema.
+  - `DataFrame.ai.count_tokens`: Estimate token usage for a given model and input text per row.
+  - `DataFrame.ai.split_text_markdown_header`: Split Markdown into hierarchical header-aware chunks.
+  - `DataFrame.ai.split_text_recursive_character`: Split text into size-bounded chunks using recursive separators.
+  - `DataFrameReader.file`: Create a DataFrame containing all files from a stage as FILE data type for downstream unstructured data processing.
 - Added a new datatype `YearMonthIntervalType` that allows users to create intervals for datetime operations.
 - Added a new function `interval_year_month_from_parts` that allows users to easily create `YearMonthIntervalType` without using SQL.
 - Added a new datatype `DayTimeIntervalType` that allows users to create intervals for datetime operations.

docs/source/snowpark/dataframe.rst

@@ -13,6 +13,7 @@ DataFrame
     DataFrameNaFunctions
     DataFrameStatFunctions
     DataFrameAnalyticsFunctions
+    DataFrameAIFunctions
 
 .. rubric:: Methods
 
@@ -120,6 +121,20 @@ DataFrame
     DataFrameAnalyticsFunctions.compute_lag
     DataFrameAnalyticsFunctions.compute_lead
     DataFrameAnalyticsFunctions.time_series_agg
+    DataFrameAIFunctions.agg
+    DataFrameAIFunctions.classify
+    DataFrameAIFunctions.complete
+    DataFrameAIFunctions.count_tokens
+    DataFrameAIFunctions.embed
+    DataFrameAIFunctions.extract
+    DataFrameAIFunctions.filter
+    DataFrameAIFunctions.parse_document
+    DataFrameAIFunctions.sentiment
+    DataFrameAIFunctions.similarity
+    DataFrameAIFunctions.split_text_markdown_header
+    DataFrameAIFunctions.split_text_recursive_character
+    DataFrameAIFunctions.summarize_agg
+    DataFrameAIFunctions.transcribe
     dataframe.map
     dataframe.map_in_pandas
 
@@ -133,6 +148,7 @@ DataFrame
 .. autosummary::
     :toctree: api/
 
+    DataFrame.ai
     DataFrame.columns
     DataFrame.na
     DataFrame.queries

docs/source/snowpark/grouping.rst

@@ -18,6 +18,7 @@ Grouping
     :toctree: api/
 
     RelationalGroupedDataFrame.agg
+    RelationalGroupedDataFrame.ai_agg
     RelationalGroupedDataFrame.apply_in_pandas
     RelationalGroupedDataFrame.applyInPandas
     RelationalGroupedDataFrame.avg

src/snowflake/snowpark/__init__.py

@@ -22,6 +22,7 @@
     "DataFrameStatFunctions",
     "DataFrameAnalyticsFunctions",
     "DataFrameNaFunctions",
+    "DataFrameAIFunctions",
     "DataFrameWriter",
     "DataFrameReader",
     "GroupingSets",
@@ -54,6 +55,7 @@
 from snowflake.snowpark.column import CaseExpr, Column
 from snowflake.snowpark.stored_procedure_profiler import StoredProcedureProfiler
 from snowflake.snowpark.dataframe import DataFrame
+from snowflake.snowpark.dataframe_ai_functions import DataFrameAIFunctions
 from snowflake.snowpark.dataframe_analytics_functions import DataFrameAnalyticsFunctions
 from snowflake.snowpark.dataframe_na_functions import DataFrameNaFunctions
 from snowflake.snowpark.dataframe_reader import DataFrameReader