Merge branch 'main' into helmeleegy-SNOW-2432713

Author: sfc-gh-helmeleegy

CHANGELOG.md

@@ -7,7 +7,6 @@
 #### New Features
 
 - Added a new function `service` in `snowflake.snowpark.functions` that allows users to create a callable representing a Snowpark Container Services (SPCS) service.
-- Added a new function `group_by_all()` to the `DataFrame` class.
 - Added `connection_parameters` parameter to `DataFrameReader.dbapi()` (PuPr) method to allow passing keyword arguments to the `create_connection` callable.
 - Added support for `Session.begin_transaction`, `Session.commit` and `Session.rollback`.
 - Added support for the following functions in `functions.py`:
@@ -67,6 +66,8 @@
 - Fixed a bug where writing Snowpark pandas dataframes on the pandas backend with a column multiindex to Snowflake with `to_snowflake` would raise `KeyError`.
 - Fixed a bug that `DataFrameReader.dbapi` (PuPr) is not compatible with oracledb 3.4.0.
 - Fixed a bug where `modin` would unintentionally be imported during session initialization in some scenarios.
+- Fixed a bug where `session.udf|udtf|udaf|sproc.register` failed when an extra session argument was passed. These methods do not expect a session argument; please remove it if provided.
+- Fixed a bug in `DataFrameGroupBuy.agg` where func is a list of tuples used to set the names of the output columns.
 
 #### Improvements
 
@@ -83,6 +84,7 @@
 - Added support for the `dtypes` parameter of `pd.get_dummies`
 - Added support for `nunique` in `df.pivot_table`, `df.agg` and other places where aggregate functions can be used.
 - Added support for `DataFrame.interpolate` and `Series.interpolate` with the "linear", "ffill"/"pad", and "backfill"/bfill" methods. These use the SQL `INTERPOLATE_LINEAR`, `INTERPOLATE_FFILL`, and `INTERPOLATE_BFILL` functions (PuPr).
+- Added support for `Dataframe.groupby.rolling()`.
 
 #### Improvements
 
@@ -93,6 +95,16 @@
   - `skew()` with `axis=1` or `numeric_only=False` parameters
   - `round()` with `decimals` parameter as a Series
   - `corr()` with `method!=pearson` parameter
+  - `shift()` with `suffix` or non-integer `periods` parameters
+  - `sort_index()` with `axis=1` or `key` parameters
+  - `sort_values()` with `axis=1`
+  - `melt()` with `col_level` parameter
+  - `apply()` with `result_type` parameter for DataFrame
+  - `pivot_table()` with `sort=True`, non-string `index` list, non-string `columns` list, non-string `values` list, or `aggfunc` dict with non-string values
+  - `fillna()` with `downcast` parameter or using `limit` together with `value`
+  - `dropna()` with `axis=1`
+
+
 - Set `cte_optimization_enabled` to True for all Snowpark pandas sessions.
 - Add support for the following in faster pandas:
   - `isin`
@@ -127,6 +139,7 @@
   - `sort_values`
   - `loc` (setting columns)
   - `to_datetime`
+  - `rename`
   - `drop`
   - `invert`
   - `duplicated`
@@ -151,6 +164,9 @@
   - `groupby.median`
   - `groupby.std`
   - `groupby.var`
+  - `groupby.nunique`
+  - `groupby.size`
+  - `groupby.apply`
   - `drop_duplicates`
 - Reuse row count from the relaxed query compiler in `get_axis_len`.
 

docs/source/modin/supported/groupby_supported.rst

@@ -153,7 +153,10 @@ Computations/descriptive stats
 |                             |                                 | will be lost. ``rule`` frequencies 's', 'min',     |
 |                             |                                 | 'h', and 'D' are supported.                        |
 +-----------------------------+---------------------------------+----------------------------------------------------+
-| ``rolling``                 | N                               |                                                    |
+| ``rolling``                 | P                               | Implemented for DataframeGroupby objects. ``N`` for|
+|                             |                                 | ``on``, non-integer ``window``, ``axis = 1``,      |
+|                             |                                 | ``method`` != ``single``, ``min_periods = 0``, or  |
+|                             |                                 | ``closed``  != ``None``.                           |
 +-----------------------------+---------------------------------+----------------------------------------------------+
 | ``sample``                  | N                               |                                                    |
 +-----------------------------+---------------------------------+----------------------------------------------------+

docs/source/snowpark/dataframe.rst

@@ -57,7 +57,6 @@ DataFrame
     DataFrame.flatten
     DataFrame.groupBy
     DataFrame.group_by
-    DataFrame.group_by_all
     DataFrame.group_by_grouping_sets
     DataFrame.intersect
     DataFrame.join

setup.py

@@ -68,7 +68,10 @@
     "lxml",  # used in XML reader unit tests
 ]
 MODIN_DEVELOPMENT_REQUIREMENTS = [
-    "scipy",  # Snowpark pandas 3rd party library testing
+    # Snowpark pandas 3rd party library testing. Cap the scipy version because
+    # Snowflake cannot find newer versions of scipy for python 3.11+. See
+    # SNOW-2452791.
+    "scipy<=1.16.0",
     "statsmodels",  # Snowpark pandas 3rd party library testing
     "scikit-learn",  # Snowpark pandas 3rd party library testing
     # plotly version restricted due to foreseen change in query counts in version 6.0.0+

src/snowflake/snowpark/_internal/analyzer/analyzer.py

@@ -91,7 +91,6 @@
 from snowflake.snowpark._internal.analyzer.grouping_set import (
     GroupingSet,
     GroupingSetsExpression,
-    GroupByAll,
 )
 from snowflake.snowpark._internal.analyzer.select_statement import (
     Selectable,
@@ -347,8 +346,6 @@ def analyze(
             )
 
         if isinstance(expr, GroupingSet):
-            if isinstance(expr, GroupByAll):
-                return "ALL"
             return self.grouping_extractor(expr, df_aliased_col_name_to_real_col_name)
 
         if isinstance(expr, WindowExpression):

src/snowflake/snowpark/_internal/analyzer/grouping_set.py

@@ -40,10 +40,6 @@ class Rollup(GroupingSet):
     pass
 
 
-class GroupByAll(GroupingSet):
-    pass
-
-
 class GroupingSetsExpression(Expression):
     def __init__(self, args: List[List[Expression]]) -> None:
         super().__init__()