Skip to content

Commit 37fe307

Browse files
committed
Merge branch 'main' of github.com:snowflakedb/snowpark-python into yixie-SNOW-2203826-flatten-filter-sort
2 parents 4635131 + c5fdf7a commit 37fe307

File tree

184 files changed

+5180
-1408
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

184 files changed

+5180
-1408
lines changed

CHANGELOG.md

Lines changed: 51 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,53 @@
11
# Release History
22

3-
## 1.41.0 (YYYY-MM-DD)
3+
## 1.42.0 (YYYY-MM-DD)
4+
5+
### Snowpark Python API Updates
6+
7+
#### New Features
8+
9+
- Added support for `Session.client_telemetry`.
10+
- Added support for `Session.udf_profiler`.
11+
12+
#### Improvements
13+
14+
- Enhanced `DataFrame.sort()` to support `ORDER BY ALL` when no columns are specified.
15+
- Catalog API now uses SQL commands instead of SnowAPI calls. This new implementation is more reliable now.
16+
17+
#### Dependency Updates
18+
19+
- Catalog API no longer uses types declared in `snowflake.core` and therefore this dependency was removed.
20+
21+
### Snowpark pandas API Updates
22+
23+
#### New Features
24+
25+
- Added support for `Dataframe.groupby.rolling()`.
26+
- Added support for mapping `np.percentile` with DataFrame and Series inputs to `Series.quantile`.
27+
- Added support for setting the `random_state` parameter to an integer when calling `DataFrame.sample` or `Series.sample`.
28+
29+
#### Bug Fixes
30+
31+
- Fixed a bug in `DataFrameGroupBy.agg` where func is a list of tuples used to set the names of the output columns.
32+
33+
#### Improvements
34+
35+
- Add support for the following in faster pandas:
36+
- `groupby.nunique`
37+
- `groupby.size`
38+
- `concat`
39+
- `copy`
40+
- `str.isdigit`
41+
- `str.islower`
42+
- `str.isupper`
43+
- `str.istitle`
44+
- `str.lower`
45+
- `str.upper`
46+
- `str.title`
47+
48+
- Make faster pandas disabled by default (opt-in instead of opt-out).
49+
50+
## 1.41.0 (2025-10-23)
451

552
### Snowpark Python API Updates
653

@@ -57,6 +104,9 @@
57104
- `st_geometryfromwkt`
58105
- `try_to_geography`
59106
- `try_to_geometry`
107+
108+
#### Improvements
109+
60110
- Added a parameter to enable and disable automatic column name aliasing for `interval_day_time_from_parts` and `interval_year_month_from_parts` functions.
61111

62112
#### Bug Fixes
@@ -67,7 +117,6 @@
67117
- Fixed a bug that `DataFrameReader.dbapi` (PuPr) is not compatible with oracledb 3.4.0.
68118
- Fixed a bug where `modin` would unintentionally be imported during session initialization in some scenarios.
69119
- Fixed a bug where `session.udf|udtf|udaf|sproc.register` failed when an extra session argument was passed. These methods do not expect a session argument; please remove it if provided.
70-
- Fixed a bug in `DataFrameGroupBuy.agg` where func is a list of tuples used to set the names of the output columns.
71120

72121
#### Improvements
73122

@@ -153,9 +202,6 @@
153202
- `groupby.median`
154203
- `groupby.std`
155204
- `groupby.var`
156-
- `groupby.nunique`
157-
- `groupby.size`
158-
- `groupby.apply`
159205
- `drop_duplicates`
160206
- Reuse row count from the relaxed query compiler in `get_axis_len`.
161207

docs/source/modin/numpy.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,12 @@ NumPy ufuncs called with Snowpark pandas arguments will ignore kwargs.
3737
| ``np.full_like`` | Mapped to pd.DataFrame(value, index=range(height), |
3838
| | columns=range(width)) |
3939
+-----------------------------+----------------------------------------------------+
40+
| ``np.percentile`` | Mapped to Series.quantile, will stack a DataFrame |
41+
| | to convert to Series. Always returns an ndarray or |
42+
| | scalar like np.percentile. Does not implement any |
43+
| | arguments other than the input array and |
44+
| | percentage(s). |
45+
+-----------------------------+----------------------------------------------------+
4046
| ``np.may_share_memory`` | Returns False |
4147
+-----------------------------+----------------------------------------------------+
4248
| ``np.abs`` | Mapped to df.abs() |

docs/source/modin/supported/dataframe_supported.rst

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -384,8 +384,13 @@ Methods
384384
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
385385
| ``rtruediv`` | P | ``level`` | |
386386
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
387-
| ``sample`` | P | | ``N`` if ``weights`` or ``random_state`` is |
388-
| | | | specified when ``axis = 0`` |
387+
| ``sample`` | P | | ``N`` if ``weights`` is specified when |
388+
| | | | ``axis = 0``, or if ``random_state`` is not |
389+
| | | | either an integer or ``None``. Setting |
390+
| | | | ``random_state`` to a value other than ``None`` |
391+
| | | | may slow down this method because the ``sample`` |
392+
| | | | implementation will use a sort instead of the |
393+
| | | | Snowflake warehouse's built-in SAMPLE construct. |
389394
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
390395
| ``select_dtypes`` | Y | | |
391396
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+

docs/source/modin/supported/groupby_supported.rst

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,10 @@ Computations/descriptive stats
153153
| | | will be lost. ``rule`` frequencies 's', 'min', |
154154
| | | 'h', and 'D' are supported. |
155155
+-----------------------------+---------------------------------+----------------------------------------------------+
156-
| ``rolling`` | N | |
156+
| ``rolling`` | P | Implemented for DataframeGroupby objects. ``N`` for|
157+
| | | ``on``, non-integer ``window``, ``axis = 1``, |
158+
| | | ``method`` != ``single``, ``min_periods = 0``, or |
159+
| | | ``closed`` != ``None``. |
157160
+-----------------------------+---------------------------------+----------------------------------------------------+
158161
| ``sample`` | N | |
159162
+-----------------------------+---------------------------------+----------------------------------------------------+

docs/source/modin/supported/series_supported.rst

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -383,8 +383,13 @@ Methods
383383
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
384384
| ``rtruediv`` | P | ``level`` | See ``truediv`` |
385385
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
386-
| ``sample`` | P | | ``N`` if ``weights`` or ``random_state`` is |
387-
| | | | specified when ``axis = 0`` |
386+
| ``sample`` | P | | ``N`` if ``weights`` is specified when |
387+
| | | | ``axis = 0``, or if ``random_state`` is not |
388+
| | | | either an integer or ``None``. Setting |
389+
| | | | ``random_state`` to a value other than ``None`` |
390+
| | | | may slow down this method because the ``sample`` |
391+
| | | | implementation will use a sort instead of the |
392+
| | | | Snowflake warehouse's built-in SAMPLE construct. |
388393
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
389394
| ``searchsorted`` | N | | |
390395
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+

recipe/meta.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% set name = "snowflake-snowpark-python" %}
2-
{% set version = "1.40.0" %}
2+
{% set version = "1.41.0" %}
33
{% set noarch_build = (os.environ.get('SNOWFLAKE_SNOWPARK_PYTHON_NOARCH_BUILD', 'false')) == 'true' %}
44
{% set build_number = os.environ.get('SNOWFLAKE_SNOWPARK_PYTHON_BUILD_NUMBER', 0) %}
55

setup.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,6 @@
6363
"pytest-assume", # sql counter check
6464
"decorator", # sql counter check
6565
"tox", # used for setting up testing environments
66-
"snowflake.core>=1.0.0, <2", # Catalog
6766
"psutil", # testing for telemetry
6867
"lxml", # used in XML reader unit tests
6968
]
@@ -231,6 +230,7 @@ def run(self):
231230
"opentelemetry": [
232231
"opentelemetry-api>=1.0.0, <2.0.0",
233232
"opentelemetry-sdk>=1.0.0, <2.0.0",
233+
"opentelemetry-exporter-otlp>=1.0.0, <2.0.0",
234234
],
235235
},
236236
classifiers=[

src/snowflake/snowpark/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
"QueryListener",
4141
"AsyncJob",
4242
"StoredProcedureProfiler",
43+
"UDFProfiler",
4344
]
4445

4546

@@ -54,6 +55,7 @@
5455
from snowflake.snowpark.async_job import AsyncJob
5556
from snowflake.snowpark.column import CaseExpr, Column
5657
from snowflake.snowpark.stored_procedure_profiler import StoredProcedureProfiler
58+
from snowflake.snowpark.udf_profiler import UDFProfiler
5759
from snowflake.snowpark.dataframe import DataFrame
5860
from snowflake.snowpark.dataframe_ai_functions import DataFrameAIFunctions
5961
from snowflake.snowpark.dataframe_analytics_functions import DataFrameAnalyticsFunctions
Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
#
2+
# Copyright (c) 2012-2025 Snowflake Computing Inc. All rights reserved.
3+
#
4+
5+
import keyword
6+
import json
7+
8+
from collections.abc import Mapping
9+
10+
from typing import (
11+
Any,
12+
Dict,
13+
Iterable,
14+
Iterator,
15+
Tuple,
16+
Optional,
17+
Union,
18+
)
19+
20+
21+
class ImmutableAttrDict(Mapping):
22+
"""
23+
An immutable mapping whose (identifier-valid, non-keyword, non-private) keys
24+
are also available as read-only attributes. Nested mappings are recursively
25+
wrapped as ImmutableAttrDict instances.
26+
27+
By default also recursively freezes:
28+
- dict / Mapping -> ImmutableAttrDict
29+
- list / tuple -> tuple
30+
- set -> frozenset
31+
- other types -> unchanged
32+
33+
Access:
34+
d.foo (if 'foo' is a valid identifier key)
35+
d['foo'] (always works)
36+
37+
Invalid / reserved keys (not identifiers, keywords, or starting with '_')
38+
are still accessible via item lookup but not as attributes.
39+
"""
40+
41+
__slots__ = ("_data", "_frozen")
42+
43+
# ------------- Construction / conversion helpers -----------------
44+
@classmethod
45+
def _convert(cls, value: Any) -> Any:
46+
"""Recursively convert nested structures."""
47+
if isinstance(value, cls):
48+
return value
49+
if isinstance(value, Mapping):
50+
# Ensure plain dict form for predictability
51+
return cls(value)
52+
if isinstance(value, list) or isinstance(value, tuple):
53+
return tuple(cls._convert(v) for v in value)
54+
if isinstance(value, set):
55+
return frozenset(cls._convert(v) for v in value)
56+
# For other iterables you could add handling if desired.
57+
return value
58+
59+
@staticmethod
60+
def _is_exposable_attr_name(name: str) -> bool:
61+
"""Return True if name can safely be set as an attribute."""
62+
return bool(
63+
name
64+
and name[0] != "_"
65+
and name.isidentifier()
66+
and not keyword.iskeyword(name)
67+
)
68+
69+
def __init__(
70+
self,
71+
mapping: Optional[Union[Mapping[str, Any], Iterable[Tuple[str, Any]]]] = None,
72+
**kwargs: Any,
73+
) -> None:
74+
combined: Dict[str, Any] = {}
75+
if mapping is not None:
76+
if isinstance(mapping, Mapping):
77+
combined.update(mapping)
78+
else:
79+
for k, v in mapping:
80+
combined[k] = v
81+
combined.update(kwargs)
82+
83+
# Deep-convert values first
84+
converted: Dict[str, Any] = {k: self._convert(v) for k, v in combined.items()}
85+
86+
object.__setattr__(self, "_data", converted)
87+
88+
# Expose attribute names only for "safe" keys
89+
for k, v in converted.items():
90+
if self._is_exposable_attr_name(k):
91+
object.__setattr__(self, k, v)
92+
93+
object.__setattr__(self, "_frozen", True)
94+
95+
# ------------- Factory methods -----------------
96+
@classmethod
97+
def from_json(cls, j: str) -> "ImmutableAttrDict":
98+
parsed = json.loads(j)
99+
if not isinstance(parsed, Mapping):
100+
raise TypeError("JSON root must be an object to build ImmutableAttrDict")
101+
return cls(parsed)
102+
103+
# ------------- Mapping interface -----------------
104+
def __getitem__(self, key: str) -> Any:
105+
return self._data[key]
106+
107+
def __iter__(self) -> Iterator[str]:
108+
return iter(self._data)
109+
110+
def __len__(self) -> int:
111+
return len(self._data)
112+
113+
def __contains__(self, key: object) -> bool:
114+
return key in self._data
115+
116+
def get(self, key: str, default: Any = None) -> Any:
117+
return self._data.get(key, default)
118+
119+
# ------------- Attribute fallback -----------------
120+
def __getattr__(self, name: str) -> Any:
121+
data = object.__getattribute__(self, "_data")
122+
if name in data and self._is_exposable_attr_name(name):
123+
return data[name]
124+
raise AttributeError(
125+
f"{type(self).__name__!r} object has no attribute {name!r}"
126+
)
127+
128+
# ------------- Immutability enforcement -----------------
129+
def __setattr__(self, name: str, value: Any) -> None:
130+
if name.startswith("_"):
131+
object.__setattr__(self, name, value)
132+
return
133+
if object.__getattribute__(self, "_frozen"):
134+
raise AttributeError(
135+
f"{type(self).__name__} is immutable; cannot set attribute {name!r}"
136+
)
137+
object.__setattr__(self, name, value)
138+
139+
def __delattr__(self, name: str) -> None:
140+
if name.startswith("_"):
141+
object.__delattr__(self, name)
142+
return
143+
if object.__getattribute__(self, "_frozen"):
144+
raise AttributeError(
145+
f"{type(self).__name__} is immutable; cannot delete attribute {name!r}"
146+
)
147+
object.__delattr__(self, name)
148+
149+
# ------------- Utilities -----------------
150+
def to_dict(self) -> Dict[str, Any]:
151+
"""
152+
Return a shallow copy of the underlying storage.
153+
Nested ImmutableAttrDict instances are preserved (not converted).
154+
For a fully "plain" structure you can use to_plain().
155+
"""
156+
return dict(self._data)
157+
158+
def to_plain(self) -> Dict[str, Any]:
159+
"""
160+
Recursively convert back to plain Python types:
161+
ImmutableAttrDict -> dict
162+
tuple/frozenset -> list (or list for frozenset)
163+
"""
164+
165+
def unwrap(v: Any) -> Any:
166+
if isinstance(v, ImmutableAttrDict):
167+
return {k: unwrap(v._data[k]) for k in v._data}
168+
if isinstance(v, tuple):
169+
return [unwrap(x) for x in v]
170+
if isinstance(v, frozenset):
171+
return [unwrap(x) for x in v]
172+
return v
173+
174+
return {k: unwrap(v) for k, v in self._data.items()}
175+
176+
def __repr__(self) -> str:
177+
return f"{type(self).__name__}({self._data})"
178+
179+
def __reduce__(self):
180+
return (type(self), (self._data,))

src/snowflake/snowpark/_internal/analyzer/analyzer.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,10 @@
114114
SnowflakeTable,
115115
SnowflakeValues,
116116
)
117-
from snowflake.snowpark._internal.analyzer.sort_expression import SortOrder
117+
from snowflake.snowpark._internal.analyzer.sort_expression import (
118+
SortOrder,
119+
SortByAllOrder,
120+
)
118121
from snowflake.snowpark._internal.analyzer.table_function import (
119122
FlattenFunction,
120123
GeneratorTableFunction,
@@ -558,6 +561,13 @@ def analyze(
558561
expr.null_ordering.sql,
559562
)
560563

564+
if isinstance(expr, SortByAllOrder):
565+
return order_expression(
566+
"ALL",
567+
expr.direction.sql,
568+
expr.null_ordering.sql,
569+
)
570+
561571
if isinstance(expr, ScalarSubquery):
562572
self.subquery_plans.append(expr.plan)
563573
return subquery_expression(expr.plan.queries[-1].sql)

0 commit comments

Comments
 (0)