SNOW-1843881: Change StructType columns to return Row objects#2820
Merged
sfc-gh-jrose merged 27 commits intomainfrom Jan 8, 2025
Merged
SNOW-1843881: Change StructType columns to return Row objects#2820sfc-gh-jrose merged 27 commits intomainfrom
sfc-gh-jrose merged 27 commits intomainfrom
Conversation
) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1852779 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Fixed AST encoding for Column `in_`, `asc`, and `desc`. Removed an unused entity and renamed `sp_column_in__seq` to `sp_column_in`. Changed the `nulls_first` optional boolean param to be an enum.
sfc-gh-jrose
added
the
NO-CHANGELOG-UPDATES
sfc-gh-aalam
approved these changes
Jan 3, 2025
Comment on lines
+705
to
+706
| ) -> Tuple[Optional[List[str]], Optional[List[Callable]]]: | ||
| if not result_meta: |
Contributor
There was a problem hiding this comment.
can we add a small description of what this returns.
sfc-gh-jrose
requested review from
sfc-gh-aling,
sfc-gh-jdu and
sfc-gh-smirzaei
and removed request for
a team
sfc-gh-aling
reviewed
Jan 4, 2025
Comment on lines
-879
to
+880
| ({"x": 1}, {"A": "a", "b": 1}, [1, 1, 1]), | ||
| ({"x": 2}, {"A": "b", "b": 2}, [2, 2, 2]), | ||
| ({"x": 1}, Row(A="a", b=1), [1, 1, 1]), | ||
| ({"x": 2}, Row(A="b", b=2), [2, 2, 2]), |
Contributor
There was a problem hiding this comment.
is this considered a breaking change or this feature is not rolled out yet?
Contributor
Author
There was a problem hiding this comment.
Yes, this is a breaking change. It only gets applied when the feature flag in snowflake.snowpark.context is set though so won't break current users until that flag is rolled out. The structured type tests have the flag enabled.
Contributor
There was a problem hiding this comment.
I assume that flag is _should_use_structured_type_semantics.
just for my understanding, we will do a BCR release when rolling out this parameter?
Contributor
Author
There was a problem hiding this comment.
That is the correct flag. I believe we will release the change in a BCR release, but for now this unblocks sas and customers that want to enabled it if we do PrPr
sfc-gh-aling
approved these changes
Jan 4, 2025
Base automatically changed from
jrose_snow_1865926_create_dataframe_default_structured
to
main
January 7, 2025 18:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1843881
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
In pyspark struct type objects are collected as Rows instead of dicts. Another recent change allowed creating struct type columns by creating a dataframe from nested Row objects. These two changes together allow round trip structured objects without manually specifying the schema.