SNOW-807303 Use SELECT * EXCLUDE for DataFrame.drop()#3316
SNOW-807303 Use SELECT * EXCLUDE for DataFrame.drop()#3316sfc-gh-aalam merged 18 commits intomainfrom
Conversation
🎉 Snyk checks have passed. No issues have been found so far.✅ security/snyk check is complete. No issues have been found. (View Details) ✅ license/snyk check is complete. No issues have been found. (View Details) |
| raise SnowparkClientExceptionMessages.DF_CANNOT_DROP_ALL_COLUMNS() | ||
|
|
||
| if self._select_statement and self._session.conf.get( | ||
| "use_simplified_query_generation" |
There was a problem hiding this comment.
is there other functionality that will use this flag?
There was a problem hiding this comment.
yeah. all new query generation improvement for dataframe APIs are protected with this parameter. .distinct(), .random_split(), .stat.sample_by(), .drop() and planning to add .rename() under this.
| ) | ||
|
|
||
| normalized_names = {quote_name(n) for n in names} | ||
| existing_names = [attr.name for attr in self._output] |
There was a problem hiding this comment.
is it possible to not check existing names if using exclude? It will trigger a describe query
There was a problem hiding this comment.
as discussed offline, if we don't do this describe query, it may lead to a BCR since SELECT * EXCLUDE (non_existing_column) raises error SQL error while it raises no error today.
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-807303
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
This PR adds the ability to generate sql using
SELECT * EXCLUDE (exclude_cols...)withDataFrame.dropAPI call.