Skip to content

SNOW-807303 Use SELECT * EXCLUDE for DataFrame.drop()#3316

Merged
sfc-gh-aalam merged 18 commits intomainfrom
aalam-SNOW-807303-select-exclude
May 7, 2025
Merged

SNOW-807303 Use SELECT * EXCLUDE for DataFrame.drop()#3316
sfc-gh-aalam merged 18 commits intomainfrom
aalam-SNOW-807303-select-exclude

Conversation

@sfc-gh-aalam
Copy link
Copy Markdown
Contributor

@sfc-gh-aalam sfc-gh-aalam commented Apr 29, 2025

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-807303

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
    • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines
  3. Please describe how your code solves the related issue.

    This PR adds the ability to generate sql using SELECT * EXCLUDE (exclude_cols...) with DataFrame.drop API call.

@sfc-gh-snowflakedb-snyk-sa
Copy link
Copy Markdown

sfc-gh-snowflakedb-snyk-sa commented Apr 29, 2025

🎉 Snyk checks have passed. No issues have been found so far.

security/snyk check is complete. No issues have been found. (View Details)

license/snyk check is complete. No issues have been found. (View Details)

@github-actions github-actions bot added the local testing Local Testing issues/PRs label May 2, 2025
@sfc-gh-aalam sfc-gh-aalam marked this pull request as ready for review May 2, 2025 22:58
@sfc-gh-aalam sfc-gh-aalam requested review from a team as code owners May 2, 2025 22:58
@sfc-gh-aalam sfc-gh-aalam reopened this May 5, 2025
@github-actions github-actions bot locked and limited conversation to collaborators May 5, 2025
raise SnowparkClientExceptionMessages.DF_CANNOT_DROP_ALL_COLUMNS()

if self._select_statement and self._session.conf.get(
"use_simplified_query_generation"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there other functionality that will use this flag?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. all new query generation improvement for dataframe APIs are protected with this parameter. .distinct(), .random_split(), .stat.sample_by(), .drop() and planning to add .rename() under this.

)

normalized_names = {quote_name(n) for n in names}
existing_names = [attr.name for attr in self._output]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to not check existing names if using exclude? It will trigger a describe query

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed offline, if we don't do this describe query, it may lead to a BCR since SELECT * EXCLUDE (non_existing_column) raises error SQL error while it raises no error today.

sfc-gh-aalam and others added 3 commits May 7, 2025 13:33
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-aalam sfc-gh-aalam merged commit df33fc9 into main May 7, 2025
39 of 40 checks passed
@sfc-gh-aalam sfc-gh-aalam deleted the aalam-SNOW-807303-select-exclude branch May 7, 2025 23:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

local testing Local Testing issues/PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants