Skip to content

SNOW-2097586: Add Debug mode for eager evaluation#3380

Merged
sfc-gh-jrose merged 15 commits intomainfrom
jrose_snow_2097586_debug_mode
Jun 20, 2025
Merged

SNOW-2097586: Add Debug mode for eager evaluation#3380
sfc-gh-jrose merged 15 commits intomainfrom
jrose_snow_2097586_debug_mode

Conversation

@sfc-gh-jrose
Copy link
Contributor

@sfc-gh-jrose sfc-gh-jrose commented May 20, 2025

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-2097586

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
    • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines
  3. Please describe how your code solves the related issue.

    This PR adds a new debug mode to sessions. Currently the only difference in debug mode is that dataframe attributes are populated on Dataframe creation instead of lazily, but in the future other features may be added. This change is done as a separate mode in order to avoid the additional describe queries for production pipelines.

The side effect of this change is that errors related to the dataframe schema become apparent on the line of code that caused them rather than on whatever line evaluates the dataframe. I've also modified the exception wrapper for column identifiers to not modify the error message if no additional context is available.

Here's an example of how the user experience changes:

import snowflake.snowpark.functions as f

df = session.create_dataframe(
    [
        ("cat", "1"),
        ("dog", "1"),
        ("cactus", "2"),
        ("coffee", "3"),
    ],
    schema=["object", "type"]
)

category_lookup = session.create_dataframe(
    [
        (1, "Animal"),
        (2, "Plant"),
        (3, "Unknown")],
    schema=["type", "category"]
)

normalized = df.select("OBJECT", f.cast(df.TYPE, IntegerType()))
filtered = normalized.select("OBJECT", "TYPE").where(f.col("TYPE") < 3)
joined = filtered.join(category_lookup, on="TYPE")

# snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01bc80ed-0e10-2a2a-000c-a909ed766ceb: 000904 (42000): SQL compilation error: error line 1 at position 157
# invalid identifier 'TYPE'
# There are existing quoted column identifiers: []. Please use one of them to reference the column. See more details on Snowflake identifier requirements https://docs.snowflake.com/en/sql-reference/identifiers-syntax

import snowflake.snowpark.context as context
context.enable_debug_mode()

normalized = df.select("OBJECT", f.cast(df.TYPE, IntegerType()))
filtered = normalized.select("OBJECT", "TYPE").where(f.col("TYPE") < 3)
# snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01bc80ef-0e10-28f2-000c-a909ed7901ff: 000904 (42000): SQL compilation error: error line 1 at position 17
# invalid identifier 'TYPE'
# There are existing quoted column identifiers: ['"OBJECT"', '"CAST (""TYPE"" AS INT)"']. Please use one of them to reference the column. See more details on Snowflake identifier requirements https://docs.snowflake.com/en/sql-reference/identifiers-syntax

@sfc-gh-snowflakedb-snyk-sa
Copy link

sfc-gh-snowflakedb-snyk-sa commented May 20, 2025

🎉 Snyk checks have passed. No issues have been found so far.

security/snyk check is complete. No issues have been found. (View Details)

license/snyk check is complete. No issues have been found. (View Details)

@sfc-gh-jrose sfc-gh-jrose requested a review from a team May 21, 2025 14:38
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
@sfc-gh-jrose sfc-gh-jrose merged commit 01b1dbf into main Jun 20, 2025
92 of 95 checks passed
@sfc-gh-jrose sfc-gh-jrose deleted the jrose_snow_2097586_debug_mode branch June 20, 2025 20:15
@github-actions github-actions bot locked and limited conversation to collaborators Jun 20, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants