Skip to content

[SPARK-53786][SQL] Default value with special column name should not conflict with real column #52504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+228 −0
Open

[SPARK-53786][SQL] Default value with special column name should not conflict with real column #52504

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
da190b3
[SPARK-53786][SQL] Default value should not conflict with special col…
szehon-ho Oct 1, 2025
ad8f55e
Add more tests
szehon-ho Oct 2, 2025
5bb76b4
Fix another case where special column is in the function
szehon-ho Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ...talyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,14 @@ trait ColumnResolutionHelper extends Logging with DataTypeErrorsBase {
u.copy(child = newChild)
}

case d @ DefaultValueExpression(u: UnresolvedAttribute, _, _) =>
Copy link
Member Author

@szehon-ho szehon-ho Oct 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: before this fix, Default value expression would fall to UnresolvedAttribute above. It would then think the default value refers to the conflicting column name and fail.

case u @ UnresolvedAttribute(nameParts) =>
          val result = withPosition(u) {
            resolveColumnByName(nameParts)
              .orElse(LiteralFunctionResolution.resolve(nameParts))
              .map {
                // We trim unnecessary alias here. Note that, we cannot trim the alias at top-level,
                // as we should resolve `UnresolvedAttribute` to a named expression. The caller side
                // can trim the top-level alias if it's safe to do so. Since we will call
                // CleanupAliases later in Analyzer, trim non top-level unnecessary alias is safe.
                case Alias(child, _) if !isTopLevel => child
                case other => other
              }
              .getOrElse(u)
          }
          logDebug(s"Resolving $u to $result")
          result

d.copy(child = LiteralFunctionResolution.resolve(u.nameParts)
.map {
case Alias(child, _) if !isTopLevel => child
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied this from the other code, we dont need it right?

case other => other
}
.getOrElse(u))

case _ => e.mapChildren(innerResolve(_, isTopLevel = false))
}
resolved.copyTagsFrom(e)
Expand Down
11 changes: 11 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3870,6 +3870,17 @@ class DataSourceV2SQLSuiteV1Filter
}
}

test("test default value conflicting with special column name") {
val t = "testcat.ns.t"
withTable("t") {
sql(s"""CREATE table $t (
c1 STRING,
c2 TIMESTAMP,
c3 BOOLEAN DEFAULT FALSE,
current_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP)""")
}
}

private def testNotSupportedV2Command(
sqlCommand: String,
sqlParams: String,
Expand Down