Skip to content

Commit 6e6e425

Browse files
stefankandicMaxGekk
authored andcommitted
[SPARK-51468][SQL] Revert "From json/xml should not change collations in the given schema"
### What changes were proposed in this pull request? After removing session-level collation (#49772) we can also revert the PR that changed the behavior of `from_json` and `from_xml` expressions to use json and not sql type representation under the hood (#48750). ### Why are the changes needed? Now that we don't have correctness problems with session level collation, using `sql` instead of `json` will lead to smaller and more efficient type representation. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing unit tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50234 from stefankandic/revertFromJsonChange. Authored-by: Stefan Kandic <stefan.kandic@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com> (cherry picked from commit 0094f44) Signed-off-by: Max Gekk <max.gekk@gmail.com>
1 parent d5bcdd4 commit 6e6e425

File tree

5 files changed

+4
-4
lines changed

5 files changed

+4
-4
lines changed

sql/api/src/main/scala/org/apache/spark/sql/functions.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6901,7 +6901,7 @@ object functions {
69016901
*/
69026902
// scalastyle:on line.size.limit
69036903
def from_json(e: Column, schema: DataType, options: Map[String, String]): Column = {
6904-
from_json(e, lit(schema.json), options.iterator)
6904+
from_json(e, lit(schema.sql), options.iterator)
69056905
}
69066906

69076907
// scalastyle:off line.size.limit
@@ -7773,7 +7773,7 @@ object functions {
77737773
*/
77747774
// scalastyle:on line.size.limit
77757775
def from_xml(e: Column, schema: StructType, options: java.util.Map[String, String]): Column =
7776-
from_xml(e, lit(schema.json), options.asScala.iterator)
7776+
from_xml(e, lit(schema.sql), options.asScala.iterator)
77777777

77787778
// scalastyle:off line.size.limit
77797779
/**

sql/connect/common/src/test/resources/query-tests/queries/function_from_json.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
}
3838
}, {
3939
"literal": {
40-
"string": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"a\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"b\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}"
40+
"string": "STRUCT\u003cid: BIGINT, a: INT, b: DOUBLE\u003e"
4141
},
4242
"common": {
4343
"origin": {
Binary file not shown.

sql/connect/common/src/test/resources/query-tests/queries/function_from_xml.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
}
3838
}, {
3939
"literal": {
40-
"string": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"a\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"b\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}"
40+
"string": "STRUCT\u003cid: BIGINT, a: INT, b: DOUBLE\u003e"
4141
},
4242
"common": {
4343
"origin": {
Binary file not shown.

0 commit comments

Comments
 (0)