-
Notifications
You must be signed in to change notification settings - Fork 22
Open
Description
Line 110 in 1362aa6
| to_date(df2.col("releaseDateAsString"), "yyyy-MM-dd")) |
When this code is run using Spark version 3.1.2 I got an error
Caused by: org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2065-10-6' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.
at org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$$anonfun$checkParsedDiff$1.applyOrElse(DateTimeFormatterHelper.scala:150)
The fix for this issue is to add the following lines of code
SparkConf conf = new SparkConf().setAppName("Spark In Action").setMaster("local[*]");
conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY");
To make it work in Spark 3 what kind of formatting is required?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels