Skip to content

Spark Legacy Configuration #12

@satyasyamnn

Description

@satyasyamnn

When this code is run using Spark version 3.1.2 I got an error

Caused by: org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2065-10-6' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.
at org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$$anonfun$checkParsedDiff$1.applyOrElse(DateTimeFormatterHelper.scala:150)

The fix for this issue is to add the following lines of code

    SparkConf conf = new SparkConf().setAppName("Spark In Action").setMaster("local[*]");
    conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY");

To make it work in Spark 3 what kind of formatting is required?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions