Skip to content

Commit 21f0b73

Browse files
seancxmaogatorsmile
authored andcommitted
[SPARK-25453][SQL][TEST][.FFFFFFFFF] OracleIntegrationSuite IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss
## What changes were proposed in this pull request? This PR aims to fix the failed test of `OracleIntegrationSuite`. ## How was this patch tested? Existing integration tests. Closes apache#22461 from seancxmao/SPARK-25453. Authored-by: seancxmao <[email protected]> Signed-off-by: gatorsmile <[email protected]>
1 parent fb8f4c0 commit 21f0b73

File tree

2 files changed

+10
-1
lines changed

2 files changed

+10
-1
lines changed

docs/sql-programming-guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1502,7 +1502,7 @@ See the [Apache Avro Data Source Guide](avro-data-source-guide.html).
15021502

15031503
* The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java's DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs.
15041504
* Some databases, such as H2, convert all names to upper case. You'll need to use upper case to refer to those names in Spark SQL.
1505-
1505+
* Users can specify vendor-specific JDBC connection properties in the data source options to do special treatment. For example, `spark.read.format("jdbc").option("url", oracleJdbcUrl).option("oracle.jdbc.mapDateToTimestamp", "false")`. `oracle.jdbc.mapDateToTimestamp` defaults to true, users often need to disable this flag to avoid Oracle date being resolved as timestamp.
15061506

15071507
# Performance Tuning
15081508

external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -442,6 +442,12 @@ class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSQLCo
442442
.option("lowerBound", "2018-07-06")
443443
.option("upperBound", "2018-07-20")
444444
.option("numPartitions", 3)
445+
// oracle.jdbc.mapDateToTimestamp defaults to true. If this flag is not disabled, column d
446+
// (Oracle DATE) will be resolved as Catalyst Timestamp, which will fail bound evaluation of
447+
// the partition column. E.g. 2018-07-06 cannot be evaluated as Timestamp, and the error
448+
// message says: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff].
449+
.option("oracle.jdbc.mapDateToTimestamp", "false")
450+
.option("sessionInitStatement", "ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY-MM-DD'")
445451
.load()
446452

447453
df1.logicalPlan match {
@@ -462,6 +468,9 @@ class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSQLCo
462468
.option("lowerBound", "2018-07-04 03:30:00.0")
463469
.option("upperBound", "2018-07-27 14:11:05.0")
464470
.option("numPartitions", 2)
471+
.option("oracle.jdbc.mapDateToTimestamp", "false")
472+
.option("sessionInitStatement",
473+
"ALTER SESSION SET NLS_TIMESTAMP_FORMAT = 'YYYY-MM-DD HH24:MI:SS.FF'")
465474
.load()
466475

467476
df2.logicalPlan match {

0 commit comments

Comments
 (0)