Skip to content

Commit d8dfc8e

Browse files
author
Mariam-Almesfer
committed
Add validation test for CurrentTimestamp
1 parent 527a72f commit d8dfc8e

File tree

2 files changed

+47
-3
lines changed

2 files changed

+47
-3
lines changed

backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1521,4 +1521,22 @@ abstract class ScalarFunctionsValidateSuite extends FunctionsValidateSuite {
15211521
}
15221522
}
15231523
}
1524+
1525+
test("current_timestamp") {
1526+
withSQLConf("spark.sql.optimizer.excludedRules" -> "") {
1527+
val df = spark.sql("SELECT current_timestamp()")
1528+
1529+
val optimizedPlan = df.queryExecution.optimizedPlan.toString()
1530+
assert(
1531+
optimizedPlan.contains("Project") && !optimizedPlan.contains("CurrentTimestamp"),
1532+
s"Expected CurrentTimestamp to be folded to a literal, but got: $optimizedPlan"
1533+
)
1534+
1535+
checkFallbackOperators(df, 1)
1536+
1537+
val result = df.collect()
1538+
assert(result.length == 1, "Expected exactly one row")
1539+
assert(result(0).getTimestamp(0) != null, "Expected a non-null timestamp")
1540+
}
1541+
}
15241542
}

docs/get-started/VeloxABFS.md

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,15 +8,41 @@ ABFS is an important data store for big data users. This doc discusses config de
88

99
# Working with ABFS
1010

11-
## Configuring ABFS Access Token
11+
## Configuring ABFS Authentication Type
1212

13-
To configure access to your storage account, replace <storage-account> with the name of your account. This property aligns with Spark configurations. By setting this config multiple times using different storage account names, you can access multiple ABFS accounts.
13+
The authentication mechanism for an Azure storage account is controlled by the following property. Replace `<storage-account>` with the name of your Azure Storage account. Defaults to `SharedKey` if not set.
14+
15+
```sh
16+
spark.hadoop.fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net SharedKey
17+
```
18+
19+
Allowed values are `SharedKey`, `OAuth`, and `SAS`.
20+
21+
## Configuring ABFS Access Key
22+
23+
To configure access to your storage account using a shared key, replace `<storage-account>` with the name of your account. This property aligns with Spark configurations. By setting this config multiple times using different storage account names, you can access multiple ABFS accounts.
1424

1525
```sh
1626
spark.hadoop.fs.azure.account.key.<storage-account>.dfs.core.windows.net XXXXXXXXX
1727
```
1828

19-
### Other authentatication methods are not yet supported.
29+
## Configuring ABFS SAS Token
30+
31+
To authenticate using a pre-generated SAS (Shared Access Signature) token, set the following property. This token provides scoped and time-limited access to specific resources.
32+
33+
```sh
34+
spark.hadoop.fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net XXXXXXXXX
35+
```
36+
37+
## Configuring ABFS OAuth
38+
39+
To authenticate using OAuth 2.0, set the following properties. Replace `<storage-account>` with the name of your Azure Storage account and `<tenant-id>` with your Azure AD tenant ID.
40+
41+
```sh
42+
spark.hadoop.fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net XXXXXXXXX
43+
spark.hadoop.fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net XXXXXXXXX
44+
spark.hadoop.fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net https://login.microsoftonline.com/<tenant-id>/oauth2/token
45+
```
2046

2147
# Local Caching support
2248

0 commit comments

Comments
 (0)