Skip to content

[GLUTEN-10950][VL][FOLLOWUP] Fix missing time unit in hadoop-aws time configuration (pre-3.4 versions) #11329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zml1206 merged 9 commits into from
Jan 10, 2026
Merged

[GLUTEN-10950][VL][FOLLOWUP] Fix missing time unit in hadoop-aws time configuration (pre-3.4 versions) #11329

Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d7f51db
[GLUTEN-10950][VL][FOLLOWUP] Add time unit to 'spark.hadoop.fs.s3a.co…
xumanbu Dec 24, 2025
0420514
add s3aTimeConfigs for config
Jan 8, 2026
d7622e7
remove spark.hadoop.fs.s3a.connection.ttl & change spark.hadoop.fs.s3…
Jan 8, 2026
78e0796
update Reference
Jan 8, 2026
68f184e
update Reference
Jan 8, 2026
652494d
Revert "update Reference"
Jan 8, 2026
d2fd551
fix codestyle
Jan 8, 2026
b874d47
add scalastyle:off for url comment
Jan 8, 2026
abcebc1
remove configValue null check
Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxTransformerApi.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,23 @@ class VeloxTransformerApi extends TransformerApi with Logging {
override def postProcessNativeConfig(
nativeConfMap: JMap[String, String],
backendPrefix: String): Unit = {
// 'spark.hadoop.fs.s3a.connection.timeout' by velox requires time unit, hadoop-aws versions
// before 3.4 do not have time unit.
val s3sConnectionTimeout = nativeConfMap.get("spark.hadoop.fs.s3a.connection.timeout")
if (NumberUtils.isCreatable(s3sConnectionTimeout)) {
nativeConfMap.put("spark.hadoop.fs.s3a.connection.timeout", s"${s3sConnectionTimeout}ms")
// S3A configurations that require time units for Velox.
// Hadoop-aws versions before 3.3.6 do not include time units by default.
Copy link
Contributor

@zml1206 zml1206 Jan 8, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why modify it? It should be before version 3.4. Suggestion:
Some configurations in Hadoop-aws versions before 3.4 do not include time units.

Copy link
Contributor Author

@xumanbu xumanbu Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my mistake.

// Reference: https://hadoop.apache.org/docs/r3.3.6/hadoop-aws/tools/hadoop-aws/#General_S3A_Client_configuration
val s3aTimeConfigs = Map(
"spark.hadoop.fs.s3a.connection.timeout" -> "ms",
"spark.hadoop.fs.s3a.connection.establish.timeout" -> "ms",
"spark.hadoop.fs.s3a.threads.keepalivetime" -> "s",
"spark.hadoop.fs.s3a.multipart.purge.age" -> "s"
)

s3aTimeConfigs.foreach {
case (configKey, defaultUnit) =>
val configValue = nativeConfMap.get(configKey)
if (configValue != null && NumberUtils.isCreatable(configValue)) {
Copy link
Contributor

@zml1206 zml1206 Jan 9, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configValue != null is redundant, NumberUtils.isCreatable contains null checks.

// Config is numeric (no unit), append the default unit for backward compatibility
nativeConfMap.put(configKey, s"$configValue$defaultUnit")
}
}
}

Expand Down