Skip to content

Commit b6ba0dd

Browse files
bersprocketscloud-fan
authored andcommitted
[DOC] Fix doc for spark.sql.parquet.recordLevelFilter.enabled
## What changes were proposed in this pull request? Updated the doc string value for spark.sql.parquet.recordLevelFilter.enabled to indicate that spark.sql.parquet.enableVectorizedReader must be disabled. The code in ParquetFileFormat uses spark.sql.parquet.recordLevelFilter.enabled only after falling back to parquet-mr (see else for this if statement): https://github.com/apache/spark/blob/d5573c578a1eea9ee04886d9df37c7178e67bb30/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L412 https://github.com/apache/spark/blob/d5573c578a1eea9ee04886d9df37c7178e67bb30/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L427-L430 Tests also bear this out. ## How was this patch tested? This is just a doc string fix: I built Spark and ran a single test. Closes apache#22865 from bersprockets/confdocfix. Authored-by: Bruce Robbins <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 4e990d9) Signed-off-by: Wenchen Fan <[email protected]>
1 parent 00771dc commit b6ba0dd

File tree

1 file changed

+2
-1
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/internal

1 file changed

+2
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,8 @@ object SQLConf {
442442
val PARQUET_RECORD_FILTER_ENABLED = buildConf("spark.sql.parquet.recordLevelFilter.enabled")
443443
.doc("If true, enables Parquet's native record-level filtering using the pushed down " +
444444
"filters. This configuration only has an effect when 'spark.sql.parquet.filterPushdown' " +
445-
"is enabled.")
445+
"is enabled and the vectorized reader is not used. You can ensure the vectorized reader " +
446+
"is not used by setting 'spark.sql.parquet.enableVectorizedReader' to false.")
446447
.booleanConf
447448
.createWithDefault(false)
448449

0 commit comments

Comments
 (0)