docs: DH-20056: Update docs for new predicate pushdown operations. (#7720)

Co-authored-by: margaretkennedy <82049573+margaretkennedy@users.noreply.github.com>

Author: lbooker42

docs/groovy/how-to-guides/predicate-pushdown.md

@@ -14,7 +14,9 @@ Filters are prioritized for pushdown in the following order (from highest to low
 
 - Filtering [single value column sources](#single-value-column-sources).
 - Range and match filtering of Parquet data columns using [row group metadata](#parquet-row-group-metadata).
-- Filtering columns with an existing Deephaven [data index](#deephaven-data-indexes).
+- Filtering columns with a cached (already loaded in memory) Deephaven [data index](#deephaven-data-indexes).
+- Filtering columns with dictionary encoding.
+- Filtering columns with an un-cached Deephaven [data index](#deephaven-data-indexes).
 
 > [!IMPORTANT]
 > Where multiple filters have the same pushdown priority, the user-supplied order will generally be maintained. Stateful filters and filter barriers are always respected during pushdown operations.
@@ -43,11 +45,26 @@ result = source.where("Test1 > 90")
 
 Parquet metadata is optional, and not all Parquet files will have it. If the metadata is not available, Deephaven will fall back to scanning the data in the row groups to apply the filter. If Parquet metadata is malformed or incorrect, it may lead to incorrect filtering results. In such cases, you can [disable](#disabling-predicate-pushdown-features) this optimization.
 
+## Parquet dictionary encoding
+
+When storing `string` data, Parquet may create a dictionary encoding for the column where a dictionary of unique values is stored separately from the column data. The column data contains references (e.g., integer indices) to the dictionary entries instead of the actual string values. This can significantly reduce storage space and improve performance for columns with many repeated values. This additionally allows for efficient filtering on these columns, as the engine can check the unique dictionary values against the filter to determine matches without scanning the entire column. If matches are found, the engine will note which integer indices in the column data correspond to the matching dictionary values and filter the data much more efficiently than loading each string value and applying the filter. Additionally, if no matches are found, the engine can skip the entire column without scanning any of the row data.
+
+Nearly all single-column filters can be optimized using the dictionary encoding.
+
+```groovy order=source,result
+import io.deephaven.parquet.table.ParquetTools
+
+source = ParquetTools.readTable("/data/examples/ParquetExamples/grades/grades.parquet")
+result = source.where("Class = `Math`")
+```
+
+If desired, you can [disable](#disabling-predicate-pushdown-features) the use of dictionary encoding during pushdown operations:
+
 ## Deephaven data indexes
 
-Deephaven allows users to create data indexes when writing data to storage as Parquet files. These indexes can speed up filtering operations by applying the filter to the index instead of the larger table.
+Deephaven allows users to create data indexes for any table. These indexes can be retained in memory or written to storage and can speed up filtering operations significantly by applying the filter to the index instead of the larger table.
 
-Starting in Deephaven v0.40.0, the predicate pushdown framework enables data indexes to be used with most filter types (not just exact matches). When a materialized (in-memory) data index exists, the engine can leverage it during `where` operations. To avoid unexpected memory usage, filter operations do not automatically materialize deferred (disk-based) data indexes.
+Starting in Deephaven v0.40.0, the predicate pushdown framework enables data indexes to be used with most filter types (not just exact matches). When a materialized (in-memory) data index for a table exists, the engine can leverage it during `where` operations. To avoid unexpected memory usage, filter operations do not automatically materialize table-level data indexes. However, if an individual file-level data index is available on disk, the engine will use it to filter data without loading the entire index into memory.
 
 This technique is effective even if only a subset of the data files are indexed. The engine will filter non-indexed files using the standard method.
 
@@ -79,8 +96,10 @@ If desired, you can [disable](#disabling-predicate-pushdown-features) the use of
 
 Under certain circumstances, you may want to disable specific predicate pushdown features. These settings are global and will affect all pushdown operations across the Deephaven engine. The following properties affect pushdown:
 
+- [`QueryTable.useDataIndexForWhere`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#USE_DATA_INDEX_FOR_WHERE) – enables the use of Deephaven table-level data indexes when filtering.
 - [`QueryTable.disableWherePushdownParquetRowGroupMetadata`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_PARQUET_ROW_GROUP_METADATA) – disables consideration of Parquet row group metadata when filtering.
-- [`QueryTable.disableWherePushdownDataIndex`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_DATA_INDEX) – disables the use of Deephaven data indexes when filtering.
+- [`QueryTable.disableWherePushdownDataIndex`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_DATA_INDEX) – disables the use of file-level Deephaven data indexes when filtering.
+- [`QueryTable.disableWherePushdownParquetDictionary`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_PARQUET_DICTIONARY) – disables the use of dictionary encoding when filtering.
 
 For more information, see the [Query table configuration](../conceptual/query-table-configuration.md) documentation.
 

docs/groovy/snapshots/530d2dbfa281d5f3b4dc8dee85000c9d.json

@@ -0,0 +1 @@
+{"file":"how-to-guides/predicate-pushdown.md","objects":{"source":{"type":"Table","data":{"columns":[{"name":"Name","type":"java.lang.String"},{"name":"Class","type":"java.lang.String"},{"name":"Test1","type":"int"},{"name":"Test2","type":"int"}],"rows":[[{"value":"Ashley"},{"value":"Math"},{"value":"92"},{"value":"94"}],[{"value":"Jeff"},{"value":"Math"},{"value":"78"},{"value":"88"}],[{"value":"Rita"},{"value":"Math"},{"value":"87"},{"value":"81"}],[{"value":"Zach"},{"value":"Math"},{"value":"74"},{"value":"70"}],[{"value":"Ashley"},{"value":"Science"},{"value":"87"},{"value":"91"}],[{"value":"Jeff"},{"value":"Science"},{"value":"90"},{"value":"83"}],[{"value":"Rita"},{"value":"Science"},{"value":"99"},{"value":"95"}],[{"value":"Zach"},{"value":"Science"},{"value":"80"},{"value":"78"}],[{"value":"Ashley"},{"value":"History"},{"value":"82"},{"value":"88"}],[{"value":"Jeff"},{"value":"History"},{"value":"87"},{"value":"92"}],[{"value":"Rita"},{"value":"History"},{"value":"84"},{"value":"85"}],[{"value":"Zach"},{"value":"History"},{"value":"76"},{"value":"78"}]]}},"result":{"type":"Table","data":{"columns":[{"name":"Name","type":"java.lang.String"},{"name":"Class","type":"java.lang.String"},{"name":"Test1","type":"int"},{"name":"Test2","type":"int"}],"rows":[[{"value":"Ashley"},{"value":"Math"},{"value":"92"},{"value":"94"}],[{"value":"Jeff"},{"value":"Math"},{"value":"78"},{"value":"88"}],[{"value":"Rita"},{"value":"Math"},{"value":"87"},{"value":"81"}],[{"value":"Zach"},{"value":"Math"},{"value":"74"},{"value":"70"}]]}}}}

docs/python/conceptual/query-table-configuration.md

@@ -75,10 +75,13 @@ A Deephaven [DataIndex](../how-to-guides/data-indexes.md) is an index that can i
 
 Pushdown predicates refer to the mechanism whereby filtering conditions are applied as early as possible, ideally at the data source (e.g., Parquet or other columnar formats), before loading data into the system. By annotating source reads with predicates, the engine pulls in only the rows that satisfy the conditions, significantly reducing I/O and improving performance.
 
-| Property Name                                            | Default Value | Description                                                                                           |
-| -------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------- |
-| `QueryTable.disableWherePushdownDataIndex`               | false         | Disables the use of [data index](../how-to-guides/data-indexes.md) within where's pushdown predicates |
-| `QueryTable.disableWherePushdownParquetRowGroupMetadata` | false         | Disables the usage of Parquet row group metadata during push-down filtering                           |
+| Property Name                                            | Default Value | Description                                                                                               |
+| -------------------------------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------- |
+| `QueryTable.useDataIndexForWhere`                        | true          | Enables the uses of table-level [data index](../how-to-guides/data-indexes.md) during `where` operations. |
+| `QueryTable.disableWherePushdownDataIndex`               | false         | Disables the use of [data index](../how-to-guides/data-indexes.md) within `where`'s predicate pushdown.   |
+| `QueryTable.disableWherePushdownParquetRowGroupMetadata` | false         | Disables the usage of Parquet row group metadata during push-down filtering.                              |
+| `QueryTable.disableWherePushdownMergedTables`            | false         | Disable predicate pushdown when filtering merged tables.                                                  |
+| `QueryTable.disableWherePushdownParquetDictionary`       | false         | Disables dictionary-encoding predicate pushdown operations.                                               |
 
 ## Parallel processing with `where`
 

docs/python/how-to-guides/predicate-pushdown.md

@@ -46,11 +46,27 @@ result = source.where(filters="Test1 > 90")
 
 Parquet metadata is optional, and not all Parquet files will have it. If the metadata is not available, Deephaven will fall back to scanning the data in the row groups to apply the filter. If Parquet metadata is malformed or incorrect, it may lead to incorrect filtering results. In such cases, you can [disable](#disabling-predicate-pushdown-features) this optimization.
 
+## Parquet dictionary encoding
+
+When storing `string` data, Parquet may create a dictionary encoding for the column where a dictionary of unique values is stored separately from the column data. The column data contains references (e.g., integer indices) to the dictionary entries instead of the actual string values. This can significantly reduce storage space and improve performance for columns with many repeated values. This additionally allows for efficient filtering on these columns, as the engine can check the unique dictionary values against the filter to determine matches without scanning the entire column. If matches are found, the engine will note which integer indices in the column data correspond to the matching dictionary values and filter the data much more efficiently than loading each string value and applying the filter. Additionally, if no matches are found, the engine can skip the entire column without scanning any of the row data.
+
+Nearly all single-column filters can be optimized using the dictionary encoding.
+
+```python order=source,result
+from deephaven import parquet
+
+# pass the path of the local Parquet file to `read`
+source = parquet.read(path="/data/examples/ParquetExamples/grades/grades.parquet")
+result = source.where(filters="Class = `Math`")
+```
+
+If desired, you can [disable](#disabling-predicate-pushdown-features) the use of dictionary encoding during pushdown operations:
+
 ## Deephaven data indexes
 
-Deephaven allows users to create data indexes when writing data to storage as Parquet files. These indexes can speed up filtering operations by applying the filter to the index instead of the larger table.
+Deephaven allows users to create data indexes for any table. These indexes can be retained in memory or written to storage and can speed up filtering operations significantly by applying the filter to the index instead of the larger table.
 
-Starting in Deephaven v0.40.0, the predicate pushdown framework enables data indexes to be used with most filter types (not just exact matches). When a materialized (in-memory) data index exists, the engine can leverage it during `where` operations. To avoid unexpected memory usage, filter operations do not automatically materialize deferred (disk-based) data indexes.
+Starting in Deephaven v0.40.0, the predicate pushdown framework enables data indexes to be used with most filter types (not just exact matches). When a materialized (in-memory) data index for a table exists, the engine can leverage it during `where` operations. To avoid unexpected memory usage, filter operations do not automatically materialize table-level data indexes. However, if an individual file-level data index is available on disk, the engine will use it to filter data without loading the entire index into memory.
 
 This technique is effective even if only a subset of the data files are indexed. The engine will filter non-indexed files using the standard method.
 
@@ -80,8 +96,11 @@ If desired, you can [disable](#disabling-predicate-pushdown-features) the use of
 
 Under certain circumstances, you may want to disable specific predicate pushdown features. These settings are global and will affect all pushdown operations across the Deephaven engine. The following properties affect pushdown:
 
+- [`QueryTable.useDataIndexForWhere`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#USE_DATA_INDEX_FOR_WHERE) – enables the use of Deephaven table-level data indexes when filtering.
 - [`QueryTable.disableWherePushdownParquetRowGroupMetadata`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_PARQUET_ROW_GROUP_METADATA) – disables consideration of Parquet row group metadata when filtering.
-- [`QueryTable.disableWherePushdownDataIndex`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_DATA_INDEX) – disables the use of Deephaven data indexes when filtering.
+- [`QueryTable.disableWherePushdownDataIndex`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_DATA_INDEX) – disables the use of file-level Deephaven data indexes when filtering.
+- [`QueryTable.disableWherePushdownParquetDictionary`](https://docs.deephaven.io/core/javadoc/io/deephaven/engine/table/impl/QueryTable.html#DISABLE_WHERE_PUSHDOWN_PARQUET_DICTIONARY) – disables the use of dictionary encoding when filtering.
+-
 
 For more information, see the [Query table configuration](../conceptual/query-table-configuration.md) documentation.