docs: document per-job adaptive parallelism strategy session variables (v2.8.0)#1027
Closed
docs: document per-job adaptive parallelism strategy session variables (v2.8.0)#1027
Conversation
Document the new per-job adaptive parallelism strategy session variables introduced in v2.8.0 (risingwavelabs/risingwave#24250): - manage-a-large-number-of-streaming-jobs.mdx: add a new 'Per-job strategy override' subsection under 'Configure adaptive parallelism strategy', explaining streaming_parallelism_strategy_for_* session variables with examples, a defaults table, and a tip on shared cluster usage. - view-configure-runtime-parameters.mdx: add rows for all five new streaming_parallelism_strategy_for_* variables and for enable_mv_selection (from #24611) to the session variable reference table.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR documents the new per-job adaptive parallelism strategy session variables introduced in RisingWave v2.8.0, which allow fine-grained control over parallelism for individual streaming jobs. Previously, adaptive_parallelism_strategy was only configurable as a global system parameter. The new session variables (streaming_parallelism_strategy_for_*) enable operators to cap or tune parallelism at the job level, with the strategy persisted and reapplied on cluster rescales.
Changes:
- Added documentation for 5 new per-job parallelism strategy session variables (
streaming_parallelism_strategy_for_{materialized_view,table,sink,source,index}) - Added documentation for the
enable_mv_selectionparameter (v2.8.0 feature) - Created a new "Per-job strategy override" subsection with examples, defaults table, and usage guidance
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| operate/view-configure-runtime-parameters.mdx | Added 6 new parameter rows (5 for per-job strategy variables, 1 for enable_mv_selection) with descriptions, defaults, and version info |
| operate/manage-a-large-number-of-streaming-jobs.mdx | Added "Per-job strategy override" subsection with SQL examples, defaults table, and usage tip; minor formatting fix on line 28 |
| | streaming_use_arrangement_backfill | true/false | Enable arrangement backfill for streaming queries. Defaults to true. | | ||
| | streaming_use_snapshot_backfill | true/false | Enable snapshot backfill for streaming queries. Provides better isolation between the backfill and streaming phases by preventing resource contention when creating materialized views, indexes, or sinks. Defaults to true. | | ||
| | enable_index_selection | true/false | Enable index selection for streaming and batch queries. Defaults to true. For details, see [Index selection](/performance/best-practices#index-selection).| | ||
| | enable_mv_selection | true/false | Enable MV selection for batch queries. When enabled, the optimizer rewrites batch queries to scan a matching materialized view instead of the raw base tables. Defaults to false. Added in v2.8.0. For details, see [MV selection for batch queries](/performance/best-practices#mv-selection-for-batch-queries). | |
There was a problem hiding this comment.
The link /performance/best-practices#mv-selection-for-batch-queries does not exist. The section "MV selection for batch queries" is not present in the best-practices.mdx file. This broken link will result in a 404 error when users click on it. Either remove the link reference or ensure the corresponding documentation section is created.
Suggested change
| | enable_mv_selection | true/false | Enable MV selection for batch queries. When enabled, the optimizer rewrites batch queries to scan a matching materialized view instead of the raw base tables. Defaults to false. Added in v2.8.0. For details, see [MV selection for batch queries](/performance/best-practices#mv-selection-for-batch-queries). | | |
| | enable_mv_selection | true/false | Enable MV selection for batch queries. When enabled, the optimizer rewrites batch queries to scan a matching materialized view instead of the raw base tables. Defaults to false. Added in v2.8.0. For details, see [MV selection for batch queries](/performance/best-practices). | |
Comment on lines
+55
to
+59
| | streaming_parallelism_strategy_for_materialized_view | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for materialized views. Overrides the global `adaptive_parallelism_strategy` system parameter for the next `CREATE MATERIALIZED VIEW`. If set to `default`, inherits the global strategy. Added in v2.8.0. For details, see [Per-job strategy override](/operate/manage-a-large-number-of-streaming-jobs#per-job-strategy-override). | | ||
| | streaming_parallelism_strategy_for_table | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for tables. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE TABLE`. Added in v2.8.0. | | ||
| | streaming_parallelism_strategy_for_sink | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sinks. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SINK`. If set to `default`, inherits the global strategy. Added in v2.8.0. | | ||
| | streaming_parallelism_strategy_for_source | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sources. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SOURCE`. Added in v2.8.0. | | ||
| | streaming_parallelism_strategy_for_index | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for indexes. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE INDEX`. If set to `default`, inherits the global strategy. Added in v2.8.0. | |
There was a problem hiding this comment.
The descriptions for streaming_parallelism_strategy_for_materialized_view, streaming_parallelism_strategy_for_sink, and streaming_parallelism_strategy_for_index don't explicitly state what their default value is. They only explain what happens "If set to default", which could be ambiguous. For consistency with the table parameter and source parameter descriptions (which explicitly state "Defaults to Bounded(4)"), consider adding "Defaults to default." at the beginning of these descriptions, similar to how line 56 and 58 are formatted.
Suggested change
| | streaming_parallelism_strategy_for_materialized_view | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for materialized views. Overrides the global `adaptive_parallelism_strategy` system parameter for the next `CREATE MATERIALIZED VIEW`. If set to `default`, inherits the global strategy. Added in v2.8.0. For details, see [Per-job strategy override](/operate/manage-a-large-number-of-streaming-jobs#per-job-strategy-override). | | |
| | streaming_parallelism_strategy_for_table | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for tables. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE TABLE`. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_sink | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sinks. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SINK`. If set to `default`, inherits the global strategy. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_source | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sources. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SOURCE`. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_index | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for indexes. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE INDEX`. If set to `default`, inherits the global strategy. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_materialized_view | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for materialized views. Defaults to `default`. Overrides the global `adaptive_parallelism_strategy` system parameter for the next `CREATE MATERIALIZED VIEW`. If set to `default`, inherits the global strategy. Added in v2.8.0. For details, see [Per-job strategy override](/operate/manage-a-large-number-of-streaming-jobs#per-job-strategy-override). | | |
| | streaming_parallelism_strategy_for_table | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for tables. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE TABLE`. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_sink | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sinks. Defaults to `default`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SINK`. If set to `default`, inherits the global strategy. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_source | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sources. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SOURCE`. Added in v2.8.0. | | |
| | streaming_parallelism_strategy_for_index | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for indexes. Defaults to `default`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE INDEX`. If set to `default`, inherits the global strategy. Added in v2.8.0. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the new per-job adaptive parallelism strategy session variables introduced in v2.8.0 (risingwavelabs/risingwave#24250).
Previously,
adaptive_parallelism_strategywas only a global system parameter. This release adds per-job session variables (streaming_parallelism_strategy_for_*) that let operators cap or tune parallelism for individual jobs at creation time — with the strategy stored persistently and reapplied on rescale.Changes
operate/manage-a-large-number-of-streaming-jobs.mdx: Add a new#### Per-job strategy overridesubsection under Configure adaptive parallelism strategy, with examples, a defaults table for all five variables, and a usage tip.operate/view-configure-runtime-parameters.mdx:streaming_parallelism_strategy_for_{materialized_view,table,sink,source,index}.enable_mv_selection(from v2.8.0 #24611, missed in previous table).Related
Mintlify
0 threads from 0 users in Mintlify