Skip to content

docs: document per-job adaptive parallelism strategy session variables (v2.8.0) #1027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lmatz wants to merge 1 commit into from
Closed

docs: document per-job adaptive parallelism strategy session variables (v2.8.0) #1027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 39 additions & 2 deletions operate/manage-a-large-number-of-streaming-jobs.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The adaptive parallelism feature in v1.7.0 ensures that every streaming job can

### Configure adaptive parallelism strategy

RisingWave introduces the [system parameter](/operate/view-configure-system-parameters)`adaptive_parallelism_strategy` in v2.3.0 to control how jobs behave when they are of the adaptive type. This provides more granular control over adaptive scheduling behavior.
RisingWave introduces the [system parameter](/operate/view-configure-system-parameters) `adaptive_parallelism_strategy` in v2.3.0 to control how jobs behave when they are of the adaptive type. This provides more granular control over adaptive scheduling behavior.

The supported values are:

Expand All @@ -41,12 +41,49 @@ When this parameter is modified, RisingWave sends a local notification to inform

All parallelism values remain subject to the limits of max parallelism and resource groups.

You can also set this parameter using the following syntax:
You can set this parameter cluster-wide using:

```sql
ALTER SYSTEM SET adaptive_parallelism_strategy TO '<strategy>';
```

#### Per-job strategy override

<Note>
Added in v2.8.0.
</Note>

In addition to the global system parameter, you can override the adaptive parallelism strategy for a specific job at creation time using session variables. The strategy is stored with the job and reapplied automatically whenever the cluster rescales — so it acts as a persistent policy, not a one-time setting.

```sql
-- Limit this MV to at most 4 actors, regardless of cluster size
SET streaming_parallelism_strategy_for_materialized_view = 'Bounded(4)';
CREATE MATERIALIZED VIEW my_low_priority_mv AS SELECT ...;

-- Use 50% of available parallelism for this sink
SET streaming_parallelism_strategy_for_sink = 'Ratio(0.5)';
CREATE SINK my_sink INTO my_table AS SELECT ...;

-- Reset to inherit the global strategy
SET streaming_parallelism_strategy_for_materialized_view = 'default';
```

The per-job session variables and their defaults are:

| Session variable | Default | Applies to |
|---|---|---|
| `streaming_parallelism_strategy_for_materialized_view` | `default` (inherits global) | `CREATE MATERIALIZED VIEW` |
| `streaming_parallelism_strategy_for_table` | `Bounded(4)` | `CREATE TABLE` |
| `streaming_parallelism_strategy_for_sink` | `default` (inherits global) | `CREATE SINK` |
| `streaming_parallelism_strategy_for_source` | `Bounded(4)` | `CREATE SOURCE` |
| `streaming_parallelism_strategy_for_index` | `default` (inherits global) | `CREATE INDEX` |

Setting a variable to `default` means the job inherits the global `adaptive_parallelism_strategy` system parameter. All values accept the same strategy syntax: `Auto`, `Full`, `Bounded(n)`, or `Ratio(r)`.

<Tip>
Use per-job overrides to reserve cluster headroom on shared deployments — for example, keep background enrichment jobs at `Bounded(4)` while letting high-priority MVs use `Full`.
</Tip>

### Limit the concurrency of creating streaming jobs

If you want to create multiple streaming jobs at once using scripts or tools such as DBT, the [system parameter](/operate/view-configure-system-parameters) `max_concurrent_creating_streaming_jobs` is helpful. It controls the maximum number of streaming jobs created concurrently. However, please do not set it too high, as it may introduce excessive pressure on the cluster.
Expand Down
6 changes: 6 additions & 0 deletions operate/view-configure-runtime-parameters.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ Below is the detailed information about the parameters you may see after using t
| streaming_parallelism_for_index | default/1,2,3,... | Specific parallelism for indexes. If set to `default`, it will fall back to the global `streaming_parallelism`. |
| streaming_parallelism_for_sink | default/1,2,3,... | Specific parallelism for sinks. If set to `default`, it will fall back to the global `streaming_parallelism`. |
| streaming_parallelism_for_source | default/1,2,3,... | Specific parallelism for sources. If set to `default`, it will fall back to the global `streaming_parallelism`. |
| streaming_parallelism_strategy_for_materialized_view | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for materialized views. Overrides the global `adaptive_parallelism_strategy` system parameter for the next `CREATE MATERIALIZED VIEW`. If set to `default`, inherits the global strategy. Added in v2.8.0. For details, see [Per-job strategy override](/operate/manage-a-large-number-of-streaming-jobs#per-job-strategy-override). |
| streaming_parallelism_strategy_for_table | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for tables. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE TABLE`. Added in v2.8.0. |
| streaming_parallelism_strategy_for_sink | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sinks. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SINK`. If set to `default`, inherits the global strategy. Added in v2.8.0. |
| streaming_parallelism_strategy_for_source | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sources. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SOURCE`. Added in v2.8.0. |
| streaming_parallelism_strategy_for_index | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for indexes. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE INDEX`. If set to `default`, inherits the global strategy. Added in v2.8.0. |
Comment on lines +55 to +59
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The descriptions for streaming_parallelism_strategy_for_materialized_view, streaming_parallelism_strategy_for_sink, and streaming_parallelism_strategy_for_index don't explicitly state what their default value is. They only explain what happens "If set to default", which could be ambiguous. For consistency with the table parameter and source parameter descriptions (which explicitly state "Defaults to Bounded(4)"), consider adding "Defaults to default." at the beginning of these descriptions, similar to how line 56 and 58 are formatted.

Suggested change
| streaming_parallelism_strategy_for_materialized_view | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for materialized views. Overrides the global `adaptive_parallelism_strategy` system parameter for the next `CREATE MATERIALIZED VIEW`. If set to `default`, inherits the global strategy. Added in v2.8.0. For details, see [Per-job strategy override](/operate/manage-a-large-number-of-streaming-jobs#per-job-strategy-override). |
| streaming_parallelism_strategy_for_table | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for tables. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE TABLE`. Added in v2.8.0. |
| streaming_parallelism_strategy_for_sink | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sinks. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SINK`. If set to `default`, inherits the global strategy. Added in v2.8.0. |
| streaming_parallelism_strategy_for_source | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sources. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SOURCE`. Added in v2.8.0. |
| streaming_parallelism_strategy_for_index | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for indexes. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE INDEX`. If set to `default`, inherits the global strategy. Added in v2.8.0. |
| streaming_parallelism_strategy_for_materialized_view | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for materialized views. Defaults to `default`. Overrides the global `adaptive_parallelism_strategy` system parameter for the next `CREATE MATERIALIZED VIEW`. If set to `default`, inherits the global strategy. Added in v2.8.0. For details, see [Per-job strategy override](/operate/manage-a-large-number-of-streaming-jobs#per-job-strategy-override). |
| streaming_parallelism_strategy_for_table | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for tables. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE TABLE`. Added in v2.8.0. |
| streaming_parallelism_strategy_for_sink | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sinks. Defaults to `default`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SINK`. If set to `default`, inherits the global strategy. Added in v2.8.0. |
| streaming_parallelism_strategy_for_source | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for sources. Defaults to `Bounded(4)`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE SOURCE`. Added in v2.8.0. |
| streaming_parallelism_strategy_for_index | default/Auto/Full/Bounded(n)/Ratio(r) | Per-job adaptive parallelism strategy for indexes. Defaults to `default`. Overrides the global `adaptive_parallelism_strategy` for the next `CREATE INDEX`. If set to `default`, inherits the global strategy. Added in v2.8.0. |

Copilot uses AI. Check for mistakes.
| streaming_max_parallelism | 256 | The maximum parallelism allowed for streaming queries. For more information, see [Configuring maximum parallelism](/deploy/k8s-cluster-scaling#configuring-maximum-parallelism). |
| streaming_enable_delta_join | true/false | Enable delta join for streaming queries. Defaults to false. |
| streaming_enable_bushy_join | true/false | Enable bushy join for streaming queries. Defaults to true. |
Expand All @@ -64,6 +69,7 @@ Below is the detailed information about the parameters you may see after using t
| streaming_use_arrangement_backfill | true/false | Enable arrangement backfill for streaming queries. Defaults to true. |
| streaming_use_snapshot_backfill | true/false | Enable snapshot backfill for streaming queries. Provides better isolation between the backfill and streaming phases by preventing resource contention when creating materialized views, indexes, or sinks. Defaults to true. |
| enable_index_selection | true/false | Enable index selection for streaming and batch queries. Defaults to true. For details, see [Index selection](/performance/best-practices#index-selection).|
| enable_mv_selection | true/false | Enable MV selection for batch queries. When enabled, the optimizer rewrites batch queries to scan a matching materialized view instead of the raw base tables. Defaults to false. Added in v2.8.0. For details, see [MV selection for batch queries](/performance/best-practices#mv-selection-for-batch-queries). |
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link /performance/best-practices#mv-selection-for-batch-queries does not exist. The section "MV selection for batch queries" is not present in the best-practices.mdx file. This broken link will result in a 404 error when users click on it. Either remove the link reference or ensure the corresponding documentation section is created.

Suggested change
| enable_mv_selection | true/false | Enable MV selection for batch queries. When enabled, the optimizer rewrites batch queries to scan a matching materialized view instead of the raw base tables. Defaults to false. Added in v2.8.0. For details, see [MV selection for batch queries](/performance/best-practices#mv-selection-for-batch-queries). |
| enable_mv_selection | true/false | Enable MV selection for batch queries. When enabled, the optimizer rewrites batch queries to scan a matching materialized view instead of the raw base tables. Defaults to false. Added in v2.8.0. For details, see [MV selection for batch queries](/performance/best-practices). |

Copilot uses AI. Check for mistakes.
| enable_join_ordering | true/false | Enable join ordering for streaming and batch queries. Defaults to true. |
| enable_locality_backfill | true/false | Enable locality backfill for streaming queries. Defaults to false. For details, see [Locality backfill](/performance/best-practices#locality-backfill).|
| enable_two_phase_agg | true/false | Enable two phase agg optimization. Defaults to true. Setting this to true will always set `FORCE_TWO_PHASE_AGG` to false. |
Expand Down
Loading