Skip to content

Clarify parallelism migration guidance: SQL for existing jobs and new default strategies #1012

@kwannoel

Description

@kwannoel

Context

Slack thread (customer + internal triage):

Related change referenced in discussion:

Current docs pages involved:

Problem

Users are confused about whether they must manually update all existing MVs/streaming jobs when adopting newer recommended defaults (bounded(4) for source/table and bounded(64) system adaptive).

The current docs explain adaptive strategy and per-object ALTER commands, but do not clearly provide:

  1. A migration playbook for clusters that were created before the new defaults.
  2. SQL patterns to generate bulk ALTER statements for existing relations.
  3. A clear split between:
  • settings that affect future jobs only, and
  • per-relation changes required for existing jobs.

This gap leads to repeated support questions and inconsistent operational changes.

Suggested Fix

Add a dedicated note/section in operate/manage-a-large-number-of-streaming-jobs.mdx (and cross-link from deploy/k8s-cluster-scaling.mdx) titled something like Migrate to recommended parallelism defaults.

Include:

  1. Recommended defaults and scope
  • streaming_parallelism_strategy_for_source = 'BOUNDED(4)'
  • streaming_parallelism_strategy_for_table = 'BOUNDED(4)'
  • adaptive_parallelism_strategy = 'BOUNDED(64)'
  • Explicitly state these defaults affect newly created jobs (unless altered later).
  1. SQL to set system defaults for future jobs
ALTER SYSTEM SET streaming_parallelism_strategy_for_source = 'BOUNDED(4)';
ALTER SYSTEM SET streaming_parallelism_strategy_for_table = 'BOUNDED(4)';
ALTER SYSTEM SET adaptive_parallelism_strategy = 'BOUNDED(64)';
  1. SQL patterns to migrate existing relations
  • Show how to inspect current parallelism and generate ALTER statements for existing relations that deviate from target values.
  • Provide separate examples for source/table vs other streaming jobs where appropriate.
  • Keep examples safe and explicit (generate SQL first, review, then execute).
  1. Operational safety notes
  • Recommend staged rollout and observation window.
  • Call out actor-count/capacity checks before and after changes.
  • Clarify there is no single switch that force-updates all existing jobs without per-relation ALTER.
  1. Version-awareness note
  • Mention that clusters on older versions may not have these defaults and should follow migration guidance.

This should reduce ambiguity around “existing vs future” behavior and make support guidance consistent with PR #24678 defaults.

Metadata

Metadata

Labels

bugSomething isn't workingdocs-agentdocumentationImprovements or additions to documentation

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions