[Improve][Flink]supports multiple parallelisms and remove flink-specific logic from API by CloverDew · Pull Request #10107 · apache/seatunnel

CloverDew · 2025-11-24T03:54:51Z

Purpose of this pull request: Fixes #9980

Problem

After implementing CDC schema evolution support in Flink engine, several issues were identified:

The coordinator's state resides in the TaskManager's JVM; if a job fails and restarts, this state is easily lost.
Flink's proprietary implementation logic should not be placed in the public API SupportSchemaEvolutionSinkWriter. Furthermore, flush requires manual API implementation.
Multi-parallelism configuration is not supported.

Solution

This PR made some minor adjustments to the architecture.:

BroadcastSchemaSinkOperator operator is introduced to solidify the response state.
Coordinator is only responsible for communication: pure messenger, stateless.
Flushing is guaranteed by checkpoints, not manual implementation

Key Changes:

1. Enhanced LocalSchemaCoordinator

JobId isolation: Prevents multi-job interference using Map<String, WeakReference<LocalSchemaCoordinator>>
Pure messenger: No persistent state, only temporary communication futures

3. Streamlined BroadcastSchemaSinkOperator

Idempotency: tracks lastProcessedEpoch using Flink state

4. API Compliance (Addresses #9980)

API separation: SupportSchemaEvolutionSinkWriter contains only generic applySchemaChange() method
Flink-specific logic moved: All coordination logic moved from API to Flink translation layer

Testing

Verified MySQL CDC → MySQL sink schema evolution scenarios
Confirmed no data loss during schema changes
Tested multi-job concurrent execution

Breaking Changes

None. This is a refactoring that maintains API compatibility while improving internal implementation.

Related Issues

Fixes [Bug][Flink] Move Flink-specific flush coordination from API to translation layer #9980: Move Flink-specific flush coordination from API to translation layer.
To address the issue of state loss, an operator-held state mechanism is introduced.
Multi-parallelism is already supported.

Check list

If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If necessary, please update incompatible-changes.md to describe the incompatibility caused by this PR.
If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)

.../src/main/java/org/apache/seatunnel/core/starter/flink/execution/SourceExecuteProcessor.java

Carl-Zhou-CN · 2025-12-15T06:42:22Z

...flink-common/src/main/java/org/apache/seatunnel/translation/flink/schema/SchemaOperator.java

+        }
+    }
+
+    private void sendSchemaChangeEventToDownstream(SchemaChangeEvent schemaChangeEvent) {


Suggested change

private void sendSchemaChangeEventToDownstream(SchemaChangeEvent schemaChangeEvent) {

private void sendSchemaChangeEventToDownStream(SchemaChangeEvent schemaChangeEvent) {

Is there a problem here?

You are correct.

Carl-Zhou-CN · 2025-12-15T06:47:23Z

...flink-common/src/main/java/org/apache/seatunnel/translation/flink/schema/SchemaOperator.java

+    private final Config pluginConfig;
+    private volatile Long lastProcessedEventTime;
+    private transient LocalSchemaCoordinator coordinator;
+    private transient Map<String, List<BufferedDataRow>> bufferedDataRows;


Doesn't it need to be stored in the checkpoint?

It's also possible to store the data, but I've already stored it at the sink end. The operators closer to the source end don't necessarily need to be materialized either, especially if the source can provide a replay strategy.
The coordinator does not need to be persisted because it is only a communication component;The BroadcastSchemaSinkOperator is the component that actually holds the coordinator's response state. The coordinator can be rebuilt after a crash, and the job manager will read the state from the sink operator, buffered data can be persisted, but the latest processing time does not need to be persisted because it has already been persisted by the sink.

I'm going to finalize the following variables; the buffered data rows can be rebuilt through reprocessing:
localSchemaState - Stores the schema state of each table; updated when the schema changes.
lastProcessedEventTime - The time of the last processed event, used to ensure event order.
schemaChangePending - Indicates whether a schema change is currently in progress.

After the downstream schema change is completed and a checkpoint is performed, failures occur during the process of re-consuming the cache. Is there a possibility that this batch of data will be lost? Because seemingly, this segment of data does not affect the checkpoint process of source and sink data, and it cannot be recovered from the state upon restart.

You're right, this situation can indeed happen. After the checkpoint, the source side submits the offset, assuming that this portion of the data has been processed and won't be replayed, so the data is lost. Caching is indeed necessary.

Carl-Zhou-CN · 2025-12-15T07:50:37Z

...n/java/org/apache/seatunnel/translation/flink/schema/coordinator/LocalSchemaCoordinator.java

+        }
+    }
+
+    private void performPeriodicCleanup() {


Under what circumstances will an expired task occur?

For example, the following scenario could lead to a timeout:

The SchemaOperator initiates an alter table add column n request.

The coordinator waits for acknowledgments from 3 sink subtasks (parallelism=3).

Subtask-0 and Subtask-1 successfully apply the schema change and send acknowledgments.

Subtask-2 crashes or is restarted during the application process.

Subtask-2 does not recover and send an acknowledgment within 5 minutes.

The request times out and is cleaned up.

Carl-Zhou-CN · 2025-12-15T07:53:49Z

...flink-common/src/main/java/org/apache/seatunnel/translation/flink/schema/SchemaOperator.java

+        long eventTime = schemaChangeEvent.getCreatedTime();
+
+        try {
+            if (lastProcessedEventTime != null && eventTime <= lastProcessedEventTime) {


Could you give an example of the scene that occurred?

Each time a change event is processed, lastProcessedEventTime is updated. Any event with a timestamp less than or equal to the current watermark is rejected; otherwise, it would lead to inconsistencies in the evolution process.

For example, consider executing the following SQL statements:

Original table structure: orders(id, user_id, amount)
-- T1=1000: Add discount field
ALTER TABLE orders ADD COLUMN discount DECIMAL(10,2);
-- T2=2000: Add status field
ALTER TABLE orders ADD COLUMN status VARCHAR(20);
-- T3=3000: Drop discount field
ALTER TABLE orders DROP COLUMN discount;

The expected final correct schema should be: orders(id, user_id, amount, status)

The correct epoch processing order should be: 1000 → 3000 → Completion

However, if T2 (2000) is processed late, the following will happen:

After processing T3 (3000), the Sink considers the current schema to be: orders(id, user_id, amount). If this late data is not rejected, and T2 (2000) is suddenly processed, the schema becomes: orders(id, user_id, amount, status). At this point, if data is written:
SeaTunnelRow row = new SeaTunnelRow(new Object[]{
1001L, // id
501L, // user_id
new BigDecimal("99.99"), // amount
"PAID" // status - but the Sink might interpret this as the discount field
});

This would result in a mapping mismatch, so I made a judgment here.

ok, thank you.

Carl-Zhou-CN · 2025-12-15T07:54:43Z

...flink-common/src/main/java/org/apache/seatunnel/translation/flink/schema/SchemaOperator.java

+                    jobId,
+                    eventTime);
+
+            String key = tableId.toString() + "#" + eventTime;


One method can be used. I see that it has been applied in several places

… persistence

TyrantLucifer

LGTM, thank you for your contribution.

LiJie20190102

LGTM, thank you for your contribution.

Carl-Zhou-CN

+1

Carl-Zhou-CN · 2026-02-10T01:51:22Z

@CloverDew Thank you for your contribution

github-actions bot added core SeaTunnel core module flink connectors-v2 e2e api labels Nov 24, 2025

CloverDew force-pushed the feature/supports-multiple-parallelisms-on-flink branch 3 times, most recently from 1e42e4d to 460e584 Compare November 24, 2025 14:00

Carl-Zhou-CN changed the title ~~[Fix][Flink]supports multiple parallelisms and remove flink-specific logic from API~~ [Impove][Flink]supports multiple parallelisms and remove flink-specific logic from API Nov 25, 2025

CloverDew changed the title ~~[Impove][Flink]supports multiple parallelisms and remove flink-specific logic from API~~ [Improve][Flink]supports multiple parallelisms and remove flink-specific logic from API Nov 25, 2025

CloverDew force-pushed the feature/supports-multiple-parallelisms-on-flink branch from 460e584 to ae2c6b3 Compare November 27, 2025 09:38

LiJie20190102 reviewed Dec 4, 2025

View reviewed changes

.../src/main/java/org/apache/seatunnel/core/starter/flink/execution/SourceExecuteProcessor.java Outdated Show resolved Hide resolved

CloverDew force-pushed the feature/supports-multiple-parallelisms-on-flink branch from 4886556 to 35fa5a4 Compare December 5, 2025 08:43

Carl-Zhou-CN reviewed Dec 15, 2025

View reviewed changes

CloverDew force-pushed the feature/supports-multiple-parallelisms-on-flink branch 2 times, most recently from 95fc353 to c9bd451 Compare December 17, 2025 13:55

github-actions bot added the CI&CD label Dec 18, 2025

CloverDew and others added 3 commits January 9, 2026 21:19

[Improve][Flink] Supports multiple parallelisms and coordinator state…

19eea3d

… persistence

improve

07c1c30

fix

83897d0

CloverDew force-pushed the feature/supports-multiple-parallelisms-on-flink branch from a39c03d to 83897d0 Compare January 9, 2026 13:22

github-actions bot removed the CI&CD label Jan 9, 2026

TyrantLucifer approved these changes Jan 13, 2026

View reviewed changes

github-actions bot added approved reviewed labels Jan 13, 2026

davidzollo requested a review from LiJie20190102 January 29, 2026 02:07

LiJie20190102 approved these changes Jan 29, 2026

View reviewed changes

corgy-w requested a review from Carl-Zhou-CN February 2, 2026 13:37

Carl-Zhou-CN approved these changes Feb 10, 2026

View reviewed changes

Carl-Zhou-CN merged commit 89b3e1c into apache:dev Feb 10, 2026
5 checks passed

	private void sendSchemaChangeEventToDownstream(SchemaChangeEvent schemaChangeEvent) {
	private void sendSchemaChangeEventToDownStream(SchemaChangeEvent schemaChangeEvent) {

Conversation

CloverDew commented Nov 24, 2025

Problem

Solution

Key Changes:

Testing

Breaking Changes

Related Issues

Check list

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CloverDew Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CloverDew Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TyrantLucifer left a comment

Choose a reason for hiding this comment

Uh oh!

LiJie20190102 left a comment

Choose a reason for hiding this comment

Uh oh!

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

Uh oh!

Carl-Zhou-CN commented Feb 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CloverDew Dec 16, 2025 •

edited

Loading

CloverDew Dec 16, 2025 •

edited

Loading