SQL Mongo CDC

[hthcool]

使用 Flink SQL Mongo CDC 时 state 过大，目前 state backend 使用了 rocksdb 且开启了 state incremental，如果从根本上解决状态过大的问题？

[yuxiqian]

Currently MongoDB CDC could only generate upsert changelog. So Flink will automatically insert a ChangelogNormalize node in downstream, which causes huge state storage cost.

We've implemented an experimental feature called scan.full-changelog, which will make use of MongoDB 6.0's pre- and post-image feature to remove the necessity of normalizer node.

It doesn't come within latest release 2.4, but you can build master branch and try it first. You may refer to snapshot docs for more details.