MySqlSource获取不到新建的表数据？已设置scanNewlyAddedTableEnabled(true)

[wufanwind]

  【现象】
  2.2版本是好的，升级到2.4版本直到后面的3.0版本都有该问题。
  现象就是我们监听的mysql的db.*的表数据，但是当mysql里面新建表时，
  mySqlSource获取不到新建表的binlog数据。
  必须重启cdc任务，从savepoint启动后就有新表的新数据了（重启之前的历史数据还是缺失的）。
  给我们造成了很多困扰。
  
  【版本】
    <dependency>
        <groupId>com.ververica</groupId>
        <artifactId>flink-sql-connector-mysql-cdc</artifactId>
        <version>2.4.1</version>
    </dependency>      
    【代码】
            mySqlSource = MySqlSource.<JSONObject>builder()
                    .hostname(host)
                    .port(Integer.valueOf(port))
                    .scanNewlyAddedTableEnabled(true)
                    .databaseList(sourceDb)
                    .tableList(sourceDb + ".*")
                    .username(user)
                    .password(password)
                    .serverId(serverId)
                    .debeziumProperties(debeziumProperties)
                    .startupOptions(StartupOptions.initial())
                    .serverTimeZone("Asia/Shanghai")
                    .connectMaxRetries(3)
                    .connectTimeout(Duration.of(60, SECONDS))
                    .deserializer(new JsonDeserializationSchemaProd(targetDb,datasourceIds,excludeSourceTbs))
                    .includeSchemaChanges(true)
                    .jdbcProperties(jdbcProperties)
                    .build();

不知道大家用cdc实时同步mysql的全库表db.*的数据有没有遇到这个问题，怎么解决的？不可能天天重启吧。。

看了文档也是要重启，很无语啊。
https://nightlies.apache.org/flink/flink-cdc-docs-master/zh/docs/connectors/flink-sources/mysql-cdc/#动态加表

[gong]

ScanNewlyAddedTableEnable can be set to false, and for newly added tables in regular matching, it will not be captured when ScanNewlyAddedTableEnable is true. ScanNewlyAddedTableEnable is true and only applies to tables that were not previously captured by the task. Subsequent modifications were made to the new tables added by the task. As for 2.2, it was good because the community made changes to the capture logic for the newly added table.@wufanwind

[wufanwind]

Thank you a million times！！！

[Karl-WangSK]

ScanNewlyAddedTableEnable can be set to false, and for newly added tables in regular matching, it will not be captured when ScanNewlyAddedTableEnable is true. ScanNewlyAddedTableEnable is true and only applies to tables that were not previously captured by the task. Subsequent modifications were made to the new tables added by the task. As for 2.2, it was good because the community made changes to the capture logic for the newly added table.@wufanwind

why ScanNewlyAddedTableEnable set to true, the binlog can't capture data of new tables? what if I want to capture tables that were not previously captured, and still want to capture new table data in binlog after snapshot finished?

[gong]

ScanNewlyAddedTableEnable can be set to false, and for newly added tables in regular matching, it will not be captured when ScanNewlyAddedTableEnable is true. ScanNewlyAddedTableEnable is true and only applies to tables that were not previously captured by the task. Subsequent modifications were made to the new tables added by the task. As for 2.2, it was good because the community made changes to the capture logic for the newly added table.@wufanwind

why ScanNewlyAddedTableEnable set to true, the binlog can't capture data of new tables? what if I want to capture tables that were not previously captured, and still want to capture new table data in binlog after snapshot finished?

@Karl-WangSK Hello, You can refer to #3184