OOM
#2273
Replies: 1 comment
-
jdbcProperties.setProperty("scan.incremental.snapshot.chunk.size", "1024"); DatabaseSetting fromDatabaseSetting = drsSetting.getFromDatabaseSetting();
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
一次性读取数据太多,造成OOM。按照官网说明在flink-conf.yaml设置了下面的参数也没用
The chunk size (number of rows) of table snapshot, captured tables are split into multiple chunks when read the snapshot of table.
scan.incremental.snapshot.chunk.size: 2048
he maximum fetch size for per poll when read table snapshot.
scan.snapshot.fetch.size: 256
cdc一次读取的数据行数
cdc.reader.fetch-size: 2000
错误信息如下,还需要什么设置不。
2023-07-07 14:47:00,579 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 225685 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:19.86
2023-07-07 14:47:05,652 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 114498 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:24.933
2023-07-07 14:47:11,803 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 285931 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:31.082
2023-07-07 14:47:17,222 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 154307 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:36.503
2023-07-07 14:47:22,892 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 306658 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:42.173
2023-07-07 14:47:28,384 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 163945 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:47.665
2023-07-07 14:47:33,909 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 311950 records for split 'egenie_baseinfo.platform_product_match:0' after 00:00:53.19
2023-07-07 14:47:39,390 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 166262 records for split 'egenie_baseinfo.platform_product_match:1' after 00:00:58.671
2023-07-07 14:47:45,214 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 313365 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:04.495
2023-07-07 14:47:52,457 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 167219 records for split 'egenie_baseinfo.platform_product_match:1' after 00:01:11.738
2023-07-07 14:47:57,005 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 314004 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:16.286
2023-07-07 14:48:07,657 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 314190 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:26.937
2023-07-07 14:48:07,657 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 167465 records for split 'egenie_baseinfo.platform_product_match:1' after 00:01:22.806
2023-07-07 14:48:18,978 INFO com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask [] - Exported 314290 records for split 'egenie_baseinfo.platform_product_match:0' after 00:01:38.258
2023-07-07 14:48:17,708 ERROR com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader [] - Execute snapshot read task for mysql split MySqlSnapshotSplit{tableId=egenie_baseinfo.platform_product_match, splitId='egenie_baseinfo.platform_product_match:1', splitKeyType=[
platform_product_match_id
BIGINT NOT NULL], splitStart=[12077194], splitEnd=[12482912], highWatermark=null} failjava.lang.OutOfMemoryError: GC overhead limit exceeded
Beta Was this translation helpful? Give feedback.
All reactions