Skip to content

[Question][MySQL CDC] how to sync several tables(each table needs to sync specific columns)? #9666

@jmbkeyes

Description

@jmbkeyes

seatunnel version:2.3.11
It's can be done by the following config file if there's no requirement for specify table columns.
`
source {
MySQL-CDC {
base-url="jdbc:mysql://10.0.0.12:3306/bocai"
username = "root"
password = "xxx"
database-names=[
"bocai"
]
table-names=[
"bocai.account",
]

          startup.mode = "initial"
          snapshot.split.size = 8096
          snapshot.fetch.size = 1024
  }

}
sink {
Jdbc {
url = "jdbc:mysql://10.0.0.181:3306/bocai?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "xxx"
database = "bocai"
table = "${table_name}"
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
generate_sink_sql = true
data_save_mode = "APPEND_DATA"
}
}
**BUT If I want to sync several tables(each table needs to sync specific columns), how to specify several tables' columns.** right now I just add source for one table, and transform for the table.
source {
MySQL-CDC {
base-url="jdbc:mysql://10.0.0.12:3306/bocai"
username = "root"
password = "xxx"
database-names=[
"bocai"
]
table-names=[
"bocai.access_log",
]

      startup.mode = "initial"
      snapshot.split.size = 8096
      snapshot.fetch.size = 1024
  plugin_output  = "my_access_log";
}

}

transform {
Sql {
query = "select id, createTime, requestMethod,requestTime from access_log"
result_table_name = "access_log"
plugin_input = "my_access_log"
}
}

`
If there're 10 tables to sync with specifying table columns. Shoud I need to add ten more sources and transforms for each table?
And for MySQL-CDC, each source acts as one mysql slaver or use the same mysql slave?

Thanks in advance!!!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions