Skip to main content
Version: Next

TableMerge

TableMerge transform plugin

Description​

TableMerge transform plugin for merge sharding-tables.

Options​

nametyperequireddefault valueDescription
databasestringnoSpecify new database name
schemastringnoSpecify new schema name
tablestringconditionalRequired only when table_transform is unset or empty; each table_transform entry still requires its own table.
table_match_regexstringno.*Regex of the table paths that the transform level rule applies to.
table_transformarrayno[]Per-table rules, see Multi-Table Transform. Exact table-path entries take precedence over the regex; an entry with only table_match_regex does not instantiate the global rule.
rule_match_modestringnoFIRST_MATCHHow multiple table_transform entries with the same table_path are evaluated. Omitting it rejects duplicate entries; FIRST_MATCH selects the first one.

Tables that match neither table_match_regex nor a table_transform entry are passed through unchanged. In the example below, source.shop does not match source.user_.*, so it keeps its original name while source.user_1 and source.user_2 are merged into user_db.user_all.

Examples​

Merge sharding-tables​

env {
parallelism = 1
job.mode = "STREAMING"
}

source {
MySQL-CDC {
plugin_output = "customers_mysql_cdc"

username = "root"
password = "123456"
table-names = ["source.user_1", "source.user_2", "source.shop"]
url = "jdbc:mysql://localhost:3306/source"
}
}

transform {
TableMerge {
plugin_input = "customers_mysql_cdc"
plugin_output = "trans_result"

table_match_regex = "source.user_.*"
database = "user_db"
table = "user_all"
}
}

sink {
Jdbc {
plugin_input = "trans_result"

driver="com.mysql.cj.jdbc.Driver"
url="jdbc:mysql://localhost:3306/sink"
user="myuser"
password="mypwd"

generate_sink_sql = true
database = "${database_name}"
table = "${table_name}"
primary_keys = ["${primary_key}"]

schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
data_save_mode = "APPEND_DATA"
}
}