Skip to main content
Version: 2.3.2

SQL

SQL transform plugin

Description​

Use SQL to transform given input row.

SQL transform use memory SQL engine, we can via SQL functions and ability of SQL engine to implement the transform task.

Options​

nametyperequireddefault value
source_table_namestringyes-
result_table_namestringyes-
querystringyes-

source_table_name [string]​

The source table name, the query SQL table name must match this field.

query [string]​

The query SQL, it's a simple SQL supported base function and criteria filter operation. But the complex SQL unsupported yet, include: multi source table/rows JOIN and AGGREGATE operation and the like.

Example​

The data read from source is a table like this:

idnameage
1Joy Ding20
2May Ding21
3Kin Dom24
4Joy Dom22

We use SQL query to transform the source data like this:

transform {
Sql {
source_table_name = "fake"
result_table_name = "fake1"
query = "select id, concat(name, '_') as name, age+1 as age from fake where id>0"
}
}

Then the data in result table fake1 will update to

idnameage
1Joy Ding_21
2May Ding_22
3Kin Dom_25
4Joy Dom_23

Job Config Example​

env {
job.mode = "BATCH"
}

source {
FakeSource {
result_table_name = "fake"
row.num = 100
schema = {
fields {
id = "int"
name = "string"
age = "int"
}
}
}
}

transform {
Sql {
source_table_name = "fake"
result_table_name = "fake1"
query = "select id, concat(name, '_') as name, age+1 as age from fake where id>0"
}
}

sink {
Console {
source_table_name = "fake1"
}
}

Changelog​

new version​

  • Add SQL Transform Connector