Common Options
Common parameters of transform plugins
This transform both supported by engine Spark and Flink.
Transform Plugin common parameters
- Spark
- Flink
name | type | required | default value |
---|---|---|---|
source_table_name | string | no | - |
result_table_name | string | no | - |
name | type | required | default value |
---|---|---|---|
source_table_name | string | no | - |
result_table_name | string | no | - |
field_name | string | no | - |
field_name [string]
When the data is obtained from the upper-level plugin, you can specify the name of the obtained field, which is convenient for use in subsequent sql plugins.
source_table_name [string]
When source_table_name
is not specified, the current plug-in processes the data set (dataset)
output by the previous plug-in in the configuration file;
When source_table_name
is specified, the current plugin is processing the data set corresponding to this parameter.
result_table_name [string]
When result_table_name
is not specified, the data processed by this plugin will not be registered as a data set that can be directly accessed by other plugins, or called a temporary table (table)
;
When result_table_name
is specified, the data processed by this plugin will be registered as a data set (dataset)
that can be directly accessed by other plugins, or called a temporary table (table)
. The dataset registered here can be directly accessed by other plugins by specifying source_table_name
.
Examples
- Spark
- Flink
split {
source_table_name = "source_view_table"
source_field = "message"
delimiter = "&"
fields = ["field1", "field2"]
result_table_name = "result_view_table"
}
The
Split
plugin will process the data in the temporary tablesource_view_table
and register the processing result as a temporary table namedresult_view_table
. This temporary table can be used by any subsequentFilter
orOutput
plugin by specifyingsource_table_name
.
split {
source_field = "message"
delimiter = "&"
fields = ["field1", "field2"]
}
Note: If
source_table_name
is not configured, output the processing result of the lastTransform
plugin in the configuration file
source {
FakeSourceStream {
result_table_name = "fake_1"
field_name = "name,age"
}
FakeSourceStream {
result_table_name = "fake_2"
field_name = "name,age"
}
}
transform {
sql {
source_table_name = "fake_1"
sql = "select name from fake_1"
result_table_name = "fake_name"
}
}
If
source_table_name
is not specified, the sql plugin will process the data offake_2
, and if it is set tofake_1
, it will process the data offake_1
.