Skip to main content
Version: 2.1.0

Sql

Transform plugin : Sql [Spark]

Description​

Use SQL to process data and support Spark's rich UDF functions

Options​

nametyperequireddefault value
sqlstringyes-
common-optionsstringno-

sql [string]​

SQL statement, the table name used in SQL is the result_table_name configured in the Source or Transform plugin

common options [string]​

Transform plugin common parameters, please refer to Transform Plugin for details

Examples​

sql {
sql = "select username, address from user_info",
}

Use the SQL plugin for field deletion. Only the username and address fields are reserved, and the remaining fields will be discarded. user_info is the result_table_name configured by the previous plugin

sql {
sql = "select substring(telephone, 0, 10) from user_info",
}

Use SQL plugin for data processing, use substring functions to intercept the telephone field

sql {
sql = "select avg(age) from user_info",
table_name = "user_info"
}

Use SQL plugin for data aggregation, use avg functions to perform aggregation operations on the original data set, and take out the average value of the age field