Assert
Assert sink connector
Description
A flink sink plugin which can assert illegal data by user defined rules
Engine Supported and plugin name
- Spark:Assert
- Flink: Assert
Options
name | type | required | default value |
---|---|---|---|
rules | ConfigList | yes | - |
rules.field_name | string | yes | - |
rules.field_type | string | no | - |
rules.field_value | ConfigList | no | - |
rules.field_value.rule_type | string | no | - |
rules.field_value.rule_value | double | no | - |
rules [ConfigList]
Rule definition of user's available data. Each rule represents one field validation.
field_name [string]
field name(string)
field_type [string]
field type (string), e.g. string,boolean,byte,short,int,long,float,double,char,void,BigInteger,BigDecimal,Instant
field_value [ConfigList]
A list value rule define the data value validation
rule_type [string]
The following rules are supported for now
NOT_NULL, // value can't be null
MIN, // define the minimum value of data
MAX, // define the maximum value of data
MIN_LENGTH, // define the minimum string length of a string data
MAX_LENGTH // define the maximum string length of a string data
rule_value [double]
the value related to rule type
Example
the whole config obey with hocon
style
Assert {
rules =
[{
field_name = name
field_type = string
field_value = [
{
rule_type = NOT_NULL
},
{
rule_type = MIN_LENGTH
rule_value = 3
},
{
rule_type = MAX_LENGTH
rule_value = 5
}
]
},{
field_name = age
field_type = int
field_value = [
{
rule_type = NOT_NULL
},
{
rule_type = MIN
rule_value = 10
},
{
rule_type = MAX
rule_value = 20
}
]
}
]
}