JsonPath
JsonPath transform plugin
Description
Support use jsonpath select data
Options
| name | type | required | default value |
|---|---|---|---|
| columns | Array | Yes | |
| row_error_handle_way | Enum | No | FAIL |
common options [string]
Transform plugin common parameters, please refer to Transform Plugin for details
row_error_handle_way [Enum]
This option is used to specify the processing method when an error occurs in the row, the default value is FAIL.
- FAIL: When
FAILis selected, data format error will block and an exception will be thrown. - SKIP: When
SKIPis selected, data format error will skip this row data. - ROUTE_TO_TABLE: not implemented by the JsonPath transform yet. The value can be configured, but its actual behavior is identical to
FAIL: rows that fail to parse make the job fail and are not routed to an error table.
This applies to path-reading errors and recognized data-conversion failures when no
column policy is configured. ROUTE_TO_TABLE is not supported by JsonPath: it does
not route errors and fails when selected for a failing value. See
Configure error data handle way for precedence and examples.
columns [array]
option
| name | type | required | default value |
|---|---|---|---|
| src_field | String | Yes | |
| dest_field | String or Array | Yes | |
| path | String or Array | Yes | |
| dest_type | String or Array | No | String |
| column_error_handle_way | Enum | No |
src_field
the json source field you want to parse
Support SeatunnelDateType
- STRING
- BYTES
- ARRAY
- MAP
- ROW
dest_field
after use jsonpath output field
This can be a single field name or an array of field names when extracting multiple values from the same source field.
dest_type
the type of dest field
This can be a single type or an array of types. If omitted for a single output field, the default type is string.
path
Jsonpath
This can be a single JSONPath expression or an array of JSONPath expressions.
column_error_handle_way [Enum]
This option is used to specify the processing method when an error occurs in the column.
- FAIL: When
FAILis selected, data format error will block and an exception will be thrown. - SKIP: When
SKIPis selected, data format error will skip this column data. - SKIP_ROW: When
SKIP_ROWis selected, data format error will skip this row data.
These policies also apply to recognized data-conversion failures, such as an invalid
date or number. Column SKIP sets the destination field to null, not the whole row.
ROUTE_TO_TABLE is not supported. See
Configure error data handle way for policy precedence and limits.
Read Json Example
The data read from source is a table like this json:
{
"data": {
"c_string": "this is a string",
"c_boolean": true,
"c_integer": 42,
"c_float": 3.14,
"c_double": 3.14,
"c_decimal": 10.55,
"c_date": "2023-10-29",
"c_datetime": "16:12:43.459",
"c_array":["item1", "item2", "item3"],
"c_map_array": [{"c_string_1":"c_string_1","c_string_2":"c_string_2","c_string_3":"c_string_3"},{"c_string_1":"c_string_1","c_string_2":"c_string_2","c_string_3":"c_string_3"}]
}
}
Assuming we want to use JsonPath to extract properties.
transform {
JsonPath {
plugin_input = "fake"
plugin_output = "fake1"
columns = [
{
"src_field" = "data"
"path" = "$.data.c_string"
"dest_field" = "c1_string"
},
{
"src_field" = "data"
"path" = "$.data.c_boolean"
"dest_field" = "c1_boolean"
"dest_type" = "boolean"
},
{
"src_field" = "data"
"path" = "$.data.c_integer"
"dest_field" = "c1_integer"
"dest_type" = "int"
},
{
"src_field" = "data"
"path" = "$.data.c_float"
"dest_field" = "c1_float"
"dest_type" = "float"
},
{
"src_field" = "data"
"path" = "$.data.c_double"
"dest_field" = "c1_double"
"dest_type" = "double"
},
{
"src_field" = "data"
"path" = "$.data.c_decimal"
"dest_field" = "c1_decimal"
"dest_type" = "decimal(4,2)"
},
{
"src_field" = "data"
"path" = "$.data.c_date"
"dest_field" = "c1_date"
"dest_type" = "date"
},
{
"src_field" = "data"
"path" = "$.data.c_datetime"
"dest_field" = "c1_datetime"
"dest_type" = "time"
},
{
"src_field" = "data"
"path" = "$.data.c_array"
"dest_field" = "c1_array"
"dest_type" = "array<string>"
},
{
"src_field" = "data"
"path" = "$.data.c_map_array"
"dest_field" = "c1_map_array"
"dest_type" = "array<map<string, string>>"
}
]
}
}
The same result can be achieved with much simpler configuration using batch field extraction with array format:
transform {
JsonPath {
plugin_input = "fake"
plugin_output = "fake1"
columns = [
{
"src_field" = "data"
"path" = ["$.data.c_string", "$.data.c_boolean", "$.data.c_integer", "$.data.c_float", "$.data.c_double", "$.data.c_decimal", "$.data.c_date", "$.data.c_datetime", "$.data.c_array", "$.data.c_map_array"]
"dest_field" = ["c1_string", "c1_boolean", "c1_integer", "c1_float", "c1_double", "c1_decimal", "c1_date", "c1_datetime", "c1_array", "c1_map_array"]
"dest_type" = ["string", "boolean", "int", "float", "double", "decimal(4,2)", "date", "time", "array<string>", "array<map<string, string>>"]
}
]
}
}
Important: When using batch field extraction, path, dest_field, and dest_type must have the same number of items. If you omit dest_type, the transform uses the single default type string, so multiple output fields should provide a dest_type array explicitly.
Then the data result table fake1 will like this
| data | c1_string | c1_boolean | c1_integer | c1_float | c1_double | c1_decimal | c1_date | c1_datetime | c1_array | c1_map_array |
|---|---|---|---|---|---|---|---|---|---|---|
| too much content not to show | this is a string | true | 42 | 3.14 | 3.14 | 10.55 | 2023-10-29 | 16:12:43.459 | ["item1", "item2", "item3"] | [{"c_string_1": "c_string_1", "c_string_2": "c_string_2", "c_string_3": "c_string_3"}, {"c_string_1": "c_string_1", "c_string_2": "c_string_2", "c_string_3": "c_string_3"}] |
Read SeatunnelRow Example
Suppose a column in a row of data is of type SeatunnelRow and that the name of the column is col
| SeatunnelRow(col) | other | |
|---|---|---|
| name | age | .... |
| a | 18 | .... |
The JsonPath transform converts the values of seatunnel into an array,
transform {
JsonPath {
plugin_input = "fake"
plugin_output = "fake1"
row_error_handle_way = FAIL
columns = [
{
"src_field" = "col"
"path" = "$[0]"
"dest_field" = "name"
"dest_type" = "string"
},
{
"src_field" = "col"
"path" = "$[1]"
"dest_field" = "age"
"dest_type" = "int"
}
]
}
}
Then the data result table fake1 will like this
| col | other | name | age |
|---|---|---|---|
| ["a",18] | ... | a | 18 |
Configure error data handle way
You can configure row_error_handle_way and column_error_handle_way to handle abnormal data. Both are optional.
The policies apply to path-reading errors and recognized value-conversion failures
(numbers, dates/times, binary data and their nested conversions). Column SKIP
sets the failed destination field to null and continues processing other columns;
column SKIP_ROW discards the row. Without a column policy, row FAIL or SKIP
applies. An explicit column policy takes precedence, so column FAIL still fails
when row SKIP is configured. ROUTE_TO_TABLE does not route records in this
transform and propagates the failure when selected for the failing value.
Unsupported conversions, unexpected programming/configuration failures and fatal
JVM errors fail the task instead of being skipped. Recognized conversion failures
use JSONPATH_ERROR_CODE-07 with safe source/destination field identifiers, destination SQL
type and the generic data conversion failure category, not a typed cause category.
Invalid float_vector shapes and non-numeric vector elements currently use the
shared converter's unsupported-type error and are not skippable by these policies.
Conversion diagnostics and skip logs omit source
records, extracted values, path expressions and original exceptions, which can
contain private data, including literals in configured paths. Path-reading
diagnostics and skip logs include only safe source/destination field identifiers
and the exception type; field identifiers outside [A-Za-z_][A-Za-z0-9_]{0,63}
are shown as <redacted>. They omit the source JSON, configured path and raw
exception message. The zero-based column_index identifies the flattened output
field position, including fields expanded from an array-valued columns entry,
even when its name is redacted. For FAIL,
ErrorDataTransformException retains a sanitized
JsonPathException cause, not the original exception: preserving its message or
stack trace could re-expose input data or a private path literal. This keeps the
cause chain present but sacrifices the original cause subtype and stack detail.
Invalid configured paths also fail the task with a sanitized exception, even
under SKIP. Unexpected failures still propagate.
Skip an invalid converted value
For input {"amount":"invalid","description":"retained"} in json_data, this
configuration produces a null amount and preserves description:
transform {
JsonPath {
row_error_handle_way = FAIL
columns = [
{ src_field = "json_data", path = "$.amount", dest_field = "amount", dest_type = "int", column_error_handle_way = SKIP },
{ src_field = "json_data", path = "$.description", dest_field = "description", dest_type = "string" }
]
}
}
Skip error data rows
Configure to skip row data with exceptions in any column
transform {
JsonPath {
row_error_handle_way = SKIP
columns = [
{
"src_field" = "json_data"
"path" = "$.f1"
"dest_field" = "json_data_f1"
},
{
"src_field" = "json_data"
"path" = "$.f2"
"dest_field" = "json_data_f2"
}
]
}
}
Skip error data column
Configure only json_data_f1 column data exceptions to skip and fill in null values, other column data exceptions will continue to throw exception interrupt handlers
transform {
JsonPath {
row_error_handle_way = FAIL
columns = [
{
"src_field" = "json_data"
"path" = "$.f1"
"dest_field" = "json_data_f1"
"column_error_handle_way" = "SKIP"
},
{
"src_field" = "json_data"
"path" = "$.f2"
"dest_field" = "json_data_f2"
}
]
}
}
Skip the row for specified column error
Configure to skip the row of data only for json_data_f1 column data exceptions, and continue to throw exceptions to interrupt the handler for other column data exceptions
transform {
JsonPath {
row_error_handle_way = FAIL
columns = [
{
"src_field" = "json_data"
"path" = "$.f1"
"dest_field" = "json_data_f1"
"column_error_handle_way" = "SKIP_ROW"
},
{
"src_field" = "json_data"
"path" = "$.f2"
"dest_field" = "json_data_f2"
}
]
}
}
Changelog
- Add JsonPath Transform