Grok

Filter plugin : Grok

• Author: InterestingLab
• Homepage: https://interestinglab.github.io/seatunnel-docs
• Version: 1.0.0

Description

Parse arbitrary text into structured data with columns using Grok Pattern. Please have a look at available grok pattern.

You can also go to http://grokdebug.herokuapp.com to debug grok patterns if you need some hint for grok pattern syntax.

Options

name	type	required	default value
named_captures_only	boolean	no	true
pattern	string	yes	-
patterns_dir	string	no	-
source_field	string	no	raw_message
target_field	string	no	root

named_captures_only [boolean]

If true, only store named captures from grok.

pattern [string]

Grok pattern.

patterns_dir [string]

The directory of pattern files. seatunnel ships by default with a bunch of patterns, so you don’t necessarily need to configure this unless you want to add additional patterns.

source_field [string]

Source field.

target_field [string]

Target field.

Example

grok {
    source_field = "raw_message"
    pattern = "%{WORD:name} is %{WORD:gender}, %{NUMBER:age} years old and weighs %{NUMBER:weight} kilograms"
    target_field = "info_detail"
}

• Input

+----------------------------------------------------+
|raw_message                                         |
+----------------------------------------------------+
|gary is male, 25 years old and weighs 68.5 kilograms|
|gary is male, 25 years old and weighs 68.5 kilograms|
+----------------------------------------------------+

• Output

+----------------------------------------------------+------------------------------------------------------------+
|raw_message                                         |info_detail                                                 |
+----------------------------------------------------+------------------------------------------------------------+
|gary is male, 25 years old and weighs 68.5 kilograms|Map(age -> 25, gender -> male, name -> gary, weight -> 68.5)|
|gary is male, 25 years old and weighs 68.5 kilograms|Map(age -> 25, gender -> male, name -> gary, weight -> 68.5)|
+----------------------------------------------------+------------------------------------------------------------+