Hdfs

Output plugin : Hdfs

• Author: InterestingLab
• Homepage: https://interestinglab.github.io/seatunnel-docs
• Version: 1.0.0

Description

Write Rows to HDFS.

Options

name	type	required	default value
options	object	no	-
partition_by	array	no	-
path	string	yes	-
path_time_format	string	no	yyyyMMddHHmmss
save_mode	string	no	error
format	string	no	json

options [object]

Custom parameters.

partition_by [array]

Partition the data based on the fields.

path [string]

File path on HDFS. Start with hdfs://.

path_time_format [string]

If path contains time variables, such as xxxx-${now}, path_time_format can be used to specify the format of HDFS path, default is yyyy.MM.dd. The commonly used time formats are listed below:

Symbol	Description
y	Year
M	Month
d	Day of month
H	Hour in day (0-23)
m	Minute in hour
s	Second in minute

The detailed time format syntax:Java SimpleDateFormat.

save_mode [string]

Save mode, supports overwrite, append, ignore and error. The detail of save_mode see save-modes.

format [string]

Output format, supports csv, json, parquet and text.

Example

hdfs {
    path = "hdfs:///var/logs-${now}"
    format = "json"
    path_time_format = "yyyy.MM.dd"
}