S3

Output plugin : S3

• Author: InterestingLab
• Homepage: https://interestinglab.github.io/seatunnel-docs
• Version: 1.0.0

Description

Write Rows to AWS S3 storage.

Options

name	type	required	default value
options	object	no	-
partition_by	array	no	-
path	string	yes	-
path_time_format	string	no	yyyyMMddHHmmss
save_mode	string	no	error
format	string	no	json

options [object]

Custom parameters.

partition_by [array]

Partition the data based on the fields.

path [string]

File path on AWS S3 storage. Start with s3://, s3a:// or s3n://.

path_time_format [string]

If path contains time variables, such as xxxx-${now}, path_time_format can be used to specify the format of s3 path, default is yyyy.MM.dd. The commonly used time formats are listed below:

Symbol	Description
y	Year
M	Month
d	Day of month
H	Hour in day (0-23)
m	Minute in hour
s	Second in minute

The detailed time format syntax:Java SimpleDateFormat.

save_mode [string]

Save mode, supports overwrite, append, ignore and error. The detail of save_mode see save-modes.

format [string]

Output format, supports csv, json, parquet and text.

Example

s3 {
    path = "s3a://var/logs"
    format = "parquet"
}