跳到主要内容
版本:2.3.0

SftpFile

Sftp file source connector

Description

Read data from sftp file server.

Key features

Options

nametyperequireddefault value
hoststringyes-
portintyes-
userstringyes-
passwordstringyes-
pathstringyes-
typestringyes-
delimiterstringno\001
parse_partition_from_pathbooleannotrue
date_formatstringnoyyyy-MM-dd
datetime_formatstringnoyyyy-MM-dd HH:mm:ss
time_formatstringnoHH:mm:ss
schemaconfigno-
common-optionsno-

host [string]

The target sftp host is required

port [int]

The target sftp port is required

username [string]

The target sftp username is required

password [string]

The target sftp password is required

path [string]

The source file path.

delimiter [string]

Field delimiter, used to tell connector how to slice and dice fields when reading text files

default \001, the same as hive's default delimiter

parse_partition_from_path [boolean]

Control whether parse the partition keys and values from file path

For example if you read a file from path sftp://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26

Every record data from file will be added these two fields:

nameage
tyrantlucifer26

Tips: Do not define partition fields in schema option

date_format [string]

Date type format, used to tell connector how to convert string to date, supported as the following formats:

yyyy-MM-dd yyyy.MM.dd yyyy/MM/dd

default yyyy-MM-dd

datetime_format [string]

Datetime type format, used to tell connector how to convert string to datetime, supported as the following formats:

yyyy-MM-dd HH:mm:ss yyyy.MM.dd HH:mm:ss yyyy/MM/dd HH:mm:ss yyyyMMddHHmmss

default yyyy-MM-dd HH:mm:ss

time_format [string]

Time type format, used to tell connector how to convert string to time, supported as the following formats:

HH:mm:ss HH:mm:ss.SSS

default HH:mm:ss

schema [config]

The schema information of upstream data.

type [string]

File type, supported as the following file types:

text csv json

If you assign file type to json, you should also assign schema option to tell connector how to parse data to the row you want.

For example:

upstream data is the following:


{"code": 200, "data": "get success", "success": true}

you should assign schema as the following:


schema {
fields {
code = int
data = string
success = boolean
}
}

connector will generate data as the following:

codedatasuccess
200get successtrue

If you assign file type to text csv, you can choose to specify the schema information or not.

For example, upstream data is the following:


tyrantlucifer#26#male

If you do not assign data schema connector will treat the upstream data as the following:

content
tyrantlucifer#26#male

If you assign data schema, you should also assign the option delimiter too except CSV file type

you should assign schema and delimiter as the following:


delimiter = "#"
schema {
fields {
name = string
age = int
gender = string
}
}

connector will generate data as the following:

nameagegender
tyrantlucifer26male

common options

Source plugin common parameters, please refer to Source Common Options for details.

Example


SftpFile {
path = "/tmp/seatunnel/sink/text"
host = "192.168.31.48"
port = 21
user = tyrantlucifer
password = tianchao
type = "text"
schema = {
name = string
age = int
}
delimiter = "#"
}

Changelog

2.3.0 2022-12-30

  • Add SftpFile Source Connector