Skip to main content
Version: 2.2.0-beta

File

File source connector

Description​

Read data from local or hdfs file.

tip

Engine Supported and plugin name

  • Spark: File
  • Flink: File

Options​

nametyperequireddefault value
formatstringnojson
pathstringyes-
common-optionsstringyes-
format [string]​

Format for reading files, currently supports text, parquet, json, orc, csv.

path [string]​
  • If read data from hdfs , the file path should start with hdfs://
  • If read data from local , the file path should start with file://

common options [string]​

Source plugin common parameters, please refer to Source Plugin for details

Examples​

file {
path = "hdfs:///var/logs"
result_table_name = "access_log"
}
file {
path = "file:///var/logs"
result_table_name = "access_log"
}