跳到主要内容
版本：Next

HdfsFile

Hdfs 文件数据源连接器

支持的引擎​

Spark
Flink
SeaTunnel Zeta

主要特性​

在 pollNext 调用中读取分片中的所有数据。读取的分片将保存在快照中。

描述​

从 hdfs 文件系统读取数据。

支持的数据源信息​

数据源支持的版本
HdfsFilehadoop 2.x 和 3.x

数据源选项​

名称类型是否必须默认值描述
pathstring是-源文件路径。
file_format_typestring是-我们支持以下文件类型：text csv parquet orc json excel xml binary。请注意，最终文件名将以文件格式的后缀结束，文本文件的后缀是 txt。
fs.defaultFSstring是-以 hdfs:// 开头的 hadoop 集群地址，例如：hdfs://hadoopcluster
read_columnslist否-数据源的读取列列表，用户可以使用它来实现字段投影。支持列投影的文件类型如下所示：[text,json,csv,orc,parquet,excel,xml]。提示：如果用户想在读取 text json csv 文件时使用此功能，必须配置 schema 选项。
hdfs_site_pathstring否-hdfs-site.xml 的路径，用于加载 namenodes 的 ha 配置
delimiter/field_delimiterstring否\001字段分隔符，用于告诉连接器在读取文本文件时如何分割字段。默认 \001，与 hive 的默认分隔符相同
parse_partition_from_pathboolean否true控制是否从文件路径解析分区键和值。例如，如果您从路径 hdfs://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26 读取文件。文件中的每条记录数据都将添加这两个字段：[name:tyrantlucifer,age:26]。提示：不要在 schema 选项中定义分区字段。
date_formatstring否yyyy-MM-dd日期类型格式，用于告诉连接器如何将字符串转换为日期，支持以下格式：yyyy-MM-dd yyyy.MM.dd yyyy/MM/dd 默认 yyyy-MM-dd。日期类型格式，用于告诉连接器如何将字符串转换为日期，支持以下格式：yyyy-MM-dd yyyy.MM.dd yyyy/MM/dd 默认 yyyy-MM-dd
datetime_formatstring否yyyy-MM-dd HH:mm:ss日期时间类型格式，用于告诉连接器如何将字符串转换为日期时间，支持以下格式：yyyy-MM-dd HH:mm:ss yyyy.MM.dd HH:mm:ss yyyy/MM/dd HH:mm:ss yyyyMMddHHmmss。默认 yyyy-MM-dd HH:mm:ss
time_formatstring否HH:mm:ss时间类型格式，用于告诉连接器如何将字符串转换为时间，支持以下格式：HH:mm:ss HH:mm:ss.SSS。默认 HH:mm:ss
remote_userstring否-用于连接到 hadoop 登录名的登录用户。它旨在用于 RPC 中的远程用户，不会有任何凭据。
krb5_pathstring否/etc/krb5.confkerberos 的 krb5 路径
kerberos_principalstring否-kerberos 的主体
kerberos_keytab_pathstring否-kerberos 的 keytab 路径
skip_header_row_numberlong否0跳过前几行，但仅适用于 txt 和 csv。例如，设置如下：skip_header_row_number = 2。然后 Seatunnel 将跳过源文件的前 2 行
schemaconfig否-上游数据的 schema 字段
sheet_namestring否-读取工作簿的工作表，仅在 file_format 为 excel 时使用。
xml_row_tagstring否-指定 XML 文件中数据行的标签名称，仅在 file_format 为 xml 时使用。
xml_use_attr_formatboolean否-指定是否使用标签属性格式处理数据，仅在 file_format 为 xml 时使用。
csv_use_header_lineboolean否false是否使用标题行解析文件，仅在 file_format 为 csv 且文件包含符合 RFC 4180 的标题行时使用
file_filter_patternstring否过滤模式，用于过滤文件。
filename_extensionstring否-过滤文件扩展名，用于过滤具有特定扩展名的文件。示例：csv .txt json .xml。
compress_codecstring否none文件的压缩编解码器
archive_compress_codecstring否none
encodingstring否UTF-8
null_formatstring否-仅在 file_format_type 为 text 时使用。null_format 定义哪些字符串可以表示为 null。例如：\N
binary_chunk_sizeint否1024仅在 file_format_type 为 binary 时使用。读取二进制文件的块大小（以字节为单位）。默认为 1024 字节。较大的值可能会提高大文件的性能，但会使用更多内存。
binary_complete_file_modeboolean否false仅在 file_format_type 为 binary 时使用。是否将完整文件作为单个块读取，而不是分割成块。启用时，整个文件内容将一次性读入内存。默认为 false。
common-options否-数据源插件通用参数，请参阅 数据源通用选项 了解详情。

delimiter/field_delimiter [string]​

delimiter 参数将在 2.3.5 版本后弃用，请使用 field_delimiter 代替。

file_filter_pattern [string]​

过滤模式，用于过滤文件。

该模式遵循标准正则表达式。详情请参考 https://en.wikipedia.org/wiki/Regular_expression。 以下是一些示例。

文件结构示例：

/data/seatunnel/20241001/report.txt
/data/seatunnel/20241007/abch202410.csv
/data/seatunnel/20241002/abcg202410.csv
/data/seatunnel/20241005/old_data.csv
/data/seatunnel/20241012/logo.png

匹配规则示例：

示例 1：匹配所有 .txt 文件，正则表达式：

/data/seatunnel/20241001/.*\.txt

此示例匹配的结果是：

/data/seatunnel/20241001/report.txt

示例 2：匹配所有以 abc 开头的文件，正则表达式：

/data/seatunnel/20241002/abc.*

此示例匹配的结果是：

/data/seatunnel/20241007/abch202410.csv
/data/seatunnel/20241002/abcg202410.csv

示例 3：匹配所有以 abc 开头，且第四个字符是 h 或 g 的文件，正则表达式：

/data/seatunnel/20241007/abc[h,g].*

此示例匹配的结果是：

/data/seatunnel/20241007/abch202410.csv

示例 4：匹配以 202410 开头的第三级文件夹和以 .csv 结尾的文件，正则表达式：

/data/seatunnel/202410\d*/.*\.csv

此示例匹配的结果是：

/data/seatunnel/20241007/abch202410.csv
/data/seatunnel/20241002/abcg202410.csv
/data/seatunnel/20241005/old_data.csv

compress_codec [string]​

文件的压缩编解码器及其支持的详细信息如下所示：

  • txt: lzo none
  • json: lzo none
  • csv: lzo none
  • orc/parquet: 自动识别压缩类型，无需额外设置。

archive_compress_codec [string]​

归档文件的压缩编解码器及其支持的详细信息如下所示：

archive_compress_codecfile_formatarchive_compress_suffix
ZIPtxt,json,excel,xml.zip
TARtxt,json,excel,xml.tar
TAR_GZtxt,json,excel,xml.tar.gz
GZtxt,json,excel,xml.gz
NONEall.*

注意：gz 压缩的 excel 文件需要压缩原始文件或指定文件后缀，例如 e2e.xls ->e2e_test.xls.gz

encoding [string]​

仅在 file_format_type 为 json,text,csv,xml 时使用。 要读取的文件的编码。此参数将由 Charset.forName(encoding) 解析。

binary_chunk_size [int]​

仅在 file_format_type 为 binary 时使用。

读取二进制文件的块大小（以字节为单位）。默认为 1024 字节。较大的值可能会提高大文件的性能，但会使用更多内存。

binary_complete_file_mode [boolean]​

仅在 file_format_type 为 binary 时使用。

是否将完整文件作为单个块读取，而不是分割成块。启用时，整个文件内容将一次性读入内存。默认为 false。

提示​

如果您使用 spark/flink，为了使用此连接器，您必须确保您的 spark/flink 集群已经集成了 hadoop。测试过的 hadoop 版本是 2.x。如果您使用 SeaTunnel Engine，则在下载和安装 SeaTunnel Engine 时会自动集成 hadoop jar。您可以检查 ${SEATUNNEL_HOME}/lib 下的 jar 包来确认这一点。

任务示例​

简单示例​

此示例定义了一个 SeaTunnel 同步任务，从 Hdfs 读取数据并将其发送到 Hdfs。

# 定义运行时环境
env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  HdfsFile {
  schema {
    fields {
      name = string
      age = int
    }
  }
  path = "/apps/hive/demo/student"
  file_format_type = "json"
  fs.defaultFS = "hdfs://namenode001"
  }
  # 如果您想获取有关如何配置 seatunnel 的更多信息和查看完整的数据源插件列表，
  # 请访问 https://seatunnel.apache.org/docs/connector-v2/source
}

transform {
  # 如果您想获取有关如何配置 seatunnel 的更多信息和查看完整的转换插件列表，
    # 请访问 https://seatunnel.apache.org/docs/transform-v2
}

sink {
    HdfsFile {
      fs.defaultFS = "hdfs://hadoopcluster"
      path = "/tmp/hive/warehouse/test2"
      file_format_type = "orc"
    }
  # 如果您想获取有关如何配置 seatunnel 的更多信息和查看完整的接收器插件列表，
  # 请访问 https://seatunnel.apache.org/docs/connector-v2/sink
}

过滤文件​

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  HdfsFile {
    path = "/apps/hive/demo/student"
    file_format_type = "json"
    fs.defaultFS = "hdfs://namenode001"
    // 文件示例 abcD2024.csv
    file_filter_pattern = "abc[DX]*.*"
  }
}

sink {
  Console {
  }
}

变更日志​

Change Log
ChangeCommitVersion
[Improve][Connector-V2] Support maxcompute sink writer with timestamp field type (#9234)https://github.com/apache/seatunnel/commit/a513c495e3dev
[improve] update file connectors config (#9034)https://github.com/apache/seatunnel/commit/8041d59dc22.3.11
[Improve][File] Add row_delimiter options into text file sink (#9017)https://github.com/apache/seatunnel/commit/92aa855a342.3.11
Revert " [improve] update localfile connector config" (#9018)https://github.com/apache/seatunnel/commit/cdc79e13ad2.3.10
[improve] update localfile connector config (#8765)https://github.com/apache/seatunnel/commit/def369a85f2.3.10
[Feature][Connector-V2] Add filename_extension parameter for read/write file (#8769)https://github.com/apache/seatunnel/commit/78b23c0ef52.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Feature][Connector-V2] Support create emtpy file when no data (#8543)https://github.com/apache/seatunnel/commit/275db789182.3.10
[Feature][Connector-V2] Support single file mode in file sink (#8518)https://github.com/apache/seatunnel/commit/e893deed502.3.10
[Feature][File] Support config null format for text file read (#8109)https://github.com/apache/seatunnel/commit/2dbf02df472.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve][Connector-V2] Support read archive compress file (#7633)https://github.com/apache/seatunnel/commit/3f98cd8a162.3.8
[Improve][Files] Support write fixed/timestamp as int96 of parquet (#6971)https://github.com/apache/seatunnel/commit/1a48a9c4932.3.6
Add support for XML file type to various file connectors such as SFTP, FTP, LocalFile, HdfsFile, and more. (#6327)https://github.com/apache/seatunnel/commit/ec533ecd9a2.3.5
[Refactor][File Connector] Put Multiple Table File API to File Base Module (#6033)https://github.com/apache/seatunnel/commit/c324d663b42.3.4
[Improve][connector-file] unifiy option between file source/sink and update document (#5680)https://github.com/apache/seatunnel/commit/8d87cf8fc42.3.4
[Feature] Support LZO compress on File Read (#5083)https://github.com/apache/seatunnel/commit/a4a19010962.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Feature][File Connector]optionrule FILE_FORMAT_TYPE is text/csv ,add parameter BaseSinkConfig.ENABLE_HEADER_WRITE: #5566 (#5567)https://github.com/apache/seatunnel/commit/0e02db768d2.3.4
[Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. (#5153)https://github.com/apache/seatunnel/commit/a3c13e59eb2.3.3
[Feature][ConnectorV2]add file excel sink and source (#4164)https://github.com/apache/seatunnel/commit/e3b97ae5d22.3.2
Change file type to file_format_type in file source/sink (#4249)https://github.com/apache/seatunnel/commit/973a2fae3c2.3.1
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector-V2][File] Support compress (#3899)https://github.com/apache/seatunnel/commit/55602f6b1c2.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Improve][Connector-V2][File] Improve file connector option rule and document (#3812)https://github.com/apache/seatunnel/commit/bd760776692.3.1
[Feature][Shade] Add seatunnel hadoop3 uber (#3755)https://github.com/apache/seatunnel/commit/5a024bdf8f2.3.0
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)https://github.com/apache/seatunnel/commit/226dc6a1192.3.0
[Feature][Connector-V2][File] Add option and factory for file connectors (#3375)https://github.com/apache/seatunnel/commit/db286e86312.3.0
[Improve][Connector] Improve write parquet (#2943)https://github.com/apache/seatunnel/commit/8fd966394b2.3.0-beta
[Fix][Connector-V2] Fix HiveSource Connector read orc table error (#2845)https://github.com/apache/seatunnel/commit/61720306e72.2.0-beta
[Improve][Connector-V2] Improve read parquet (#2841)https://github.com/apache/seatunnel/commit/e19bc82f9b2.2.0-beta
[Improve][Connector-V2] Refactor hdfs file sink connector code structure (#2701)https://github.com/apache/seatunnel/commit/6129c025672.2.0-beta
[#2606]Dependency management split (#2630)https://github.com/apache/seatunnel/commit/fc047be69b2.2.0-beta
[chore][connector-common] Rename SeatunnelSchema to SeaTunnelSchema (#2538)https://github.com/apache/seatunnel/commit/7dc2a273882.2.0-beta
[Feature][Connector-V2] Add hdfs file json support (#2451)https://github.com/apache/seatunnel/commit/84f6b17c152.2.0-beta
[Improve][Connector-V2] Refactor the package of hdfs file connector (#2402)https://github.com/apache/seatunnel/commit/87d0624c5b2.2.0-beta
[Feature][Connector-V2] Add hdfs file source connector (#2420)https://github.com/apache/seatunnel/commit/4fb6f2a2162.2.0-beta
[Feature][Connector-V2] Add json file sink & json format (#2385)https://github.com/apache/seatunnel/commit/dd68c06b0a2.2.0-beta
[Imporve][Connector-V2] Remove redundant type judge logic because of pr #2315 (#2370)https://github.com/apache/seatunnel/commit/42e8c25e502.2.0-beta
[Feature][Connector-V2] Support orc file format in file connector (#2369)https://github.com/apache/seatunnel/commit/f44fe1e0332.2.0-beta
[improve][UT] Upgrade junit to 5.+ (#2305)https://github.com/apache/seatunnel/commit/362319ff3e2.2.0-beta
[Connector-V2] Add parquet writer in file connector (#2273)https://github.com/apache/seatunnel/commit/c95cc72cfa2.2.0-beta
[checkstyle] Improved validation scope of MagicNumber (#2194)https://github.com/apache/seatunnel/commit/6d08b5f3692.2.0-beta
[Connector-V2] Add Hive sink connector v2 (#2158)https://github.com/apache/seatunnel/commit/23ad4ee7352.2.0-beta
[Connector-V2] Add File Sink Connector (#2117)https://github.com/apache/seatunnel/commit/e2283da64f2.2.0-beta