跳到主要内容
版本:3.1.0

Hbase

Hbase 源连接器

引擎支持​

Spark
Flink
SeaTunnel Zeta

描述​

从 Apache HBase 表读取数据。支持普通扫描、行键范围扫描、时间戳范围扫描、二进制行键、自定义命名空间和并行批读取。

本连接器以批处理模式运行,读取的是扫描范围在任务启动时刻的快照状态,并非 CDC 源;扫描启动后发生的新写入不会进入结果。

主要特性​

选项​

名称类型必填默认值描述
zookeeper_quorumstring是-HBase ZooKeeper 地址,例如 hadoop001:2181,hadoop002:2181。
tablestring是-要扫描的 HBase 表。自定义 namespace 请使用 namespace:table。
schemaconfig是-SeaTunnel 表结构。行键写作 rowkey,普通单元格写作 列簇:列名。
hbase_extra_configconfig否-额外的 HBase 或 Hadoop 客户端配置。
cachingint否-1每次 RPC 获取的行数。-1 表示使用 HBase 客户端默认值。
batchint否-1每次 RPC 返回的最大单元格数量。-1 表示使用 HBase 客户端默认值。
cache_blocksboolean否false扫描结果是否写入 HBase block cache。
is_binary_rowkeyboolean否false是否把行键字段按二进制字节处理。
start_rowkeystring否-范围扫描的起始行键。
end_rowkeystring否-范围扫描的结束行键。
start_row_inclusiveboolean否true扫描结果是否包含 start_rowkey。
end_row_inclusiveboolean否false扫描结果是否包含 end_rowkey。
start_timestamplong否-时间范围扫描的起始时间戳,包含该时间。
end_timestamplong否-时间范围扫描的结束时间戳,不包含该时间。
common-options否-Source 插件通用参数,例如 plugin_output。

zookeeper_quorum [string]​

HBase 的 zookeeper 集群主机,例如:“hadoop001:2181,hadoop002:2181,hadoop003:2181”

table [string]​

要读取的表名,例如:“seatunnel” 如果表在自定义 namespace 下,请使用 namespace:table 形式(如 ns1:seatunnel_test);未填写 namespace 时,SeaTunnel 会使用 HBase 的默认命名空间 default。

schema [config]​

HBase 使用字节数组进行存储。因此,您需要为表中的每一列配置数据类型。 行键列使用 rowkey,HBase 单元格使用 列簇:列名 形式,例如 info:name。 更多信息请参阅:模式声明指南。

hbase_extra_config [config]​

HBase 的额外配置

caching​

caching 参数用于设置在扫描过程中一次从服务器端获取的行数。这可以减少客户端与服务器之间的往返次数,从而提高扫描效率。默认值:-1

batch​

batch 参数用于设置在扫描过程中每次返回的最大单元格数。这对于处理有很多列的行特别有用,可以避免一次性返回过多数据,从而节省内存并提高性能。

cache_blocks​

cache_blocks 参数用于设置在扫描过程中是否缓存数据块。默认情况下,HBase 会在扫描时将数据块缓存到块缓存中。如果设置为 false,则在扫描过程中不会缓存数据块,从而减少内存使用。在 SeaTunnel 中默认值为 false。

is_binary_rowkey​

HBase 的行键既可以是文本字符串,也可以是二进制数据。在 SeaTunnel 中,行键默认设置为文本字符串(即 is_binary_rowkey 默认值为 false)

start_rowkey​

扫描起始行

end_rowkey​

扫描结束行

start_row_inclusive​

设置扫描范围是否包含起始行。当设置为 true 时,扫描结果将包含起始行。默认值: true (包含)。

注意: 在大多数情况下,应保持默认值 (true)。仅当您有特定需求需要排除起始行时才修改此参数。

end_row_inclusive​

设置扫描范围是否包含结束行。当设置为 false 时,扫描结果将不包含结束行,遵循左闭右开的区间约定 [start, end)。默认值: false (不包含)。

注意: 在大多数情况下,应保持默认值 (false),这遵循 HBase 标准的左闭右开区间约定。仅当您需要在扫描结果中包含结束行时才修改此参数。

重要提示: 使用多个分片并行读取时,这两个参数的组合对数据完整性很重要:

  • 默认配置 (start_row_inclusive=true, end_row_inclusive=false): 推荐配置,可以确保跨分片时不丢失数据、不产生重复数据。每个分片遵循 [start, end) 左闭右开区间约定。
  • 都设置为 false (start_row_inclusive=false, end_row_inclusive=false): 可能导致数据丢失,因为边界行会被所有分片排除在外。
  • 都设置为 true (start_row_inclusive=true, end_row_inclusive=true): 可能导致数据重复,因为边界行会被相邻的多个分片重复包含。

start_timestamp​

时间范围扫描的起始时间戳(包含)。单位为毫秒(epoch)。时间范围遵循 [start, end) 左闭右开约定。如果只设置 start_timestamp,则最大值视为无限上界。

end_timestamp​

时间范围扫描的结束时间戳(不包含)。单位为毫秒(epoch)。时间范围遵循 [start, end) 左闭右开约定。如果只设置 end_timestamp,则最小值视为无限下界。

说明:

  • start_timestamp 必须大于等于 0,end_timestamp 必须大于 0;若两者同时配置,需要满足 start_timestamp < end_timestamp(遵循 [start, end) 约定,start_timestamp == end_timestamp 将导致空扫描)。
  • 当 start_rowkey / end_rowkey 与 start_timestamp / end_timestamp 同时配置时,会同时应用行键范围与时间范围限制,最终返回两者的交集。

常用选项​

Source 插件常用参数,具体请参考 Source 常用选项

示例​

按行键和时间范围读取​

source {
Hbase {
zookeeper_quorum = "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table = "seatunnel_test"
caching = 1000
batch = 100
cache_blocks = false
is_binary_rowkey = false
start_rowkey = "B"
end_rowkey = "C"
start_timestamp = 1700000000000
end_timestamp = 1700003600000
schema = {
columns = [
{
name = "rowkey"
type = string
},
{
name = "columnFamily1:column1"
type = boolean
},
{
name = "columnFamily1:column2"
type = double
},
{
name = "columnFamily2:column1"
type = bigint
}
]
}
}
}

读取命名空间下的表​

source {
Hbase {
zookeeper_quorum = "hbase_e2e:2181"
table = "ns1:seatunnel_test"
schema = {
columns = [
{ name = rowkey, type = string },
{ name = "info:name", type = string }
]
}
}
}

读取二进制行键​

source {
Hbase {
zookeeper_quorum = "hbase_e2e:2181"
table = "binary_rowkey_table"
is_binary_rowkey = true
caching = 500
batch = 100
schema = {
columns = [
{ name = rowkey, type = bytes },
{ name = "info:name", type = string },
{ name = "info:score", type = double }
]
}
}
}

当 is_binary_rowkey = true 时,请在 schema 中把行键列声明为 bytes,并在后续的 transform 节点里自行解码。

Kerberos 示例​

备注:

  • connector-hbase 不会解析 krb5_path / kerberos_principal / kerberos_keytab_path。
  • 需要在运行环境中提前完成 Kerberos 登录并保证 krb5.conf 可被 JVM 访问(例如 kinit -kt ... 或 JVM -Djava.security.krb5.conf=...),同时将 HBase/Hadoop 的安全配置写入 hbase_extra_config。
source {
Hbase {
zookeeper_quorum = "zk1:2181,zk2:2181,zk3:2181"
table = "source_table"
caching = 1000
batch = 200
cache_blocks = false
is_binary_rowkey = false

# HBase安全配置
hbase_extra_config = {
"hbase.security.authentication" = "kerberos"
"hadoop.security.authentication" = "kerberos"
"hbase.master.kerberos.principal" = "hbase/_HOST@REALM"
"hbase.regionserver.kerberos.principal" = "hbase/_HOST@REALM"
"hbase.rpc.protection" = "authentication"
"hbase.zookeeper.useSasl" = "false"
}

schema = {
columns = [
{ name = "rowkey", type = string },
{ name = "info:name", type = string },
{ name = "info:score", type = string }
]
}
}
}

变更日志​

Change Log
ChangeCommitVersion
[Improve][Common] Add HashUtils.bucketIndex for hash-to-bucket routing (#11987)https://github.com/apache/seatunnel/commit/be53a1d3d13.0.0
[Improve][Connector-V2][HBase] Migrate timestamp validation to OptionRule (#11803)https://github.com/apache/seatunnel/commit/0a5708bf643.0.0
[Fix][connector-hbase] Fix namespace handling for HBase source (#10295)https://github.com/apache/seatunnel/commit/d722474bc32.3.13
[Feature][Connector-V2][HBase] Support time-range scan with min/max timestamp in HBaseSource (#10318)https://github.com/apache/seatunnel/commit/402291d3592.3.13
[Fix][Connector-V2][Hbase] Fix ERROR_WHEN_DATA_EXISTS NPE on empty table (#10336)https://github.com/apache/seatunnel/commit/9d58bc01ac2.3.13
[Improve][Connector-V2][HBase] Support DATE/TIME/TIMESTAMP/DECIMAL in sink and fix DECIMAL deserialization (#10291)https://github.com/apache/seatunnel/commit/2cc680fe652.3.13
[Fix[Connector-V2][Hbase] Avoid duplicate split assignment on restore (#10310)https://github.com/apache/seatunnel/commit/75bc71beb82.3.13
[Fix][Connector-V2][Hbase] Fix HBase sink binary rowkey handling (#10300)https://github.com/apache/seatunnel/commit/84b039d4fa2.3.13
[Fix][Connector-V2][Hbase] Fix source reader only scanning first split (#10287)https://github.com/apache/seatunnel/commit/d393d2a82f2.3.13
[Fix][Connector-V2][HBase] Ensure fully qualified table name is used in tableExists method and add unit tests (#10126)https://github.com/apache/seatunnel/commit/53c50f39442.3.13
[Improve][Connector-V2][HBase] Support configurable range scan boundary inclusion policies (#10011)https://github.com/apache/seatunnel/commit/40bf6560f52.3.13
[Feature][Connector-V2] Support row range boundaries for HBaseSource (#9983)https://github.com/apache/seatunnel/commit/d7b8f37b412.3.13
[Fix][Core] Add shade module for apache commons lang3 (#9895)https://github.com/apache/seatunnel/commit/abb9124b052.3.13
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Improve] hbase options (#8923)https://github.com/apache/seatunnel/commit/b6a702b58f2.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)https://github.com/apache/seatunnel/commit/e2fb6795772.3.8
[Feature][Connector-V2][Hbase] implement hbase catalog (#7516)https://github.com/apache/seatunnel/commit/b978792cb12.3.8
[Feature][Connector-V2] Support multi-table sink feature for HBase (#7169)https://github.com/apache/seatunnel/commit/025fa3bb882.3.8
[hotfix][connector-v2-hbase]fix and optimize hbase source problem (#7148)https://github.com/apache/seatunnel/commit/34a6b8e9f62.3.7
[Improve][hbase] The specified column is written to the specified column family (#5234)https://github.com/apache/seatunnel/commit/49d397c61d2.3.6
[feature][connector-v2-hbase-sink] Support Connector v2 HBase sink TTL data writing (#7116)https://github.com/apache/seatunnel/commit/adafd802552.3.6
[E2E][HBase]Refactor hbase e2e (#6859)https://github.com/apache/seatunnel/commit/1da9bd6ce42.3.6
[Connector]Add hbase source connector (#6348)https://github.com/apache/seatunnel/commit/f108a5e6582.3.6
[Feature][HbaseSink]support array data. (#6100)https://github.com/apache/seatunnel/commit/b5920147662.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Hotfix][Connector-v2][HbaseSink]Fix default timestamp (#4958)https://github.com/apache/seatunnel/commit/3d8f3bf9022.3.3
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector-V2][Hbase] Introduce hbase sink connector (#4049)https://github.com/apache/seatunnel/commit/68bda94a4c2.3.1