Hbase
Hbase 接收器连接器
引擎支持
Spark
Flink
SeaTunnel Zeta
描述
将 SeaTunnel 数据写入 Apache HBase。支持创建或复用目标表、写入单表或多表、把字段映射到一个或多个列簇,并可控制空值、行键、WAL、时间戳和已有数据的处理方式。
主要特性
选项
| 名称 | 类型 | 是否必须 | 默认值 | 描述 |
|---|---|---|---|---|
| zookeeper_quorum | string | 是 | - | HBase ZooKeeper 地址,例如 hadoop001:2181,hadoop002:2181。 |
| table | string | 是 | - | 目标 HBase 表。多表写入时可使用 ${table_name} 等占位符。 |
| rowkey_column | list | 是 | - | 用来组成 HBase rowkey 的上游字段名。 |
| family_name | config | 是 | - | 上游字段到 HBase 列簇的映射,也可以用 all_columns 统一写入一个列簇。 |
| rowkey_delimiter | string | 否 | "" | 多个字段组成 rowkey 时使用的连接符。 |
| version_column | string | 否 | - | 用作 HBase 单元格时间戳的上游字段。 |
| null_mode | string | 否 | skip | null 值写入方式,支持 skip 和 empty。 |
| wal_write | boolean | 否 | false | 写入时是否记录 HBase WAL。 |
| write_buffer_size | int | 否 | 8 1024 1024 | HBase 客户端写入缓冲区大小,单位字节。 |
| encoding | string | 否 | utf8 | STRING/DECIMAL/DATE/TIME/TIMESTAMP/ARRAY 的编码,支持 utf8 和 gbk。 |
| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 写入前如何处理目标表结构。 |
| data_save_mode | enum | 否 | APPEND_DATA | 写入前如何处理目标端已有数据。 |
| hbase_extra_config | config | 否 | - | 额外的 HBase 或 Hadoop 客户端配置。 |
| ttl | long | 否 | -1 | 写入 HBase 单元格的过期时间(毫秒);-1 表示永不过期。 |
| multi_table_sink_replica | int | 否 | 1 | 多表写入时每张表对应的 Sink Writer 副本数。 |
| common-options | 否 | - | Sink 插件通用参数,例如 plugin_input。 |
zookeeper_quorum [string]
HBase 的 zookeeper 集群主机,示例: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table [string]
要写入的表名, 例如: "seatunnel"
如果表在自定义 namespace 下,请使用 namespace:table 形式(如 ns1:seatunnel_test);未填写 namespace 时,SeaTunnel 会写入到 HBase 默认命名空间 default。
rowkey_column [list]
行键的列名列表, 例如: ["id", "uuid"]
family_name [config]
字段的列簇名称映射。例如,上游的行如下所示:
| id | name | age |
|---|---|---|
| 1 | tyrantlucifer | 27 |
id 作为行键,其他字段可以写入不同列簇。例如:
family_name { name = "info1" age = "info2" }
这表示 name 写入列簇 info1,age 写入列簇 info2。
如果要将其他字段写入同一列簇,可以分配
family_name { all_columns = "info" }
这表示所有字段都会写入列簇 info。
rowkey_delimiter [string]
连接多行键的分隔符,默认 ""
version_column [string]
版本列名称,您可以使用它来分配 hbase 记录的时间戳
null_mode [string]
写入 null 值的模式,支持 [ skip , empty], 默认 skip
- skip: 当字段为 null ,连接器不会将此字段写入 hbase
- empty: 当字段为 null 时,连接器会为此字段写入空值
wal_write [boolean]
wal log 写入标志,默认值 false
write_buffer_size [int]
hbase 客户端的写入缓冲区大小,默认 8 1024 1024
encoding [string]
字符串类字段的编码(STRING/DECIMAL/DATE/TIME/TIMESTAMP/ARRAY),支持 [utf8, gbk],默认 utf8
数据类型
Hbase 存储字节,连接器支持:
- TINYINT/SMALLINT/INT/BIGINT/FLOAT/DOUBLE/BOOLEAN/BYTES
- STRING/DECIMAL/DATE/TIME/TIMESTAMP/ARRAY(使用 encoding 序列化为字符串后写入)
hbase_extra_config [config]
hbase 扩展配置
multi_table_sink_replica [int]
多表写入时,每张表对应的 Sink Writer 副本数。通常保持默认值即可;只有单表写入压力较大、需要更多写入并行度时再调大。
schema_save_mode [enum]
写入前如何处理目标 HBase 表结构。常用值包括 RECREATE_SCHEMA、CREATE_SCHEMA_WHEN_NOT_EXIST 和
ERROR_WHEN_SCHEMA_NOT_EXIST。
data_save_mode [enum]
写入前如何处理目标端已有数据。支持 DROP_DATA、APPEND_DATA 和 ERROR_WHEN_DATA_EXISTS。
常见选项
Sink 插件常用参数,详见 Sink 常用选项 Sink Common Options
注意事项
table = "${table_name}"会把每张上游表写入同名 HBase 表,也可以和固定文本组合,例如ods_${table_name}。family_name { all_columns = "info" }会把所有非 rowkey 字段写入同一个列簇。不同字段需要写入不同列簇时,请使用逐字段映射。schema_save_mode控制是否创建或重建表,data_save_mode控制保留、清空已有数据,还是在已有数据存在时报错。
案例
写入单表
env {
parallelism = 1
job.mode = "BATCH"
}
sink {
Hbase {
zookeeper_quorum = "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table = "seatunnel_test"
rowkey_column = ["name"]
family_name {
all_columns = "seatunnel"
}
}
}
目标表不存在时自动创建
sink {
Hbase {
zookeeper_quorum = "hbase_e2e:2181"
table = "seatunnel_test_with_create_when_not_exists"
rowkey_column = ["name"]
family_name {
all_columns = info
}
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
data_save_mode = "APPEND_DATA"
}
}
每次写入前重建目标表
当任务允许在每次运行时删除并重建目标表时,可以使用 schema_save_mode = "RECREATE_SCHEMA"。
配合 data_save_mode = "DROP_DATA" 可以同时清空已有数据;如果只想刷新表结构,保留
APPEND_DATA 即可。
sink {
Hbase {
zookeeper_quorum = "hbase_e2e:2181"
table = "seatunnel_test_with_recreate_schema"
rowkey_column = ["name"]
family_name {
all_columns = info
}
schema_save_mode = "RECREATE_SCHEMA"
}
}
Kerberos 示例
备注:
connector-hbase不会解析krb5_path/kerberos_principal/kerberos_keytab_path。- 需要在运行环境中提前完成 Kerberos 登录并保证
krb5.conf可被 JVM 访问(例如kinit -kt ...或 JVM-Djava.security.krb5.conf=...),同时将 HBase/Hadoop 的安全配置写入hbase_extra_config。
sink {
Hbase {
zookeeper_quorum = "zk1:2181,zk2:2181,zk3:2181"
table = "target_table"
rowkey_column = ["rowkey"]
family_name {
all_columns = "info"
}
# HBase安全配置
hbase_extra_config = {
"hbase.security.authentication" = "kerberos"
"hadoop.security.authentication" = "kerberos"
"hbase.master.kerberos.principal" = "hbase/_HOST@REALM"
"hbase.regionserver.kerberos.principal" = "hbase/_HOST@REALM"
"hbase.rpc.protection" = "authentication"
"hbase.zookeeper.useSasl" = "false"
}
}
}
写入多表
env {
# You can set engine configuration here
execution.parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
tables_configs = [
{
schema = {
table = "hbase_sink_1"
fields {
name = STRING
c_string = STRING
c_double = DOUBLE
c_bigint = BIGINT
c_float = FLOAT
c_int = INT
c_smallint = SMALLINT
c_boolean = BOOLEAN
time = BIGINT
}
}
rows = [
{
kind = INSERT
fields = ["label_1", "sink_1", 4.3, 200, 2.5, 2, 5, true, 1627529632356]
}
]
},
{
schema = {
table = "hbase_sink_2"
fields {
name = STRING
c_string = STRING
c_double = DOUBLE
c_bigint = BIGINT
c_float = FLOAT
c_int = INT
c_smallint = SMALLINT
c_boolean = BOOLEAN
time = BIGINT
}
}
rows = [
{
kind = INSERT
fields = ["label_2", "sink_2", 4.3, 200, 2.5, 2, 5, true, 1627529632357]
}
]
}
]
}
}
sink {
Hbase {
zookeeper_quorum = "hadoop001:2181,hadoop002:2181,hadoop003:2181"
table = "${table_name}"
rowkey_column = ["name"]
family_name {
all_columns = "info"
}
}
}
写入指定列族
Hbase {
zookeeper_quorum = "hbase_e2e:2181"
table = "assign_cf_table"
rowkey_column = ["id"]
family_name {
c_double = "cf1"
c_bigint = "cf2"
}
}
变更日志
Change Log
| Change | Commit | Version |
|---|---|---|
| [Improve][Common] Add HashUtils.bucketIndex for hash-to-bucket routing (#11987) | https://github.com/apache/seatunnel/commit/be53a1d3d1 | 3.0.0 |
| [Improve][Connector-V2][HBase] Migrate timestamp validation to OptionRule (#11803) | https://github.com/apache/seatunnel/commit/0a5708bf64 | 3.0.0 |
| [Fix][connector-hbase] Fix namespace handling for HBase source (#10295) | https://github.com/apache/seatunnel/commit/d722474bc3 | 2.3.13 |
| [Feature][Connector-V2][HBase] Support time-range scan with min/max timestamp in HBaseSource (#10318) | https://github.com/apache/seatunnel/commit/402291d359 | 2.3.13 |
| [Fix][Connector-V2][Hbase] Fix ERROR_WHEN_DATA_EXISTS NPE on empty table (#10336) | https://github.com/apache/seatunnel/commit/9d58bc01ac | 2.3.13 |
| [Improve][Connector-V2][HBase] Support DATE/TIME/TIMESTAMP/DECIMAL in sink and fix DECIMAL deserialization (#10291) | https://github.com/apache/seatunnel/commit/2cc680fe65 | 2.3.13 |
| [Fix[Connector-V2][Hbase] Avoid duplicate split assignment on restore (#10310) | https://github.com/apache/seatunnel/commit/75bc71beb8 | 2.3.13 |
| [Fix][Connector-V2][Hbase] Fix HBase sink binary rowkey handling (#10300) | https://github.com/apache/seatunnel/commit/84b039d4fa | 2.3.13 |
| [Fix][Connector-V2][Hbase] Fix source reader only scanning first split (#10287) | https://github.com/apache/seatunnel/commit/d393d2a82f | 2.3.13 |
| [Fix][Connector-V2][HBase] Ensure fully qualified table name is used in tableExists method and add unit tests (#10126) | https://github.com/apache/seatunnel/commit/53c50f3944 | 2.3.13 |
| [Improve][Connector-V2][HBase] Support configurable range scan boundary inclusion policies (#10011) | https://github.com/apache/seatunnel/commit/40bf6560f5 | 2.3.13 |
| [Feature][Connector-V2] Support row range boundaries for HBaseSource (#9983) | https://github.com/apache/seatunnel/commit/d7b8f37b41 | 2.3.13 |
| [Fix][Core] Add shade module for apache commons lang3 (#9895) | https://github.com/apache/seatunnel/commit/abb9124b05 | 2.3.13 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [Improve] hbase options (#8923) | https://github.com/apache/seatunnel/commit/b6a702b58f | 2.3.10 |
| [Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6eeb | 2.3.10 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef800016 | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
| [Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700) | https://github.com/apache/seatunnel/commit/e2fb679577 | 2.3.8 |
| [Feature][Connector-V2][Hbase] implement hbase catalog (#7516) | https://github.com/apache/seatunnel/commit/b978792cb1 | 2.3.8 |
| [Feature][Connector-V2] Support multi-table sink feature for HBase (#7169) | https://github.com/apache/seatunnel/commit/025fa3bb88 | 2.3.8 |
| [hotfix][connector-v2-hbase]fix and optimize hbase source problem (#7148) | https://github.com/apache/seatunnel/commit/34a6b8e9f6 | 2.3.7 |
| [Improve][hbase] The specified column is written to the specified column family (#5234) | https://github.com/apache/seatunnel/commit/49d397c61d | 2.3.6 |
| [feature][connector-v2-hbase-sink] Support Connector v2 HBase sink TTL data writing (#7116) | https://github.com/apache/seatunnel/commit/adafd80255 | 2.3.6 |
| [E2E][HBase]Refactor hbase e2e (#6859) | https://github.com/apache/seatunnel/commit/1da9bd6ce4 | 2.3.6 |
| [Connector]Add hbase source connector (#6348) | https://github.com/apache/seatunnel/commit/f108a5e658 | 2.3.6 |
| [Feature][HbaseSink]support array data. (#6100) | https://github.com/apache/seatunnel/commit/b592014766 | 2.3.4 |
| [Improve][Common] Introduce new error define rule (#5793) | https://github.com/apache/seatunnel/commit/9d1b2582b2 | 2.3.4 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de7408100 | 2.3.4 |
| [Hotfix][Connector-v2][HbaseSink]Fix default timestamp (#4958) | https://github.com/apache/seatunnel/commit/3d8f3bf902 | 2.3.3 |
| [Improve][build] Give the maven module a human readable name (#4114) | https://github.com/apache/seatunnel/commit/d7cd601051 | 2.3.1 |
| [Improve][Project] Code format with spotless plugin. (#4101) | https://github.com/apache/seatunnel/commit/a2ab166561 | 2.3.1 |
| [Feature][Connector-V2][Hbase] Introduce hbase sink connector (#4049) | https://github.com/apache/seatunnel/commit/68bda94a4c | 2.3.1 |