RocketMQ
RocketMQ Sink 连接器
支持的 Apache RocketMQ 版本
- 4.9.0 或更新版本
引擎支持
Spark
Flink
SeaTunnel Zeta
主要特性
描述
将 SeaTunnel 数据行写入 Apache RocketMQ topic。该 Sink 支持 JSON 和文本消息体、消息 tag、同步发送、按字段选择分区,以及在 exactly.once = true 时使用事务消息保证精确一次写入。
Sink 参数
| 参数名 | 类型 | 是否必填 | 默认值 | 描述 |
|---|---|---|---|---|
| topic | String | 是 | - | RocketMQ topic 名称。 |
| name.srv.addr | String | 是 | - | RocketMQ NameServer 地址,例如 localhost:9876。 |
| acl.enabled | Boolean | 否 | false | 是否启用 RocketMQ ACL 鉴权。 |
| access.key | String | 否 | - | 访问密钥。acl.enabled = true 时必填。 |
| secret.key | String | 否 | - | 秘密密钥。acl.enabled = true 时必填。 |
| producer.group | String | 否 | SeaTunnel-Producer-Group | RocketMQ 生产者组 ID。 |
| tag | String | 否 | - | 写入每条消息时使用的 RocketMQ tag。 |
| partition.key.fields | List | 否 | - | 会被序列化为 RocketMQ 消息 key 的字段名。配置的字段必须存在于上游 schema 中。 |
| format | String | 否 | json | 消息格式。支持 json 和 text。 |
| field.delimiter | String | 否 | , | format = text 时使用的字段分隔符。 |
| producer.send.sync | Boolean | 否 | false | 是否同步发送消息。为 false 时异步发送。 |
| exactly.once | Boolean | 否 | false | 是否使用事务消息实现精确一次写入。 |
| max.message.size | int | 否 | 4194304 | 最大消息体大小,单位字节。 |
| send.message.timeout | int | 否 | 3000 | 发送消息超时时间,单位毫秒。 |
| common-options | config | 否 | - | Sink 连接器通用参数,详情请参考 Sink 通用参数。 |
参数说明
partition.key.fields
partition.key.fields 控制 RocketMQ 消息 key。SeaTunnel 会把这些字段的值序列化成
JSON,并写入 Message.keys。在非事务发送时,同一个 key 还会交给 RocketMQ
的哈希队列选择器,因此相同 key 的数据会进入同一个队列。如果不配置该参数,则由
RocketMQ 自行选择队列。
例如,上游字段中有 c_int 时,可以这样把 c_int 用作消息 key:
partition.key.fields = ["c_int"]
exactly.once
Sink 支持通过 RocketMQ 事务消息实现精确一次写入。该能力默认关闭。确认 RocketMQ 集群和作业 checkpoint 配置满足事务写入要求后,可设置 exactly.once = true。
当 format = text 时,SeaTunnel 会按上游 schema 的字段顺序序列化,并使用 field.delimiter 拼接字段。当 format = json 时,每行数据会写成一个 JSON 对象。
producer.send.sync
producer.send.sync = true 表示生产者会等待 RocketMQ 确认每次发送请求。默认值为 false 时,消息会异步发送。
任务示例
写入 JSON 消息
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
row.num = 10
schema = {
fields {
c_string = string
c_int = int
c_timestamp = timestamp
}
}
}
}
sink {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topic = "test_topic"
partition.key.fields = ["c_int"]
producer.send.sync = true
}
}
写入文本消息
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
row.num = 10
schema = {
fields {
id = bigint
content = string
}
}
}
}
sink {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topic = "test_text_topic"
format = text
field.delimiter = ","
producer.send.sync = true
}
}
写入带 tag 的消息
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
row.num = 10
schema = {
fields {
c_string = string
c_int = int
}
}
}
}
sink {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topic = "test_topic_message_tag"
tag = "test_tag"
partition.key.fields = ["c_string"]
producer.send.sync = true
}
}
RocketMQ 读写 RocketMQ
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
}
source {
Rocketmq {
name.srv.addr = "rocketmq-e2e:9876"
topics = "test_topic_source"
plugin_output = "rocketmq_table"
format = json
start.mode = "CONSUME_FROM_FIRST_OFFSET"
consumer.group = "rocketmq_to_rocketmq_group"
schema = {
fields {
id = bigint
c_string = string
}
}
}
}
sink {
Rocketmq {
plugin_input = "rocketmq_table"
name.srv.addr = "rocketmq-e2e:9876"
topic = "test_topic_sink"
partition.key.fields = ["id"]
exactly.once = true
}
}
FAQ
RocketMQ Sink 如何配置消息路由与分区键?
可以通过配置 partition.key.fields 声明一个或多个数据列作为分区键。连接器将对指定字段值进行 Hash 计算,确保具有相同 Key 值的消息始终投递至 RocketMQ 的同一个消息队列。
RocketMQ Sink 如何提供精确一次(Exactly-Once)投递保证?
当设置 exactly.once = true 时,Sink 将启用 RocketMQ 两阶段事务消息机制。在 Checkpoint 执行过程中发送半消息(Half Message),待计算引擎 Checkpoint 确认成功后再正式提交该批消息供下游消费。
支持哪些消息体序列化格式?
支持 json 和 text 格式。系统可将每行结构化数据解析为标准 JSON 对象或按指定分隔符拼接为纯文本字符串写入 RocketMQ。
变更日志
Change Log
| Change | Commit | Version |
|---|---|---|
| [Fix][Connector-V2][RocketMQ] Close consumer client before interrupting its executor (#12099) | https://github.com/apache/seatunnel/commit/934276b2f7 | 3.1.0 |
| [Fix][Connector-V2] Keep RocketMQ group offsets when a later topic has no retry route (#12648) | https://github.com/apache/seatunnel/commit/07f32fffbd | 3.1.0 |
| [Improve][Connector-V2] Correct two RocketMQ comments that no longer hold (#12438) | https://github.com/apache/seatunnel/commit/524312d741 | 3.1.0 |
| [Improve][Connector-V2] Note the multi-topic early return in currentOffsets (#12417) | https://github.com/apache/seatunnel/commit/6934fd0062 | 3.1.0 |
| [Fix][Connector-V2] Surface RocketMQ group-offset lookup failures instead of rewinding (#12349) | https://github.com/apache/seatunnel/commit/b70652be8a | 3.1.0 |
| [Fix][Zeta] Reuse fixed slots after master failover (#11458) | https://github.com/apache/seatunnel/commit/a9cda80f4b | 3.0.0 |
| [Improve][Common] Add HashUtils.bucketIndex for hash-to-bucket routing (#11987) | https://github.com/apache/seatunnel/commit/be53a1d3d1 | 3.0.0 |
| [Improve][Connector-V2] Migrate RocketMQ Source validation to declarative OptionRule (#11158) | https://github.com/apache/seatunnel/commit/c8fb49358a | 3.0.0 |
| Test/rocketmq restore e2e (#10778) | https://github.com/apache/seatunnel/commit/6d2a104659 | 3.0.0 |
| [Improve][Connector-V2] Complete OptionRule declarations for RocketMQ source and sink (#10701) | https://github.com/apache/seatunnel/commit/7becf69b70 | 3.0.0 |
| [Feature][Connector-V2] Support multi-table read for RocketMQ source (#10619) | https://github.com/apache/seatunnel/commit/85615468b7 | 3.0.0 |
| [Fix][Core] Add shade module for apache commons lang3 (#9895) | https://github.com/apache/seatunnel/commit/abb9124b05 | 2.3.13 |
| [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671) | https://github.com/apache/seatunnel/commit/9212a77140 | 2.3.12 |
| [improve] rocketmq options (#9251) | https://github.com/apache/seatunnel/commit/4cbe3b9172 | 2.3.12 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [Improve][Connector-V2] RocketMQ Source add message tag config (#8825) | https://github.com/apache/seatunnel/commit/5913e8c35f | 2.3.10 |
| [Improve][Connector-V2] Add optional flag for rocketmq connector to skip parse errors instead of failing (#8737) | https://github.com/apache/seatunnel/commit/701f17b5d4 | 2.3.10 |
| [Improve][Connector-V2] RocketMQ Sink add message tag config (#7996) | https://github.com/apache/seatunnel/commit/97a1b00e48 | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
| [Fix][Connector-V2] Fix some throwable error not be caught (#7657) | https://github.com/apache/seatunnel/commit/e19d73282e | 2.3.8 |
| [Feature][Kafka] Support multi-table source read (#5992) | https://github.com/apache/seatunnel/commit/60104602d1 | 2.3.6 |
| [Fix][connector-rocketmq] commit a correct offset to broker & reduce ThreadInterruptedException log (#6668) | https://github.com/apache/seatunnel/commit/b7480e1a89 | 2.3.6 |
| [fix][connector-rocketmq]Fix a NPE problem when checkpoint.interval is set too small(#6624) (#6625) | https://github.com/apache/seatunnel/commit/6e0c81d492 | 2.3.5 |
| [Test][E2E] Add thread leak check for connector (#5773) | https://github.com/apache/seatunnel/commit/1f2f3fc5f0 | 2.3.4 |
| [Fix][Connector] Rocketmq source startOffset greater than endOffset error (#6287) | https://github.com/apache/seatunnel/commit/cd44b5894e | 2.3.4 |
| [Improve][Common] Introduce new error define rule (#5793) | https://github.com/apache/seatunnel/commit/9d1b2582b2 | 2.3.4 |
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755) | https://github.com/apache/seatunnel/commit/8de7408100 | 2.3.4 |
| [Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260) | https://github.com/apache/seatunnel/commit/51c0d709ba | 2.3.4 |
| [Improve][pom] Formatting pom (#4761) | https://github.com/apache/seatunnel/commit/1d6d3815ec | 2.3.2 |
| [Hotfix][Connector-V2][RocketMQ] Fix rocketmq spark e2e test cases (#4583) | https://github.com/apache/seatunnel/commit/e711f6ef4c | 2.3.2 |
| [Feature][Connector-V2] Add rocketmq source and sink (#4007) | https://github.com/apache/seatunnel/commit/e333897552 | 2.3.2 |