Sls
Sls Sink 连接器
支持的引擎
Spark
Flink
SeaTunnel Zeta
主要特性
描述
Sls sink 连接器用于把 SeaTunnel 数据写入阿里云日志服务 SLS。每条 SeaTunnel 数据会先序列化为
JSON,然后作为 SLS 日志项写入,日志内容的 key 为 content。
支持的数据源信息
使用 Sls 连接器前,需要通过 install-plugin.sh 或 Maven 中央仓库获取以下依赖。
| 数据源 | 支持版本 | Maven |
|---|---|---|
| Sls | Universal | Download |
Sink 选项
| 名称 | 类型 | 是否必填 | 默认值 | 描述 |
|---|---|---|---|---|
| endpoint | String | 是 | - | 阿里云 SLS 访问地址,例如 cn-hangzhou.log.aliyuncs.com 或内网访问地址。值不能为空或仅包含空白字符。 |
| project | String | 是 | - | 阿里云 SLS Project。值不能为空或仅包含空白字符。 |
| logstore | String | 是 | - | 阿里云 SLS Logstore。值不能为空或仅包含空白字符。 |
| access_key_id | String | 是 | - | 阿里云 AccessKey ID。值不能为空或仅包含空白字符。 |
| access_key_secret | String | 是 | - | 阿里云 AccessKey Secret。值不能为空或仅包含空白字符。 |
| source | String | 否 | SeaTunnel-Source | 写入 SLS log group 的 source 标记。 |
| topic | String | 否 | SeaTunnel-Topic | 写入 SLS log group 的 topic 标记。 |
注意事项
- 配置的 RAM 用户需要有向目标 project 和 logstore 写入日志的权限。
- sink 在收到数据时立即写入,不提供精确一次提交语义。流处理模式下连接器按行写入;checkpoint 只对下游状态有用,并不能保证 SLS 端的写入语义。
- 每条数据都会被序列化为 JSON,并写入 SLS 日志项
content字段,不会映射到其它日志 key。 - 不要在日志或任务说明里打印
access_key_secret。
任务示例
写入数据到 SLS(批处理)
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
row.num = 10
map.size = 10
array.size = 10
bytes.length = 10
string.length = 10
schema = {
fields = {
id = "int"
name = "string"
description = "string"
weight = "string"
}
}
}
}
sink {
Sls {
endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
project = "project1"
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
source = "seatunnel-demo"
topic = "fake-source"
}
}
写入数据到 SLS(流处理)
流处理模式下,连接器会保持 SLS Producer 的连接持续打开,每来一行数据就写入一条。
可以配置 checkpoint.interval 保护下游状态,但需要清楚每条 PutLogs 调用互相独立,
重试只在 Producer 会话内进行,不会跨重启。
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 30000
}
source {
FakeSource {
row.num = 10
map.size = 10
array.size = 10
bytes.length = 10
string.length = 10
schema = {
fields = {
id = "int"
name = "string"
description = "string"
weight = "string"
}
}
}
}
sink {
Sls {
endpoint = "cn-hangzhou.log.aliyuncs.com"
project = "project1"
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
source = "seatunnel-streaming"
topic = "fake-source"
}
}
变更日志
Change Log
| Change | Commit | Version |
|---|---|---|
| [Improve][Connector-V2] Validate SLS required options (#12352) | https://github.com/apache/seatunnel/commit/6eea1da1a8 | 3.1.0 |
| [Fix][Connector-V2] Return catalog tables from Aerospike and SLS sinks (#12234) | https://github.com/apache/seatunnel/commit/ea1fb0d850 | 3.0.0 |
| [Improve][Connector] Replace printStackTrace with proper logging (#10176) | https://github.com/apache/seatunnel/commit/86c8450a7f | 2.3.13 |
| [Chore] fix typos filed -> field (#9757) | https://github.com/apache/seatunnel/commit/e3e1c67d29 | 2.3.12 |
| [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671) | https://github.com/apache/seatunnel/commit/9212a77140 | 2.3.12 |
| [improve] sls options (#9260) | https://github.com/apache/seatunnel/commit/126164508b | 2.3.11 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [Improve] restruct connector common options (#8634) | https://github.com/apache/seatunnel/commit/f3499a6eeb | 2.3.10 |
| [Feature]Check Chinese comments in the code (#8319) | https://github.com/apache/seatunnel/commit/d58fce1caf | 2.3.9 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef800016 | 2.3.9 |
| [Improve][Sls] Add sls sink connector、e2e、doc (#7830) | https://github.com/apache/seatunnel/commit/048c47d966 | 2.3.9 |
| [Fix][Connector-V2] Fix some throwable error not be caught (#7657) | https://github.com/apache/seatunnel/commit/e19d73282e | 2.3.8 |
| [Feature][Connector-V2] add Aliyun SLS connector #3733 (#7348) | https://github.com/apache/seatunnel/commit/527c7c7b5f | 2.3.7 |