跳到主要内容
版本:3.1.0

Sls

Sls 源连接器

支持的引擎​

Spark
Flink
SeaTunnel Zeta

主要特性​

描述​

Sls source 连接器用于从阿里云日志服务 SLS 读取日志。它支持批处理和流处理任务,并且可以按 SLS shard 并行读取。流处理模式下,SeaTunnel 会在 checkpoint 完成时提交 SLS 游标,任务重启后 可以从已经提交的位置继续读取。

可以用两种方式读取日志:

  • 配置 schema,把指定的 SLS 日志字段解析成 SeaTunnel 字段。
  • 不配置 schema,把每条 SLS 日志作为 JSON 字符串写入单个 content 字段。

支持的数据源信息​

使用 Sls 连接器前,需要通过 install-plugin.sh 或 Maven 中央仓库获取以下依赖。

数据源支持版本Maven
SlsUniversalDownload

Source 选项​

名称类型是否必填默认值描述
endpointString是-阿里云 SLS 访问地址,例如 cn-hangzhou.log.aliyuncs.com 或内网访问地址。值不能为空或仅包含空白字符。
projectString是-阿里云 SLS Project。值不能为空或仅包含空白字符。
logstoreString是-阿里云 SLS Logstore。值不能为空或仅包含空白字符。
access_key_idString是-阿里云 AccessKey ID。值不能为空或仅包含空白字符。
access_key_secretString是-阿里云 AccessKey Secret。值不能为空或仅包含空白字符。
start_modeearliest, group_cursor, latest否group_cursor初始读取位置。earliest 从最早位置读取,latest 从最新位置读取,group_cursor 使用消费者组已提交的游标。
consumer_groupString否SeaTunnel-Consumer-GroupSLS 消费者组名称。不同任务如果需要独立保存读取位置,应该使用不同的消费者组。
auto_cursor_resetbegin, end否end当 start_mode = group_cursor 但消费者组还没有已提交游标时,使用的初始化位置。
batch_sizeInt否1000每次从每个 shard 拉取的最大日志条数。
partition-discovery.interval-millisLong否-1动态发现 SLS shard 变化的间隔。小于或等于 0 表示不定时发现。
schemaConfig否-用于解析 SLS 日志字段的 SeaTunnel 表结构。不配置时输出单个 content 字符串字段,字段值是整条日志的 JSON 内容。

注意事项​

  • 配置的 RAM 用户需要有读取目标 project、logstore、shard、消费者组、checkpoint 和日志数据的权限。
  • 批处理模式会读取当前分配的 shard 后结束;持续消费日志请使用流处理模式。
  • 不要在日志或任务说明里打印 access_key_secret。

任务示例​

配置 Schema 读取日志​

env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 30000
}

source {
Sls {
endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
project = "project1"
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
consumer_group = "seatunnel-sls-demo"
start_mode = "group_cursor"
auto_cursor_reset = "begin"
batch_size = 1000
schema = {
fields = {
id = "int"
name = "string"
description = "string"
weight = "string"
}
}
}
}

sink {
Console {}
}

不配置 Schema 读取日志​

不配置 schema 时,输出结果只有一个 content 字段,字段值是由 SLS 日志内容组成的 JSON 字符串。

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Sls {
endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
project = "project1"
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}

sink {
Console {}
}

周期性发现新 Shard​

默认情况下,连接器只在任务启动时枚举一次 shard。把 partition-discovery.interval-millis 设置为正值后,任务运行期间新创建的 shard 也会被持续发现。下面的示例每 5 分钟刷新一次 shard 列表:

env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 30000
}

source {
Sls {
endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
project = "project1"
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
consumer_group = "seatunnel-sls-demo"
partition-discovery.interval-millis = 300000
schema = {
fields = {
id = "int"
name = "string"
}
}
}
}

sink {
Console {}
}

变更日志​

Change Log
ChangeCommitVersion
[Improve][Connector-V2] Validate SLS required options (#12352)https://github.com/apache/seatunnel/commit/6eea1da1a83.1.0
[Fix][Connector-V2] Return catalog tables from Aerospike and SLS sinks (#12234)https://github.com/apache/seatunnel/commit/ea1fb0d8503.0.0
[Improve][Connector] Replace printStackTrace with proper logging (#10176)https://github.com/apache/seatunnel/commit/86c8450a7f2.3.13
[Chore] fix typos filed -> field (#9757)https://github.com/apache/seatunnel/commit/e3e1c67d292.3.12
[Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671)https://github.com/apache/seatunnel/commit/9212a771402.3.12
[improve] sls options (#9260)https://github.com/apache/seatunnel/commit/126164508b2.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Feature]Check Chinese comments in the code (#8319)https://github.com/apache/seatunnel/commit/d58fce1caf2.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Improve][Sls] Add sls sink connector、e2e、doc (#7830)https://github.com/apache/seatunnel/commit/048c47d9662.3.9
[Fix][Connector-V2] Fix some throwable error not be caught (#7657)https://github.com/apache/seatunnel/commit/e19d73282e2.3.8
[Feature][Connector-V2] add Aliyun SLS connector #3733 (#7348)https://github.com/apache/seatunnel/commit/527c7c7b5f2.3.7