Sls

Sls source connector

支持的引擎

Spark
Flink
Seatunnel Zeta

主要特性

• batch
• stream
• exactly-once
• column projection
• parallelism
• support user-defined split

描述

从阿里云Sls日志服务中读取数据。

支持的数据源信息

为了使用Sls连接器，需要以下依赖关系。 它们可以通过install-plugin.sh或Maven中央存储库下载。

数据源	支持的版本	Maven
Sls	Universal	Download

Source Options

Name	Type	Required	Default	Description
project	String	Yes	-	阿里云 Sls 项目
logstore	String	Yes	-	阿里云 Sls 日志库
endpoint	String	Yes	-	阿里云访问服务点
access_key_id	String	Yes	-	阿里云访问用户ID
access_key_secret	String	Yes	-	阿里云访问用户密码
start_mode	StartMode[earliest],[group_cursor],[latest]	No	group_cursor	消费者的初始消费模式
consumer_group	String	No	SeaTunnel-Consumer-Group	Sls消费者组id，用于区分不同的消费者组
auto_cursor_reset	CursorMode[begin],[end]	No	end	当消费者组中没有记录读取游标时，初始化读取游标
batch_size	Int	No	1000	每次从SLS中读取的数据量
partition-discovery.interval-millis	Long	No	-1	动态发现主题和分区的间隔

任务示例

简单示例

此示例读取sls的logstore1的数据并将其打印到客户端。如果您尚未安装和部署SeaTunnel，则需要按照安装SeaTunnel中的说明安装和部署SeaTunnel。然后按照[快速启动SeaTunnel引擎](../../Start-v2/locale/Quick-Start SeaTunnel Engine.md)中的说明运行此作业。

创建RAM用户及授权, 请确认RAM用户有足够的权限来读取及管理数据，参考：RAM自定义授权示例

# Defining the runtime environment
env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 30000
}

source {
  Sls {
    endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
    project = "project1"
    logstore = "logstore1"
    access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
    access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    schema = {
      fields = {
            id = "int"
            name = "string"
            description = "string"
            weight = "string"
      }
    }
  }
}

sink {
  Console {
  }
}

变更日志

Change Log

Change	Commit	Version
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6eeb	dev
[Feature]Check Chinese comments in the code (#8319)	https://github.com/apache/seatunnel/commit/d58fce1caf	2.3.9
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef800016	2.3.9
[Improve][Sls] Add sls sink connector、e2e、doc (#7830)	https://github.com/apache/seatunnel/commit/048c47d966	2.3.9
[Fix][Connector-V2] Fix some throwable error not be caught (#7657)	https://github.com/apache/seatunnel/commit/e19d73282e	2.3.8
[Feature][Connector-V2] add Aliyun SLS connector #3733 (#7348)	https://github.com/apache/seatunnel/commit/527c7c7b5f	2.3.7