DataHub

DataHub sink connector

Description

A sink plugin which use send message to DataHub

Key features

• exactly-once

Options

name	type	required	default value
endpoint	string	yes	-
accessId	string	yes	-
accessKey	string	yes	-
project	string	yes	-
topic	string	yes	-
timeout	int	yes	-
retryTimes	int	yes	-
common-options		no	-

endpoint [string]

your DataHub endpoint start with http （string）

accessId [string]

your DataHub accessId which cloud be access from Alibaba Cloud (string)

accessKey[string]

your DataHub accessKey which cloud be access from Alibaba Cloud (string)

project [string]

your DataHub project which is created in Alibaba Cloud (string)

topic [string]

your DataHub topic (string)

timeout [int]

the max connection timeout (int)

retryTimes [int]

the max retry times when your client put record failed (int)

common options

Sink plugin common parameters, please refer to Sink Common Options for details

Example

sink {
 DataHub {
  endpoint="yourendpoint"
  accessId="xxx"
  accessKey="xxx"
  project="projectname"
  topic="topicname"
  timeout=3000
  retryTimes=3
 }
}

Changelog

Change Log

Change	Commit	Version
[improve] datahub sink options (#8744)	https://github.com/apache/seatunnel/commit/88f35bd70	2.3.10
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)	https://github.com/apache/seatunnel/commit/6b7c53d03	2.3.9
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)	https://github.com/apache/seatunnel/commit/8de740810	2.3.4
[Improve][build] Give the maven module a human readable name (#4114)	https://github.com/apache/seatunnel/commit/d7cd60105	2.3.1
[Improve][Project] Code format with spotless plugin. (#4101)	https://github.com/apache/seatunnel/commit/a2ab16656	2.3.1
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)	https://github.com/apache/seatunnel/commit/226dc6a11	2.3.0
[Improve][Connector-V2][DataHub] Unified exception for DataHub sink connector & change package name of DataHub (#3446)	https://github.com/apache/seatunnel/commit/395635fa1	2.3.0
[improve][connector] The Factory#factoryIdentifier must be consistent with PluginIdentifierInterface#getPluginName (#3328)	https://github.com/apache/seatunnel/commit/d9519d696	2.3.0
[Improve][Connector-V2][DataHub] Add DataHub Sink Factory (#3323)	https://github.com/apache/seatunnel/commit/685978d06	2.3.0
[#2606]Dependency management split (#2630)	https://github.com/apache/seatunnel/commit/fc047be69	2.2.0-beta
[Feature][Connector-V2]Support datahub sink (#2558)	https://github.com/apache/seatunnel/commit/43600a704	2.2.0-beta