Skip to main content
Version: 2.1.2

Kafka

Description

To consume data from Kafka , supported Kafka version >= 0.10.0 .

tip

Engine Supported and plugin name

  • Spark: KafkaStream
  • Flink: Kafka

Options

nametyperequireddefault value
topicsstringyes-
consumer.group.idstringyes-
consumer.bootstrap.serversstringyes-
consumer.*stringno-
common-optionsstringyes-

topics [string]

Kafka topic name. If there are multiple topics, use , to split, for example: "tpc1,tpc2"

consumer.group.id [string]

Kafka consumer group id, used to distinguish different consumer groups

consumer.bootstrap.servers [string]

Kafka cluster address, separated by ,

consumer.* [string]

In addition to the above necessary parameters that must be specified by the Kafka consumer client, users can also specify multiple consumer client non-mandatory parameters, covering all consumer parameters specified in the official Kafka document.

The way to specify parameters is to add the prefix consumer. to the original parameter name. For example, the way to specify auto.offset.reset is: consumer.auto.offset.reset = latest . If these non-essential parameters are not specified, they will use the default values given in the official Kafka documentation.

common options [string]

Source plugin common parameters, please refer to Source Plugin for details

Examples

kafkaStream {
topics = "seatunnel"
consumer.bootstrap.servers = "localhost:9092"
consumer.group.id = "seatunnel_group"
}