KafkaStream

Input plugin : Kafka

• Author: InterestingLab
• Homepage: https://interestinglab.github.io/seatunnel-docs
• Version: 1.1.0

Description

Read events from one or more kafka topics. Supporting Kafka >= 0.10.0

Options

name	type	required	default value	engine
topics	string	yes	-	all streaming
consumer.group.id	string	yes	-	spark streaming
consumer.bootstrap.servers	string	yes	-	all streaming
consumer.*	string	no	-	all streaming

topics [string]

Kafka topic. Multiple topics separated by commas. For example, "tpc1,tpc2".

consumer.group.id [string]

Kafka consumer group id, a unique string that identifies the consumer group this consumer belongs to. Only works on Spark Streaming application.

consumer.bootstrap.servers [string]

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.This string should be in the form host1:port1,host2:port2,....

consumer [string]

In addition to the above parameters that must be specified for the consumer client, you can also specify multiple kafka's consumer parameters described in consumerconfigs.

The Spark Structured Streaming optional configurations refer to Structured Streaming + Kafka Integration Guide

The way to specify parameters is to use the prefix "consumer" before the parameter. For example, rebalance.max.retries is specified as: consumer.rebalance.max.retries = 100.If you do not specify these parameters, it will be set the default values according to Kafka documentation

Examples

• Spark Streaming

kafkaStream {
    topics = "seatunnel"
    consumer.bootstrap.servers = "localhost:9092"
    consumer.group.id = "seatunnel_group"
    consumer.rebalance.max.retries = 100
}

• Spark Structured Streaming

kafkaStream {
    topics = "seatunnel"
    consumer.bootstrap.servers = "localhost:9092"
    consumer.group.id = "seatunnel_group"
    consumer.rebalance.max.retries = 100
    consumer.failOnDataLoss = false
}