Redis

Redis sink connector

Description

Used to write data to Redis.

Key features

• exactly-once

Options

name	type	required	default value
host	string	yes	-
port	int	yes	-
key	string	yes	-
data_type	string	yes	-
batch_size	int	no	10
user	string	no	-
auth	string	no	-
db_num	int	no	0
mode	string	no	single
nodes	list	yes when mode=cluster	-
format	string	no	json
expire	long	no	-1
common-options		no	-

host [string]

Redis host

port [int]

Redis port

key [string]

The value of key you want to write to redis.

For example, if you want to use value of a field from upstream data as key, you can assign it to the field name.

Upstream data is the following:

code	data	success
200	get success	true
500	internal error	false

If you assign field name to code and data_type to key, two data will be written to redis:

1. 200 -> {code: 200, message: true, data: get success}
2. 500 -> {code: 500, message: false, data: internal error}

If you assign field name to value and data_type to key, only one data will be written to redis because value is not existed in upstream data's fields:

1. value -> {code: 500, message: false, data: internal error}

Please see the data_type section for specific writing rules.

Of course, the format of the data written here I just take json as an example, the specific or user-configured format prevails.

data_type [string]

Redis data types, support key hash list set zset

• key

Each data from upstream will be updated to the configured key, which means the later data will overwrite the earlier data, and only the last data will be stored in the key.

• hash

Each data from upstream will be split according to the field and written to the hash key, also the data after will overwrite the data before.

• list

Each data from upstream will be added to the configured list key.

• set

Each data from upstream will be added to the configured set key.

• zset

Each data from upstream will be added to the configured zset key with a weight of 1. So the order of data in zset is based on the order of data consumption.

batch_size [int]

ensure the batch write size in single-machine mode; no guarantees in cluster mode.

user [string]

redis authentication user, you need it when you connect to an encrypted cluster

auth [string]

Redis authentication password, you need it when you connect to an encrypted cluster

db_num [int]

Redis database index ID. It is connected to db 0 by default

mode [string]

redis mode, single or cluster, default is single

nodes [list]

redis nodes information, used in cluster mode, must like as the following format:

["host1:port1", "host2:port2"]

format [string]

The format of upstream data, now only support json, text will be supported later, default json.

When you assign format is json, for example:

Upstream data is the following:

code	data	success
200	get success	true

Connector will generate data as the following and write it to redis:

{"code":  200, "data":  "get success", "success":  "true"}

expire [long]

Set redis expiration time, the unit is second. The default value is -1, keys do not automatically expire by default.

common options

Sink plugin common parameters, please refer to Sink Common Options for details

Example

simple:

Redis {
  host = localhost
  port = 6379
  key = age
  data_type = list
}

Changelog

2.2.0-beta 2022-09-26

• Add Redis Sink Connector

next version

• [Improve] Support redis cluster mode connection and user authentication 3188