INFINI Easysearch

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Description

A sink plugin which use send data to INFINI Easysearch.

Using Dependency

Depenndency easysearch-client

Key features

• exactly-once
• cdc

tip

Engine Supported

• Supported all versions released by INFINI Easysearch.

Data Type Mapping

Easysearch Data Type	SeaTunnel Data Type
STRING KEYWORD TEXT	STRING
BOOLEAN	BOOLEAN
BYTE	BYTE
SHORT	SHORT
INTEGER	INT
LONG	LONG
FLOAT HALF_FLOAT	FLOAT
DOUBLE	DOUBLE
Date	LOCAL_DATE_TIME_TYPE

Sink Options

name	type	required	default value
hosts	array	yes	-
index	string	yes	-
primary_keys	list	no
key_delimiter	string	no	_
username	string	no
password	string	no
max_retry_count	int	no	3
max_batch_size	int	no	10
tls_verify_certificate	boolean	no	true
tls_verify_hostnames	boolean	no	true
tls_keystore_path	string	no	-
tls_keystore_password	string	no	-
tls_truststore_path	string	no	-
tls_truststore_password	string	no	-
common-options		no	-

hosts [array]

INFINI Easysearch cluster http address, the format is host:port , allowing multiple hosts to be specified. Such as ["host1:9200", "host2:9200"].

index [string]

INFINI Easysearch index name.Index support contains variables of field name,such as seatunnel_${age},and the field must appear at seatunnel row. If not, we will treat it as a normal index.

primary_keys [list]

Primary key fields used to generate the document _id, this is cdc required options.

key_delimiter [string]

Delimiter for composite keys ("_" by default), e.g., "$" would result in document _id "KEY1$KEY2$KEY3".

username [string]

security username

password [string]

security password

max_retry_count [int]

one bulk request max try size

max_batch_size [int]

batch bulk doc max size

tls_verify_certificate [boolean]

Enable certificates validation for HTTPS endpoints

tls_verify_hostname [boolean]

Enable hostname validation for HTTPS endpoints

tls_keystore_path [string]

The path to the PEM or JKS key store. This file must be readable by the operating system user running SeaTunnel.

tls_keystore_password [string]

The key password for the key store specified

tls_truststore_path [string]

The path to PEM or JKS trust store. This file must be readable by the operating system user running SeaTunnel.

tls_truststore_password [string]

The key password for the trust store specified

common options

Sink plugin common parameters, please refer to Sink Common Options for details

Examples

Simple

sink {
    Easysearch {
        hosts = ["localhost:9200"]
        index = "seatunnel-${age}"
    }
}

CDC(Change data capture) event

sink {
    Easysearch {
        hosts = ["localhost:9200"]
        index = "seatunnel-${age}"
        
        # cdc required options
        primary_keys = ["key1", "key2", ...]
    }
}

SSL (Disable certificates validation)

sink {
    Easysearch {
        hosts = ["https://localhost:9200"]
        username = "admin"
        password = "admin"
        
        tls_verify_certificate = false
    }
}

SSL (Disable hostname validation)

sink {
    Easysearch {
        hosts = ["https://localhost:9200"]
        username = "admin"
        password = "admin"
        
        tls_verify_hostname = false
    }
}

SSL (Enable certificates validation)

sink {
    Easysearch {
        hosts = ["https://localhost:9200"]
        username = "admin"
        password = "admin"
        
        tls_keystore_path = "${your Easysearch home}/config/certs/http.p12"
        tls_keystore_password = "${your password}"
    }
}

Changelog

Change Log

Change	Commit	Version
[Improve] easysearch options (#8951)	https://github.com/apache/seatunnel/commit/349f142962	2.3.10
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)	https://github.com/apache/seatunnel/commit/55ed90ecaf	2.3.10
[Improve] restruct connector common options (#8634)	https://github.com/apache/seatunnel/commit/f3499a6eeb	2.3.10
[Improve][dist]add shade check rule (#8136)	https://github.com/apache/seatunnel/commit/51ef800016	2.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)	https://github.com/apache/seatunnel/commit/6b7c53d03c	2.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)	https://github.com/apache/seatunnel/commit/e2fb679577	2.3.8
[Feature][Doris] Add Doris type converter (#6354)	https://github.com/apache/seatunnel/commit/5189991843	2.3.6
[Hotfix] Fix compile error (#6463)	https://github.com/apache/seatunnel/commit/943bd48449	2.3.5
[Improve][Connector-V2] Support INFINI Easysearch (#5933)	https://github.com/apache/seatunnel/commit/41e628840a	2.3.5