跳到主要内容
版本:3.1.0

Typesense

Typesense 源连接器

支持的引擎​

SeaTunnel Zeta

描述​

从 Typesense collection 读取文档。该 source 支持有界批读取,也可以通过 query 传入 Typesense 查询参数。

Source 是有界读取:每次作业只会读取一次匹配 query 的全部文档,然后结束。如果需要变更捕获,请在接收端借助 Typesense 自身的变化追踪机制实现。

主要特性​

选项​

名称类型必填默认值
hostsarray是-
collectionstring是-
schemaconfig是-
api_keystring是-
protocolstring否http
querystring否-
batch_sizeint否100
common-options否-

hosts [array]​

Typesense 的访问地址,格式为 host:port,例如:["typesense-01:8108"]。支持配置多个地址。配置多个节点时,Source 会把搜索请求发往第一个可达节点,列表本身不用于并行扫描分片。

collection [string]​

要读取的 Typesense collection 名,例如:companies。

schema [config]​

typesense 需要读取的列。有关更多信息,请参阅:guide。

api_key [string]​

Typesense 安全认证的 api_key。请把它当作敏感凭据处理;在共享环境运行时,建议通过作业密钥或环境变量注入。

protocol [string]​

连接 Typesense 使用的协议,默认是 http。如果使用 Typesense Cloud 或启用了 TLS 的服务, 请设置为 https。

query [string]​

Typesense 查询参数,例如 q=*&filter_by=num_employees:>9000。不配置时读取默认查询返回的文档。

所有合法的 Typesense 搜索参数都可以追加进来,包括 q、query_by、filter_by、sort_by、page 和 per_page,连接器会原样转发给 Typesense 搜索接口。

batch_size [int]​

读取数据时每批查询的文档数量。每次请求使用 Typesense 的 per_page 参数,因此该值必须在 1 与 Typesense 服务端 per_page 上限(通常是 250)之间。如果日志中看到分页被截断,请调小该值。

常用选项​

Source 插件常用参数,具体请参考 Source 常用选项

任务示例​

带过滤条件读取文档​

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Typesense {
hosts = ["localhost:8108"]
collection = "companies"
api_key = "xyz"
query = "q=*&filter_by=num_employees:>9000"
batch_size = 100
schema = {
fields {
company_name_list = array<string>
company_name = string
num_employees = long
country = string
id = string
c_row = {
c_int = int
c_string = string
c_array_int = array<int>
}
}
}
}
}

sink {
Console {}
}

使用自定义查询条件读取子集​

组合 query_by 与 sort_by 可以控制 Typesense 搜索的字段以及结果集的排序方式。

env {
parallelism = 1
job.mode = "BATCH"
}

source {
Typesense {
hosts = ["localhost:8108"]
collection = "companies"
api_key = "xyz"
query = "q=acme&query_by=company_name&filter_by=country:=US&sort_by=num_employees:desc"
batch_size = 50
schema = {
fields {
company_name = string
num_employees = long
country = string
id = string
}
}
}
}

sink {
Console {}
}

变更日志​

Change Log
ChangeCommitVersion
[Fix][Connector-V2] Add opt-in collision-safe Easysearch and Typesense IDs (#12839)https://github.com/apache/seatunnel/commit/68c686438f3.1.0
[Improve][Connector-V2] Validate Typesense connection options (#12175)https://github.com/apache/seatunnel/commit/fdb6beb3ff3.0.0
[Improve][Connector-V2] Improve source split round-robin assignment for Easysearch, AmazonDynamoDB, TiDB CDC and Typesense (#11607)https://github.com/apache/seatunnel/commit/9487cc8e163.0.0
fix typesense add missing import for MULTI_TABLE_SINK_REPLICA to pass… (#10898)https://github.com/apache/seatunnel/commit/d8e9244a2b3.0.0
[Fix][Core] Add shade module for apache commons lang3 (#9895)https://github.com/apache/seatunnel/commit/abb9124b052.3.13
[Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671)https://github.com/apache/seatunnel/commit/9212a771402.3.12
[improve] typesense options (#9398)https://github.com/apache/seatunnel/commit/bf20a3e6a82.3.12
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)https://github.com/apache/seatunnel/commit/55ed90ecaf2.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Feature]Check Chinese comments in the code (#8319)https://github.com/apache/seatunnel/commit/d58fce1caf2.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)https://github.com/apache/seatunnel/commit/e2fb6795772.3.8
[Feature][Connector-V2] Support typesense connector (#7450)https://github.com/apache/seatunnel/commit/138d2a4eb22.3.8