Skip to main content
Version: Next

Typesense

Typesense Source Connector

Support Those Engines​

SeaTunnel Zeta

Description​

Reads documents from a Typesense collection. The source supports bounded batch reads and can pass Typesense search parameters through query.

The source is bounded. Each job reads every document that matches the configured query once and then completes. Use Typesense's own change tracking on the receiver side if you need change-data capture.

Key Features​

Options​

NameTypeRequiredDefault
hostsarrayyes-
collectionstringyes-
schemaconfigyes-
api_keystringyes-
protocolstringnohttp
querystringno-
batch_sizeintno100
common-optionsno-

hosts [array]​

The access address of Typesense. Use the host:port format, for example: ["typesense-01:8108"]. Multiple hosts are supported. When several nodes are configured, the source issues its search requests to the first reachable node; the list is not used for parallel scan sharding.

collection [string]​

The name of the Typesense collection to read from, for example: "companies".

schema [config]​

The columns to be read from Typesense. For more information, please refer to the guide.

api_key [string]​

The api_key for Typesense security authentication. Treat this value as a secret and prefer passing it via a job secret or environment variable when running on shared infrastructure.

protocol [string]​

The protocol used to connect to Typesense. The default value is http. Use https for Typesense Cloud or other TLS-enabled endpoints.

query [string]​

Typesense search parameters, for example q=*&filter_by=num_employees:>9000. If it is not set, the source reads all documents returned by the default search.

Any valid Typesense search parameter can be appended, including q, query_by, filter_by, sort_by, page, and per_page. The connector forwards them to the Typesense search API unchanged.

batch_size [int]​

The number of records to query per batch when reading data. Each request uses the Typesense per_page parameter, so the value must be between 1 and the Typesense server-side per_page limit (typically 250). Lower the value if you see truncated pages in the logs.

Common Options​

For common parameters of Source plugins, please refer to Source Common Options.

Task Example​

Read Documents With A Filter​

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Typesense {
    hosts = ["localhost:8108"]
    collection = "companies"
    api_key = "xyz"
    query = "q=*&filter_by=num_employees:>9000"
    batch_size = 100
    schema = {
      fields {
        company_name_list = array<string>
        company_name = string
        num_employees = long
        country = string
        id = string
        c_row = {
          c_int = int
          c_string = string
          c_array_int = array<int>
        }
      }
    }
  }
}

sink {
  Console {}
}

Read A Subset With A Custom Query​

Combine query_by and sort_by to control which fields Typesense searches and how the result set is ordered.

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Typesense {
    hosts = ["localhost:8108"]
    collection = "companies"
    api_key = "xyz"
    query = "q=acme&query_by=company_name&filter_by=country:=US&sort_by=num_employees:desc"
    batch_size = 50
    schema = {
      fields {
        company_name = string
        num_employees = long
        country = string
        id = string
      }
    }
  }
}

sink {
  Console {}
}

Changelog​

Change Log
ChangeCommitVersion
[Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671)https://github.com/apache/seatunnel/commit/9212a771402.3.12
[improve] typesense options (#9398)https://github.com/apache/seatunnel/commit/bf20a3e6a82.3.12
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix] Fix error log name for SourceSplitEnumerator implements class (#8817)https://github.com/apache/seatunnel/commit/55ed90ecaf2.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Feature]Check Chinese comments in the code (#8319)https://github.com/apache/seatunnel/commit/d58fce1caf2.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700)https://github.com/apache/seatunnel/commit/e2fb6795772.3.8
[Feature][Connector-V2] Support typesense connector (#7450)https://github.com/apache/seatunnel/commit/138d2a4eb22.3.8