Skip to main content
Version: Next

TDengine

TDengine source connector

Description​

Read data from TDengine super tables.

The source reads data in batch mode by querying a time range from one super table. You can read all sub tables under the super table, limit the read to specific sub tables, and select only part of the columns.

Each source split reads one TDengine sub table. The output schema always adds subtable_name as the first field so that downstream sinks can keep the original TDengine sub table name. This is also the field consumed by the TDengine sink when writing rows back to TDengine.

Key features​

Options​

nametyperequireddefault value
urlstringyes-
usernamestringyes-
passwordstringyes-
databasestringyes-
stablestringyes-
sub_tableslistno-
lower_boundstringyes-
upper_boundstringyes-
read_columnslistno-

url [string]​

The TDengine REST JDBC URL.

e.g.

jdbc:TAOS-RS://localhost:6041/

username [string]​

The username used to connect to TDengine.

password [string]​

The password used to connect to TDengine.

database [string]​

The TDengine database name.

stable [string]​

The TDengine super table name.

sub_tables [list]​

A list of sub table names. If it is not configured, all sub tables under the configured super table are read. If it is configured, only the listed sub tables are read.

lower_bound [string]​

The inclusive lower bound of the query time range. The connector adds timestamp_column >= lower_bound to each sub table query. Use a TDengine-compatible timestamp string, for example 2018-10-03 14:38:05.000.

upper_bound [string]​

The exclusive upper bound of the query time range. The connector adds timestamp_column < upper_bound to each sub table query. Use a TDengine-compatible timestamp string, for example 2018-10-03 14:38:16.801.

read_columns [list]​

A list of column names to read. If it is not configured, all columns are read. When reading from a super table, put TAGS columns at the end of the list. Do not include subtable_name; the connector adds it automatically as the first output field.

The order of read_columns decides the output field order after subtable_name. If the result is written to a TDengine sink, keep normal columns before TAGS columns so the sink can split column values and TAGS values correctly.

Examples​

Read all sub tables in a time range​

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power"
    stable = "meters"
    lower_bound = "2018-10-03 14:38:05.000"
    upper_bound = "2018-10-03 14:38:16.801"
    plugin_output = "tdengine_result"
  }
}

Read selected sub tables and columns​

source {
  TDengine {
    url = "jdbc:TAOS-RS://localhost:6041/"
    username = "root"
    password = "taosdata"
    database = "power"
    stable = "meters"
    lower_bound = "2018-10-03 14:38:05.000"
    upper_bound = "2018-10-03 14:38:16.801"
    sub_tables = ["d1001", "d1002"]
    read_columns = ["ts", "current", "voltage", "phase", "off", "nc", "location", "groupid"]
  }
}

Read from TDengine and write to TDengine​

env {
  parallelism = 2
  job.mode = "BATCH"
}

source {
  TDengine {
    url = "jdbc:TAOS-RS://tdengine-src:6041/"
    username = "root"
    password = "taosdata"
    database = "power"
    stable = "meters"
    lower_bound = "2018-10-03 14:38:05.000"
    upper_bound = "2018-10-03 14:38:16.801"
    plugin_output = "tdengine_result"
  }
}

sink {
  TDengine {
    url = "jdbc:TAOS-RS://tdengine-sink:6041/"
    username = "root"
    password = "taosdata"
    database = "power2"
    stable = "meters2"
    timezone = "UTC"
  }
}

Changelog​

Change Log
ChangeCommitVersion
[Feature][connector-tdengine] Support subtable and fieldNames in tdengine source (#9593)https://github.com/apache/seatunnel/commit/b136a0dc432.3.12
[improve] tdengine options (#9399)https://github.com/apache/seatunnel/commit/ff122fe4052.3.12
[Feature][Connector-V2] Support multi-table sink feature for TDengine (#9215)https://github.com/apache/seatunnel/commit/98b593f0952.3.11
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[Fix][Connector-V2] Fix NullPointerException when column or tag contains null value in TDengine sink (#9158)https://github.com/apache/seatunnel/commit/a047cab5462.3.11
[Fix][Connector][TDEngine] TDEngine support NCHAR type (#8411)https://github.com/apache/seatunnel/commit/88c92ae1b12.3.9
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve][Connector-V2] Close all ResultSet after used (#7389)https://github.com/apache/seatunnel/commit/853e9732122.3.8
[Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read datahttps://github.com/apache/seatunnel/commit/a18fca80062.3.7
[Bugfix][TDengine] Fix the issue of losing the driver due to multiple calls to the submit job REST API #6581 (#6596)https://github.com/apache/seatunnel/commit/470bb974342.3.5
[improve][connector-tdengine] support read bool column from tdengine (#6025)https://github.com/apache/seatunnel/commit/af39235ee32.3.4
[Bugfix][TDengine] Fix the degree of multiple parallelism affects driver loading (#6020)https://github.com/apache/seatunnel/commit/b6ebbd47b22.3.4
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
[Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)https://github.com/apache/seatunnel/commit/51c0d709ba2.3.4
[Hotfix][Connector] Fixed TDengine connector using jdbc driver to cause loading error (#4598)https://github.com/apache/seatunnel/commit/78f7989b812.3.2
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Feature][Connector-V2] add tdengine source (#2832)https://github.com/apache/seatunnel/commit/acf4d5b1b42.3.1