跳到主要内容
版本:Next

IoTDB

IoTDB 数据读取器

支持引擎

Spark
Flink
SeaTunnel Zeta

描述

用于从 IoTDB 中读取数据。

主要特性

支持的数据源信息

数据源支持的版本地址
IoTDB2.0 <= versionlocalhost:6667

数据类型映射

IoTDB 数据类型SeaTunnel 数据类型
BOOLEANBOOLEAN
INT32TINYINT
INT32SMALLINT
INT32INT
INT64BIGINT
FLOATFLOAT
DOUBLEDOUBLE
TEXTSTRING
STRINGSTRING
TIMESTAMPBIGINT
TIMESTAMPTIMESTAMP
BLOBSTRING
DATEDATE

Source 选项

名称类型是否必填默认值描述
node_urlsArray-IoTDB 集群地址,格式为 ["host1:port"]["host1:port","host2:port"]
usernameString-IoTDB 用户名
passwordString-IoTDB 用户密码
sql_dialectStringtreeIoTDB 模型,tree:树模型;table:表模型
databaseString-要查询的数据库名,只在表模型中生效
sqlString-要执行的 SQL 查询语句
schemaConfig-数据模式定义
fetch_sizeInteger-单次获取数据量:查询时每次从 IoTDB 获取的数据量
lower_boundLong-时间范围下界(通过时间列进行数据分片时使用)
upper_boundLong-时间范围上界(通过时间列进行数据分片时使用)
num_partitionsInteger-分区数量(通过时间列进行数据分片时使用):
- 1 个分区:使用完整时间范围
- 若分区数 < (上界 -下界),则使用差值作为实际分区数
default_thrift_buffer_sizeInteger-Thrift 协议缓冲区大小
max_thrift_frame_sizeInteger-Thrift 最大帧尺寸
enable_cache_leaderBoolean-是否启用 Leader 节点缓存
common-options-Source 插件常用参数,详见 [Source common Options](../Source common Options.md)

我们可以使用时间列进行分区查询。

num_partitions [int]

分区数量

upper_bound [long]

时间范围上界

lower_bound [long]

时间范围下界

     将时间范围分割成 numPartitions 个分区

若 numPartitions = 1,使用完整的时间范围
若 numPartitions < (upper_bound - lower_bound),使用 (upper_bound - lower_bound) 个分区

例:lower_bound = 1, upper_bound = 10, numPartitions = 2
sql = "select * from test where age > 0 and age < 10"

分区结果:
split 1: select * from test where (time >= 1 and time < 6) and ( age > 0 and age < 10 )
split 2: select * from test where (time >= 6 and time < 11) and ( age > 0 and age < 10 )

示例

示例 1:读取 IoTDB 树模型数据

env {
parallelism = 2
job.mode = "BATCH"
}

source {
IoTDB {
node_urls = ["localhost:6667"]
username = "root"
password = "root"
sql = "SELECT temperature, moisture, c_int, c_bigint, c_float, c_double, c_string, c_boolean FROM root.test_group.* WHERE time < 4102329600000 align by device"
schema {
fields {
ts = timestamp
device_name = string
temperature = float
moisture = bigint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_string = string
c_boolean = boolean
}
}
}
}

sink {
Console {
}
}

上游 IoTDB 的数据格式如下所示:

IoTDB> SELECT temperature, moisture, c_int, c_bigint, c_float, c_double, c_string, c_boolean FROM root.test_group.* WHERE time < 4102329600000 align by device;
+------------------------+------------------------+--------------+-----------+--------+--------------+----------+---------+---------+----------+
| Time| Device| temperature| moisture| c_int| c_bigint| c_float| c_double| c_string| c_boolean|
+------------------------+------------------------+--------------+-----------+--------+--------------+----------+---------+---------+----------+
|2022-09-25T00:00:00.001Z|root.test_group.device_a| 36.1| 100| 1| 21474836470| 1.0f| 1.0d| abc| true|
|2022-09-25T00:00:00.001Z|root.test_group.device_b| 36.2| 101| 2| 21474836470| 2.0f| 2.0d| abc| true|
|2022-09-25T00:00:00.001Z|root.test_group.device_c| 36.3| 102| 3| 21474836470| 3.0f| 3.0d| abc| true|
+------------------------+------------------------+--------------+-----------+--------+--------------+----------+---------+---------+----------+

读取到 SeaTunnelRow 的数据格式如下所示:

tsdevice_nametemperaturemoisturec_intc_bigintc_floatc_doublec_stringc_boolean
1664035200001root.test_group.device_a36.11001214748364701.0f1.0dabctrue
1664035200001root.test_group.device_b36.21012214748364702.0f2.0dabctrue
1664035200001root.test_group.device_c36.31023214748364703.0f3.0dabctrue

示例 2:读取 IoTDB 表模型数据

env {
parallelism = 2
job.mode = "BATCH"
}

source {
IoTDB {
node_urls = ["localhost:6667"]
username = "root"
password = "root"
sql_dialect = "table"
database = "test_database"
sql = "SELECT time, sn, type, bidprice, bidsize, domain, buyno, askprice FROM test_table"
schema {
fields {
ts = timestamp
sn = string
type = string
bidprice = int
bidsize = double
domain = boolean
buyno = bigint
askprice = string
}
}
}
}

sink {
Console {
}
}

若查询语句中明确了数据库,则无需使用 database 参数

上游 IoTDB 的数据格式如下所示:

IoTDB> SELECT time, sn, type, bidprice, bidsize, domain, buyno, askprice FROM test_table
+-----------------------------+------+----+--------+------------------+------+-----+-----------+
| time| sn|type|bidprice| bidsize|domain|buyno| askprice|
+-----------------------------+------+----+--------+------------------+------+-----+-----------+
|2025-07-30T17:52:34.851+08:00|0700HK| L1| 9|10.323907796459721| true| 10|-1064754527|
|2025-07-30T17:52:34.951+08:00|0700HK| L1| 10| 9.844574317657585| false| 9|-1088662576|
|2025-07-30T17:52:35.051+08:00|0700HK| L1| 9| 9.272974132434069| true| 9| 402003616|
+-----------------------------+------+----+--------+------------------+------+-----+-----------+

读取到 SeaTunnelRow 的数据格式如下所示:

tssntypebidpricebidsizedomainbuynoaskprice
2025-07-30T17:52:34.8510700HKL1910.323907796459721true10-1064754527
2025-07-30T17:52:34.9510700HKL1109.844574317657585false9-1088662576
2025-07-30T17:52:35.0510700HKL199.272974132434069true9402003616

变更日志

Change Log
ChangeCommitVersion
[Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118)https://github.com/apache/seatunnel/commit/4f5adeb1c72.3.11
[improve] iotdb options (#8965)https://github.com/apache/seatunnel/commit/6e073935f42.3.10
[Improve] restruct connector common options (#8634)https://github.com/apache/seatunnel/commit/f3499a6eeb2.3.10
[Improve][dist]add shade check rule (#8136)https://github.com/apache/seatunnel/commit/51ef8000162.3.9
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)https://github.com/apache/seatunnel/commit/6b7c53d03c2.3.9
[Improve][Common] Introduce new error define rule (#5793)https://github.com/apache/seatunnel/commit/9d1b2582b22.3.4
[Improve] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)https://github.com/apache/seatunnel/commit/8de74081002.3.4
Support config column/primaryKey/constraintKey in schema (#5564)https://github.com/apache/seatunnel/commit/eac76b4e502.3.4
[Doc] update iotdb document (#5404)https://github.com/apache/seatunnel/commit/856aedb3c92.3.4
[Improve][Connector-V2] Remove scheduler in IoTDB sink (#5270)https://github.com/apache/seatunnel/commit/299637868c2.3.4
[Hotfix] Fix com.google.common.base.Preconditions to seatunnel shade one (#5284)https://github.com/apache/seatunnel/commit/ed5eadcf732.3.3
Merge branch 'dev' into merge/cdchttps://github.com/apache/seatunnel/commit/4324ee19122.3.1
[Improve][Project] Code format with spotless plugin.https://github.com/apache/seatunnel/commit/423b5830382.3.1
[improve][api] Refactoring schema parse (#4157)https://github.com/apache/seatunnel/commit/b2f573a13e2.3.1
[Improve][build] Give the maven module a human readable name (#4114)https://github.com/apache/seatunnel/commit/d7cd6010512.3.1
[Improve][Project] Code format with spotless plugin. (#4101)https://github.com/apache/seatunnel/commit/a2ab1665612.3.1
[Improve][SourceConnector] Unified schema parameter, update IoTDB sou… (#3896)https://github.com/apache/seatunnel/commit/a0959c5fd12.3.1
[Feature][Connector] add get source method to all source connector (#3846)https://github.com/apache/seatunnel/commit/417178fb842.3.1
[Feature][API &amp; Connector &amp; Doc] add parallelism and column projection interface (#3829)https://github.com/apache/seatunnel/commit/b9164b8ba12.3.1
[Hotfix][OptionRule] Fix option rule about all connectors (#3592)https://github.com/apache/seatunnel/commit/226dc6a1192.3.0
[Improve][Connector-V2][Iotdb] Unified exception for iotdb source & sink connector (#3557)https://github.com/apache/seatunnel/commit/7353fed6d62.3.0
[Feature][Connector V2] expose configurable options in IoTDB (#3387)https://github.com/apache/seatunnel/commit/06359ea76a2.3.0
[Improve][Connector-V2][IotDB]Add IotDB sink parameter check (#3412)https://github.com/apache/seatunnel/commit/91240a3dcb2.3.0
[Bug][Connector-v2] Fix IoTDB connector sink NPE (#3080)https://github.com/apache/seatunnel/commit/e5edf024332.3.0-beta
[Imporve][Connector-V2] Imporve iotdb connector (#2917)https://github.com/apache/seatunnel/commit/3da11ce19b2.3.0-beta
[DEV][Api] Replace SeaTunnelContext with JobContext and remove singleton pattern (#2706)https://github.com/apache/seatunnel/commit/cbf82f755c2.2.0-beta
[#2606]Dependency management split (#2630)https://github.com/apache/seatunnel/commit/fc047be69b2.2.0-beta
[chore][connector-common] Rename SeatunnelSchema to SeaTunnelSchema (#2538)https://github.com/apache/seatunnel/commit/7dc2a273882.2.0-beta
[Connectors-V2]Support IoTDB Source (#2431)https://github.com/apache/seatunnel/commit/7b78d6c9222.2.0-beta
[Feature][Connector-V2] Support IoTDB sink (#2407)https://github.com/apache/seatunnel/commit/c1bbbd59d52.2.0-beta