Milvus
Milvus 源连接器
描述
这个Milvus源连接器从Milvus或Zilliz Cloud读取数据,它具有以下功能:
- 支持按分区读写数据
- 支持将动态模式数据读入元数据列
- json数据将转换为json字符串,并将sink转换为json
- 自动重试以绕过速率限制和grpc限制
关键特性
数据类型映射
Milvus 数据类型 | SeaTunnel 数据类型 |
---|---|
INT8 | TINYINT |
INT16 | SMALLINT |
INT32 | INT |
INT64 | BIGINT |
FLOAT | FLOAT |
DOUBLE | DOUBLE |
BOOL | BOOLEAN |
JSON | STRING |
ARRAY | ARRAY |
VARCHAR | STRING |
FLOAT_VECTOR | FLOAT_VECTOR |
BINARY_VECTOR | BINARY_VECTOR |
FLOAT16_VECTOR | FLOAT16_VECTOR |
BFLOAT16_VECTOR | BFLOAT16_VECTOR |
SPARSE_FLOAT_VECTOR | SPARSE_FLOAT_VECTOR |
源选项
名称 | 类型 | 必需 | 默认值 | 描述 |
---|---|---|---|---|
url | String | 是 | - | 连接到Milvus或Zilliz Cloud的URL. |
token | String | 是 | - | 用户:密码 |
database | String | 是 | default | 从哪个数据库读取数据. |
collection | String | 否 | - | 如果设置,将只读取一个集合,否则将读取数据库下的所有集合. |
任务示例
source {
Milvus {
url = "http://127.0.0.1:19530"
token = "username:password"
database = "default"
}
}
变更日志
Change Log
Change | Commit | Version |
---|---|---|
[Fix][Connector-V2] Fix load state check in MilvusSourceReader to consider partition-level status (#8937) | https://github.com/apache/seatunnel/commit/bde235090 | 2.3.10 |
[Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef80001 | 2.3.9 |
[Improve][Core] Refactor common options of column/row (#7911) | https://github.com/apache/seatunnel/commit/d1582afee | 2.3.9 |
[Feature][connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885) | https://github.com/apache/seatunnel/commit/6a31f9172 | 2.3.9 |
[Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03 | 2.3.9 |
[Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700) | https://github.com/apache/seatunnel/commit/e2fb67957 | 2.3.8 |
[Improve][Connector-V2] Optimize milvus code (#7691) | https://github.com/apache/seatunnel/commit/1eddb8e1b | 2.3.8 |
[Improve][Connector-V2] Optimize milvus-connector config code (#7658) | https://github.com/apache/seatunnel/commit/f831f7a5e | 2.3.8 |
[Improve][Connector-V2] update vectorType (#7446) | https://github.com/apache/seatunnel/commit/1bba72385 | 2.3.8 |
[Improve][API] Move catalog open to SaveModeHandler (#7439) | https://github.com/apache/seatunnel/commit/8c2c5c79a | 2.3.8 |
[Feature][Connector-V2] Fake Source support produce vector data (#7401) | https://github.com/apache/seatunnel/commit/6937d10ac | 2.3.8 |
[Feature][Connector-V2][Milvus] Support Milvus source & sink (#7158) | https://github.com/apache/seatunnel/commit/0c69b9166 | 2.3.6 |