⭐️
Milvus
Milvus数据接收器
描述
Milvus sink连接器将数据写入Milvus或Zilliz Cloud，它具有以下功能：
- 支持按分区读写数据
- 支持从元数据列写入动态模式数据
- json数据将转换为json字符串进行写入
- 自动重试以绕过 ratelimit 限制 和 grpc 限制
主要特性
##数据类型映射
| Milvus数据类型 | SeaTunnel 数据类型 |
|---|---|
| INT8 | TINYINT |
| INT16 | SMALLINT |
| INT32 | INT |
| INT64 | BIGINT |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| BOOL | BOOLEAN |
| JSON | STRING |
| ARRAY | ARRAY |
| VARCHAR | STRING |
| FLOAT_VECTOR | FLOAT_VECTOR |
| BINARY_VECTOR | BINARY_VECTOR |
| FLOAT16_VECTOR | FLOAT16_VECTOR |
| BFLOAT16_VECTOR | BFLOAT16_VECTOR |
| SPARSE_FLOAT_VECTOR | SPARSE_FLOAT_VECTOR |
Sink 选项
| 名字 | 类型 | 是否必传 | 默认值 | 描述 |
|---|---|---|---|---|
| url | String | 是 | - | 连接到Milvus或Zilliz Cloud的URL。 |
| token | String | 是 | - | 用户：密码 |
| database | String | 否 | - | 将数据写入哪个数据库，默认为源数据库。 |
| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 当表不存在时自动创建表。 |
| enable_auto_id | boolean | 否 | false | 主键列启用autoId。 |
| enable_upsert | boolean | 否 | false | 是否启用upsert。 |
| enable_dynamic_field | boolean | 否 | true | 是否启用带动态字段的创建表。 |
| batch_size | int | 否 | 1000 | 写入批大小。 |
| partition_key | String | 否 | Milvus分区键字段 |
任务示例
sink {
Milvus {
url = "http://127.0.0.1:19530"
token = "username:password"
batch_size = 1000
}
}
变更日志
Change Log
| Change | Commit | Version |
|---|---|---|
| [Feature][Transform-V2] Support vector series sql function (#9765) | https://github.com/apache/seatunnel/commit/a40114cf7a | 2.3.12 |
| [Improve][Connector-milvus]update milvus-sdk-java to 2.5.11 (#9710) | https://github.com/apache/seatunnel/commit/08ebbaa8bd | 2.3.12 |
| [Chore] fix typos filed -> field (#9757) | https://github.com/apache/seatunnel/commit/e3e1c67d29 | 2.3.12 |
| [Improve][Connector-V2] Optimize Milvus doc and e2e test case (#9766) | https://github.com/apache/seatunnel/commit/e67466f73e | 2.3.12 |
| [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (#9671) | https://github.com/apache/seatunnel/commit/9212a77140 | 2.3.12 |
| [Improve][API] Add metadata schema into catalog table (#9586) | https://github.com/apache/seatunnel/commit/385814e7f1 | 2.3.12 |
| [Feature][Transform] Support define sink column type (#9114) | https://github.com/apache/seatunnel/commit/ab7119e507 | 2.3.11 |
| [Feature][Checkpoint] Add check script for source/sink state class serialVersionUID missing (#9118) | https://github.com/apache/seatunnel/commit/4f5adeb1c7 | 2.3.11 |
| [improve] milvus options (#9165) | https://github.com/apache/seatunnel/commit/5247e17640 | 2.3.11 |
| [Fix][Connector-V2] Fix load state check in MilvusSourceReader to consider partition-level status (#8937) | https://github.com/apache/seatunnel/commit/bde235090b | 2.3.10 |
| [Improve][dist]add shade check rule (#8136) | https://github.com/apache/seatunnel/commit/51ef800016 | 2.3.9 |
| [Improve][Core] Refactor common options of column/row (#7911) | https://github.com/apache/seatunnel/commit/d1582afee6 | 2.3.9 |
| [Feature][connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885) | https://github.com/apache/seatunnel/commit/6a31f91729 | 2.3.9 |
| [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786) | https://github.com/apache/seatunnel/commit/6b7c53d03c | 2.3.9 |
| [Fix][Connector-V2] Fix known directory create and delete ignore issues (#7700) | https://github.com/apache/seatunnel/commit/e2fb679577 | 2.3.8 |
| [Improve][Connector-V2] Optimize milvus code (#7691) | https://github.com/apache/seatunnel/commit/1eddb8e1b1 | 2.3.8 |
| [Improve][Connector-V2] Optimize milvus-connector config code (#7658) | https://github.com/apache/seatunnel/commit/f831f7a5ec | 2.3.8 |
| [Improve][Connector-V2] update vectorType (#7446) | https://github.com/apache/seatunnel/commit/1bba72385b | 2.3.8 |
| [Improve][API] Move catalog open to SaveModeHandler (#7439) | https://github.com/apache/seatunnel/commit/8c2c5c79a1 | 2.3.8 |
| [Feature][Connector-V2] Fake Source support produce vector data (#7401) | https://github.com/apache/seatunnel/commit/6937d10ac3 | 2.3.8 |
| [Feature][Connector-V2][Milvus] Support Milvus source & sink (#7158) | https://github.com/apache/seatunnel/commit/0c69b9166e | 2.3.6 |