跳到主要内容

SeaTunnel 2.1.3 released! Introducing in Assert Sink connector and NullRate, Nulltf Transform

· 阅读需 3 分钟

More than a month after the release of Apache SeaTunnel(Incubating) 2.1.2, we have been collecting user and developer feedback to bring you version 2.1.3. The new version introduces the Assert Sink connector, which is an inurgent need in the community, and two Transforms, NullRate and Nulltf. Some usability problems in the previous version have also been fixed, improving stability and efficiency.

This article will introduce the details of the update of Apache SeaTunnel(Incubating) version 2.1.3.

Major feature updates​

Introduces Assert Sink connector​

Assert Sink connector is introduced in SeaTunnel version 2.1.3to verify data correctness. Special thanks to Lhyundeadsoul for his contribution.

Add two Transforms​

In addition, the 2.1.3 version also adds two Transforms, NullRate and Nulltf, which are used to detect data quality and convert null values ​​in the data to generate default values. These two Transforms can effectively improve the availability of data and reduce the frequency of abnormal situations. Special thanks to wsyhj and Interest1-wyt for their contributions.

At present, SeaTunnel has supported 9 types of Transforms including Common Options, Json, NullRate, Nulltf, Replace, Split, SQL, UDF, and UUID, and the community is welcome to contribute more Transform types.

For details of Transform, please refer to the official documentation: https://seatunnel.apache.org/docs/2.1.3/category/transform

ClickhouseFile connector supports Rsync data transfer method now​

At the same time, SeaTunnel 2.1.3 version brings Rsync data transfer mode support to ClickhouseFile connector, users can now choose SCP and Rsync data transfer modes. Thanks to Emor-nj for contributing to this feature.

Specific feature updates:​

Optimization​

  • Refactored Spark TiDB-related parameter information
  • Refactor the code to remove redundant code warning information
  • Optimize connector jar package loading logic
  • Add Plugin Discovery module
  • Add documentation for some modules
  • Upgrade common-collection from version 4 to 4.4
  • Upgrade common-codec version to 1.13

Bug Fix​

In addition, in response to the feedback from users of version 2.1.2, we also fixed some usability issues, such as the inability to use the same components of Source and Sink, and further improved the stability.

  • Fixed the problem of Hudi Source loading twice
  • Fix the problem that the field TwoPhaseCommit is not recognized after Doris 0.15
  • Fixed abnormal data output when accessing Hive using Spark JDBC
  • Fix JDBC data loss when partition_column (partition mode) is set
  • Fix KafkaTableStream schema JSON parsing error
  • Fix Shell script getting APP_DIR path error
  • Updated Flink RunMode enumeration to get correct help messages for run modes
  • Fix the same source and sink registered connector cache error
  • Fix command line parameter -t( — check) conflict with Flink deployment target parameter
  • Fix Jackson type conversion error problem
  • Fix the problem of failure to run scripts in paths other than SeaTunnel_Home

Acknowledgment​

Thanks to all the contributors (GitHub ID, in no particular order,), it is your efforts that fuel the launch of this version, and we look forward to more contributions to the Apache SeaTunnel(Incubating) community!

leo65535, CalvinKirs, mans2singh, ashulin, wanghuan2054, lhyundeadsoul, tobezhou33, Hisoka-X, ic4y, wsyhj, Emor-nj, gleiyu, smallhibiscus, Bingz2, kezhenxu94, youyangkou, immustard, Interest1-wyt, superzhang0929, gaaraG, runwenjun