5 posts tagged with "SeaTunnel"

In enterprise-level data integration, data consistency is one of the core concerns for technical decision-makers. However, behind this seemingly simple requirement lies complex technical challenges and architectural designs.

When using SeaTunnel for batch and streaming data synchronization, enterprise users typically focus on these questions:

🔍 "How to ensure data integrity between source and target databases?" 🔄 "Can data duplication or loss be avoided after task interruption or recovery?" ⚙️ "How to guarantee consistency during full and incremental data synchronization?"

This article uses Apache SeaTunnel 2.3.13 as its configuration baseline and explains how Read Consistency, Write Consistency, and State Consistency work together. "Zero loss" and "zero duplication" are conditional outcomes, not defaults: they require compatible source and sink semantics, successful checkpoints, correct primary or unique keys, and the documented exactly-once settings.

The examples and terminology below follow the versioned documentation for MySQL-CDC Source, JDBC Source, JDBC Sink, and job environment configuration.

I. Understanding the Three Dimensions of Data Consistency​

In data integration, "consistency" is not a single concept but a set of guarantees covering multiple dimensions. For practical analysis, this article groups the relevant SeaTunnel mechanisms into three dimensions:

Read Consistency​

Read Consistency ensures that data obtained from the source system maintains logical integrity at a specific point in time or event sequence. This dimension addresses the question of "what data to capture":

• Full Read: Obtaining a complete data snapshot at a specific point in time
• Incremental Capture: Accurately recording all data change events (CDC mode)
• Lock-free Snapshot Consistency: When exactly_once = true, using low and high watermarks to reconcile changes that occur during a snapshot

Write Consistency​

Write Consistency ensures data is reliably and correctly written to the target system, addressing "how to write safely":

• Idempotent Writing: Replaying the same key updates one target record when a stable primary/unique key and upsert semantics are available
• Transaction Integrity: Committing the records handled by a sink writer in a checkpoint-aligned transaction when the sink supports it
• Error Handling: Recovering from a completed checkpoint, with replay behavior determined by the source and sink

State Consistency​

State Consistency is the bridge connecting read and write ends, ensuring state tracking and recovery throughout the data synchronization process:

• Position Management: Recording read progress for precise incremental synchronization
• Checkpoint Mechanism: Periodically saving task state
• Checkpoint Recovery: Restoring a completed checkpoint; records after that checkpoint may be replayed unless the sink is idempotent or transactionally exactly-once

II. MySQL Synchronization Architecture: CDC vs. JDBC Mode Comparison​

SeaTunnel provides two mainstream MySQL data synchronization modes: JDBC Batch Mode and CDC Real-time Capture Mode. They serve different workloads and have different recovery and delivery characteristics.

CDC Mode: Low-latency Binlog Change Capture​

The MySQL-CDC connector uses an embedded Debezium framework to read and parse MySQL's binlog change stream:

Core Advantages:

• Low Latency: Reads binlog changes continuously; observed latency depends on source load, network, and job resources
• Reduced Polling: Avoids repeated table polling, while the initial snapshot still consumes source resources
• Completeness: Captures complete events for INSERT/UPDATE/DELETE
• Change Metadata: Emits row-level change events with binlog position metadata

Recovery and Ordering Characteristics:

• Checkpointed binlog filename and position for recovery
• Supports multiple startup modes (Initial snapshot + incremental / Incremental only)
• Preserves the order observed by a source reader; end-to-end ordering still depends on table routing, parallelism, and downstream processing

MySQL-CDC does not turn one source transaction into one atomic downstream transaction. It emits individual row change events, while checkpoint and sink semantics determine the delivery guarantee.

JDBC Mode: SQL-based Batch Synchronization Solution​

The JDBC connector reads data from MySQL through SQL queries, suitable for periodic full synchronization or low-frequency change scenarios:

Core Advantages:

• Simple Development: Based on standard SQL, flexible configuration
• Full Synchronization: Suitable for initializing large amounts of data
• Filtering Capability: Supports complex WHERE condition filtering
• Parallel Loading: Multi-shard parallel reading based on primary key or range

Recovery Characteristics:

• Tracks JDBC splits, not a row offset inside an in-flight split
• Reassigns or replays unfinished splits after failure
• Table-level parallel processing

Therefore, JDBC Source recovery is split-level. A failed in-flight split can be read again from its boundary, so duplicate prevention must be provided by an idempotent or transactionally exactly-once sink.

III. Read Consistency: How to Ensure Complete Source Data Capture​

CDC Mode: Binlog-based Precise Incremental Reading​

The MySQL-CDC connector's read consistency is based on two core mechanisms: Initial Snapshot and Binlog Position Tracking.

Startup Modes and Consistency Guarantee:

SeaTunnel's MySQL-CDC provides multiple startup modes to meet consistency requirements for different scenarios:

1. Initial Mode: Creates a full snapshot and then continues with incremental binlog reading. Set exactly_once = true when the snapshot must backfill changes between its low and high watermarks.

   MySQL-CDC {
     startup.mode = "initial"
     exactly_once = true
   }
   

2. Latest Mode: Only captures the latest changes after connector startup

   MySQL-CDC {
     startup.mode = "latest"
   }
   

3. Specific Mode: Starts synchronization from specified binlog position

   MySQL-CDC {
     startup.mode = "specific"
     startup.specific-offset.file = "mysql-bin.000003"
     startup.specific-offset.pos = 4571
   }
   

There is also an earliest startup mode, which starts from the earliest available offset, and a timestamp startup mode (startup.timestamp), which starts from a user-supplied millisecond timestamp.

JDBC Mode: Shard-based Efficient Batch Reading​

The JDBC connector supports parallel reading through a configurable sharding strategy:

Sharding Strategy and Consistency:

• Primary/Unique Key Sharding: Splits a table by a supported key when one is available
• Configured Partition Column: Uses partition_column when automatic key discovery is not suitable
• Even or Sampled Splitting: Selects a split strategy according to the key distribution and configured thresholds

Example configuration for SeaTunnel JDBC reading shards:

Jdbc {
  url = "jdbc:mysql://source_mysql:3306/test"
  driver = "com.mysql.cj.jdbc.Driver"
  user = "root"
  password = "password"
  table_path = "test.users"
  split.size = 10000
  split.even-distribution.factor.upper-bound = 100
  split.even-distribution.factor.lower-bound = 0.05
  split.sample-sharding.threshold = 1000
}

Through this approach, SeaTunnel achieves:

• Parallel processing of independent splits
• Checkpoint tracking of pending split state
• Replay of an unfinished split from its split boundary after recovery

This is not row-level breakpoint resume. If replay could reach the target twice, use target primary/unique keys with idempotent upsert or enable a supported exactly-once sink.

IV. Write Consistency: How to Ensure Target Data Accuracy​

In the data writing phase, SeaTunnel provides configurable mechanisms for controlling replay and transaction behavior at the target MySQL database.

Idempotent Writing: Ensuring No Data Duplication​

SeaTunnel's JDBC Sink connector implements idempotent writing through multiple strategies:

Upsert Mode:

Example configuration for idempotent writing:

Jdbc {
  url = "jdbc:mysql://target_mysql:3306/test"
  driver = "com.mysql.cj.jdbc.Driver"
  user = "root"
  password = "password"
  generate_sink_sql = true
  database = "test"
  table = "users"
  primary_keys = ["id"]
  enable_upsert = true
}

Batch Commit and Optimization:

JDBC Sink uses explicit, fixed configuration for batching and retries:

• Fixed Batch Size: batch_size controls how many buffered records trigger a flush
• Checkpoint-aligned Flush: Buffered records are also flushed as part of checkpoint processing
• Configured Retries: max_retries controls batch execution retries and defaults to 0; it must remain 0 when XA exactly-once is enabled

Distributed Transaction: XA Guarantee and Two-Phase Commit​

For connector paths that support it, JDBC Sink coordinates per-writer XA transactions with SeaTunnel checkpoints:

Example configuration for enabling XA distributed transactions:

Jdbc {
  url = "jdbc:mysql://target_mysql:3306/test"
  driver = "com.mysql.cj.jdbc.Driver"
  user = "root"
  password = "password"
  generate_sink_sql = true
  database = "test"
  table = "users"
  primary_keys = ["id"]
  enable_upsert = true
  max_retries = 0
  is_exactly_once = true
  xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
  max_commit_attempts = 3
}

XA Transaction Scope:

• Each sink writer prepares its XA transaction for a checkpoint
• The prepared transaction is committed after the checkpoint completes
• Recovery handles the writer's pending/prepared transaction according to the connector protocol

This provides checkpoint-aligned exactly-once delivery for each supported JDBC sink writer. It does not preserve a source transaction as one downstream transaction, and it is not a single global atomic transaction across multiple tables, writers, or databases. Cross-system business atomicity requires a separate transaction design.

V. State Consistency: Breakpoint Resume and Failure Recovery​

Checkpoint-based state management provides a recovery boundary for supported source and sink connectors.

Distributed Checkpoint Mechanism​

In distributed execution, checkpoints coordinate recoverable task state:

Core Implementation Principles:

1. Position Recording: Records a CDC split offset; JDBC Source records split state but not a row offset inside an in-flight split
2. Checkpoint Trigger: Periodically schedules checkpoints according to checkpoint.interval
3. State Persistence: Persists state information to storage system
4. Failure Recovery: Restores the latest completed checkpoint; work after that checkpoint can be replayed

Conditional End-to-End Delivery Semantics​

SeaTunnel coordinates Source and Sink states through checkpoints. The resulting delivery guarantee depends on both connectors and their configuration:

With an at-least-once sink, replay can produce duplicate writes. Idempotent upsert can absorb duplicates when a stable primary/unique key exists. JDBC XA exactly-once additionally requires is_exactly_once = true, a compatible XA data source, max_retries = 0, checkpointing, and database support.

Checkpoint Configuration Example:

env {
  checkpoint.interval = 5000
  checkpoint.timeout = 60000
}

VI. Practical Configuration: MySQL CDC to MySQL Full + Incremental Sync​

Let's demonstrate how to configure SeaTunnel for reliable MySQL to MySQL data synchronization through a practical example.

Classic CDC Mode Configuration​

The following SeaTunnel 2.3.13 example enables MySQL-CDC snapshot consistency and checkpoint-aligned JDBC XA delivery. The guarantee is conditional on stable source/target primary keys, an XA-capable MySQL driver and server, durable checkpoint storage, and successful checkpoint completion. It is not a global transaction across the two target tables.

env {
  job.mode = "STREAMING"
  parallelism = 3
  checkpoint.interval = 60000
  checkpoint.timeout = 120000
}

source {
  MySQL-CDC {
    url = "jdbc:mysql://source_mysql:3306/test_db"
    username = "root"
    password = "password"
    database-names = [
      "test_db"
    ]
    table-names = [
      "test_db.mysqlcdc_to_mysql_table1",
      "test_db.mysqlcdc_to_mysql_table2"
    ]
    server-id = "5400-5408"

    # Initialization mode (full + incremental)
    startup.mode = "initial"
    exactly_once = true

    # Enable DDL changes
    schema-changes.enabled = true

    # Parallel read configuration
    snapshot.split.size = 8096
    snapshot.fetch.size = 1024
  }
}

transform {
  # Optional data transformation processing
}

sink {
  Jdbc {
    url = "jdbc:mysql://mysql_target:3306/test_db?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "root"
    password = "password"
    generate_sink_sql = true
    database = "${database_name}"
    table = "${table_name}"
    primary_keys = ["${primary_key}"]
    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
    data_save_mode = "APPEND_DATA"
    enable_upsert = true
    max_retries = 0
    is_exactly_once = true
    xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource"
  }
}

Before production use, verify that ${primary_key} resolves for every routed table and that the target has matching primary or unique keys. If those prerequisites are not available, describe the job as at-least-once rather than zero-duplication.

VII. Consistency Validation and Monitoring​

After deployment, consistency must be validated independently. Record a logical cut such as a source binlog position, wait for the target to reach it, and compare fixed snapshots or use a quiesced window. Comparing a changing source with a lagging target does not prove inconsistency or consistency.

Data Consistency Validation Methods​

1. Count Comparison: Compare record counts for the same primary-key range and the same consistency window

   -- Source database
   SELECT COUNT(*) FROM source_db.users;
   
   -- Target database
   SELECT COUNT(*) FROM target_db.users;
   

2. Deterministic Range Digest: Read canonical rows in primary-key order for a bounded range and compute a strong digest such as SHA-256 in a reconciliation process

   SELECT id, name, updated_at
   FROM users
   WHERE id >= ? AND id < ?
   ORDER BY id;
   

   Serialize every field with an explicit NULL marker and unambiguous length/escaping rules before hashing. Compare both the row count and digest for each range. Avoid SUM(CRC32(CONCAT_WS(...))): CRC32 collisions and NULL handling can hide differences.

3. Primary-key Drill-down: When a range differs, compare individual rows by primary key. Random sampling is useful for diagnosis but is not proof of full consistency.

Consistency Monitoring Metrics​

During SeaTunnel task execution, monitor actual connector and checkpoint signals:

• CDCRecordFetchDelay: Delay observed while fetching CDC records
• CDCRecordEmitDelay: Delay observed while emitting CDC records
• Checkpoint Status: Completion, timeout, and failure signals from the engine
• External Reconciliation Results: Count, digest, and row-level differences produced by a separate validation job or data-quality platform

"Write success rate" and "data deviation rate" are not built-in SeaTunnel consistency proofs. Define them in the external monitoring system with an explicit time window and denominator.

VIII. Best Practices and Performance Optimization​

The following recommendations follow the SeaTunnel 2.3.13 connector contracts. Benchmark them with representative data and failure scenarios before production rollout.

Consistency Scenario Configuration Recommendations​

1. High Reliability Scenario (e.g., core business data):

   • Enable MySQL-CDC exactly_once and periodic checkpoints
   • Use JDBC XA only with a compatible driver/database and keep max_retries = 0
   • Configure stable target primary/unique keys and idempotent upsert
   • Store checkpoints durably and test restart, timeout, and prepared-transaction recovery

2. High Performance Scenario (e.g., analytical applications):

   • Use CDC mode + batch writing
   • Disable XA only when at-least-once delivery or idempotent replay is acceptable
   • Increase batch size
   • Optimize parallelism settings

3. Large-scale Initialization Scenario:

   • Prefer MySQL-CDC initial mode when one job must cover snapshot and incremental changes
   • Use JDBC initialization only with a coordinated cutover that records the corresponding binlog position
   • Configure appropriate shard size
   • Adjust parallelism to match server resources
   • Do not switch from JDBC to CDC ad hoc; an uncoordinated cutover can create a gap or overlap

Common Issues and Solutions​

1. Unstable Network Environment:

   • Tune connection timeout and job-level recovery settings
   • Keep JDBC Sink max_retries = 0 when XA exactly-once is enabled
   • Rely on completed checkpoints and verify replay behavior
   • Consider using smaller batch sizes

2. High Concurrency Write Scenario:

   • Tune job parallelism against the target database's connection and write capacity
   • Consider table partitioning or larger batches after measuring lock and transaction pressure

3. Resource-constrained Environment:

   • Reduce parallelism
   • Increase checkpoint interval only after accepting the larger recovery/replay window
   • Optimize JVM memory configuration

IX. Conclusion: SeaTunnel's Path to Consistency Guarantee​

SeaTunnel provides the building blocks for reliable batch and streaming synchronization, but the final guarantee is a property of the complete job configuration and external systems. Source offsets, completed checkpoints, idempotent keys, and sink transactions must be evaluated together.

SeaTunnel's consistency guarantee philosophy can be summarized as:

1. Source Recovery State: CDC offsets or JDBC split state define where recovery resumes
2. Checkpoint Coordination: Completed checkpoints align recoverable source and sink state
3. Explicit Sink Semantics: Idempotent upsert or supported XA determines how replay is handled
4. Independent Verification: Consistent-window reconciliation validates the result

With these prerequisites in place, SeaTunnel can provide zero-loss and zero-duplication delivery for supported connector paths. It does not automatically provide cross-table or cross-database atomicity, and achievable scale and latency must be established by workload-specific testing.

If you have more questions about SeaTunnel's data consistency mechanism, welcome to join the community.

Ask a data engineer whether their pipeline is ETL or ELT and you'll get an instant answer. Old-school engineers say ETL. dbt users say ELT.

Both answers are incomplete. There's a third pattern that more accurately describes what modern data pipelines actually do — and it's been hiding in plain sight: EtLT.

The Three Paradigms​

ETL: Transform Before You Land​

Raw data is extracted, passed through a dedicated transformation tier (Spark, DataStage, Informatica), and only then written to the destination warehouse.

Source ──► [Transform tier] ──► Destination

Pros: The warehouse always holds clean, business-ready data. Compliance controls are enforced before data ever lands.
Cons: The transform tier becomes a bottleneck. Schema changes require coordinated updates across multiple layers. Running a dedicated compute cluster for transforms is expensive.

ELT: Land First, Transform In-Place​

Popularized by dbt. Raw data lands directly in the warehouse (BigQuery, Snowflake, ClickHouse), and SQL does the transformations in place. The transform "tier" is just the warehouse itself.

Source ──► Destination (raw layer) ──► [SQL inside warehouse] ──► Business layer

Pros: Raw data is preserved for auditing. You reuse warehouse compute. Iteration is fast.
Cons: Sensitive fields — SSNs, email addresses, phone numbers — land in plaintext. There's no second chance to mask them once they're in the warehouse. Data quality issues only surface after Load, at which point downstream models may already be contaminated.

EtLT: A Lightweight Transform In-Transit​

Source ──► [tiny t] ──► Destination (sanitized raw layer) ──► [T inside warehouse]

The tiny t is a small set of row-level transformations that happen while data is in flight:

The big T (post-Load Transform) is where actual business logic lives: multi-table JOINs, metric calculations, aggregations, ML feature engineering.

Why Engineers Keep Overlooking EtLT​

The reason is tooling — not concept.

Legacy ETL tools (Informatica, DataStage) made transformations expensive and complex. Engineers overcorrected by pushing all logic into the pipeline, which made pipelines brittle.

Modern ELT tools (Airbyte, Fivetran) swung to the opposite extreme: move data from source to destination with almost no in-transit processing, then let dbt handle everything.

The gap neither camp fills: when you need both in-transit operations (masking, filtering) and complex analytical SQL in the destination, you end up with awkward workarounds in both tool families.

EtLT fills exactly that gap.

SeaTunnel Is Built for EtLT​

Apache SeaTunnel describes itself in its official documentation as an "EL(T) data integration platform" — the parentheses around T are intentional. The Transform step is lightweight and optional. This is not a marketing choice; it's an architectural constraint.

The Three-Layer Model Maps Directly to EtLT​

SeaTunnel's execution model has three stages: Source → Transform → Sink.

Source (E)
  └──► Transform (tiny t)  ← optional, lightweight processing
        └──► Sink (L)
              └──► [Warehouse SQL / dbt] (T)  ← big T lives here

SeaTunnel draws a clear line around what Transform can do. The official docs state:

"Transform can only be used for some simple transformations of data, such as converting a column to uppercase/lowercase, modifying column names, or splitting one column into multiple columns."

This describes the intended scope of the built-in transforms. In the current Zeta SQL Transform, JOIN and GROUP BY are not supported, so cross-table joins and aggregations belong in the destination system or another dedicated processing layer.

Built-in Transforms Cover the Typical tiny t Operations​

SeaTunnel provides both map and flat-map transform interfaces. Its current built-in transforms focus on processing individual records and schemas rather than cross-row aggregation, which makes them a practical fit for the tiny t layer.

Where EtLT Matters Most: CDC Pipelines​

Real-time CDC sync is one of SeaTunnel's core use cases — and it's also where pure ELT breaks down for compliance.

Here's the fundamental problem: once a value from a binlog event lands in your warehouse, there is no second chance to mask it. A phone number or SSN written to a ClickHouse table can't be un-written by a downstream dbt model. The original plaintext is already persisted.

EtLT solves what ELT cannot: it applies compliance transformations inside the only window that exists before the data reaches its destination.

A minimal configuration shape for a SeaTunnel CDC job with tiny t transforms is shown below. Replace the example credentials and endpoints before running it:

env {
  job.mode = "STREAMING"
}

source {
  MySQL-CDC {
    plugin_output = "raw_user_info"
    url = "jdbc:mysql://localhost:3306/orders"
    username = "seatunnel"
    password = "change-me"
    server-id = 5601-5604
    table-names = ["orders.user_info"]
  }
}

transform {
  # Mask phone numbers in-transit — the raw value never reaches the warehouse
  Replace {
    plugin_input = "raw_user_info"
    plugin_output = "masked_user_info"
    replace_field = "phone"
    pattern = "(\\d{3})\\d{4}(\\d{4})"
    replacement = "$1****$2"
    is_regex = true
  }
}

sink {
  # L: land into the ClickHouse raw layer
  Clickhouse {
    plugin_input = "masked_user_info"
    host = "clickhouse-host:8123"
    database = "raw"
    table = "user_info"
    username = "default"
    password = "change-me"
    primary_key = "id"
    support_upsert = true
    allow_experimental_lightweight_delete = true
  }
}

Once the data is in ClickHouse, you build wide tables, compute retention metrics, and run analytical queries — that's the big T. dbt models are a natural fit here.

SeaTunnel + dbt: Full-Stack EtLT​

dbt owns the big T after Load. SeaTunnel owns everything from source to Load (including tiny t). They're complementary, not competing.

Source
  └── SeaTunnel (E + tiny t + L)
        └── dbt (T)
              └── BI / ML

SeaTunnel handles data arrival and in-transit hygiene. dbt handles data modeling and business logic. Each tool does what it's good at and nothing more.

The Trap: Trying to Push Big T Into SeaTunnel​

Because SeaTunnel supports SQL Transform, engineers sometimes try writing GROUP BY aggregations there. The current Zeta SQL Transform explicitly rejects GROUP BY and JOIN queries.

If you need pre-Load aggregation, you have two real options:

1. Do it in the destination system — this is the whole point of EtLT and ELT.
2. Use a dedicated processing job outside the SeaTunnel transform chain — at that point you are building a full streaming ETL pipeline rather than keeping the transformation in the tiny t layer.

Clear boundaries make failures easier to trace. When tiny t and big T are collapsed into the same layer, it becomes nearly impossible to reason about where something went wrong.

Summary​

SeaTunnel is a natural fit for EtLT — not because it claims the label, but because its Source/Transform/Sink separation, current built-in transform scope, and explicit "EL(T)" positioning all point to the same shape: move data fast, sanitize lightly in-flight, and leave the heavy lifting to the destination.

Further Reading​

• Apache SeaTunnel — About — official EL(T) positioning (see the "SeaTunnel Work Flowchart" section)
• SeaTunnel Transform Connectors — full list of built-in lightweight transforms
• dbt — the natural partner for the big T in EtLT
• SeaTunnel API Deep Dive — deep dive into the single-row map contract

Tongcheng Travel's data channel evolved over several years into four parallel systems for offline transfer, real-time integration, Sqoop jobs, and SeaTunnel jobs. Each system solved a problem at a particular stage, but their overlapping capabilities, different execution engines, and separate operational models eventually became a barrier to platform-wide governance.

At an Apache SeaTunnel Meetup, Xiaochen Zhou, who works on the data platform at Tongcheng Travel, explained how the company consolidated those systems into a unified batch and streaming data channel based on the Apache SeaTunnel Zeta Engine. The project had three non-negotiable goals: keep the migration transparent to application teams, prove data consistency before switching traffic, and improve execution efficiency and operational stability.

This article summarizes the architecture, migration safeguards, AI-assisted task generation, validation design, and future direction presented in that session.

For the past two decades, most enterprise data engineering systems have been built on one default assumption:

People understand the system. The system executes the pipeline.

Engineers understand the business context, break a requirement into steps, write SQL, Spark jobs, shell scripts, synchronization tasks, and scheduling workflows, and then let the system run them. The scheduler does not need to understand the business. The sync engine does not need to understand the metric. It only needs to execute the predefined flow reliably.

That model supported the era of data warehouses, data lakes, BI reporting, and batch scheduling very well.

But now that assumption is starting to break down.

Enterprise data systems are becoming more complex in every direction:

• More data sources
• Longer pipelines
• Stronger real-time requirements
• Faster business changes
• More conflicting metric definitions
• More AI application data, model feedback data, vector indexes, and unstructured content

In this environment, enterprises do not just need more pipelines, and they do not just need a better Copilot that can write SQL faster.

They increasingly need a Data Engineering Agent that can understand the system, plan tasks, call tools, validate outcomes, and accumulate experience over time.

In that shift, Apache SeaTunnel becomes especially important.

Because in the agent era, it is not enough for a system to "think." It also has to connect to real data sources, capture changes, execute synchronization, process incremental updates, preserve consistency, and move data to target systems in a reliable and cost-effective way.

In other words:

The agent understands the goal and plans the action. SeaTunnel turns that action into real, reliable, and recoverable data movement.

That is why SeaTunnel is well positioned to become a core execution foundation in the evolution from ETL, ELT, and EtLT to agent-driven data engineering.

ETL to ELT: the first major shift​

Traditional ETL is straightforward:

1. Extract data from the source.
2. Transform it in an intermediate layer.
3. Load the processed result into the target system.

This model fit the early data warehouse era well.

At that time, data sources were relatively limited, the pipeline was easier to understand, and compute resources were more centralized. Enterprises wanted to clean the data, standardize the structure, and define the core logic before loading data into the warehouse.

At its core, ETL is a deterministic pipeline model.

Its key assumption is:

People define the process in advance. The system executes the process.

Later, with the rise of cloud warehouses, data lakes, lakehouse architectures, and elastic compute, ELT became more popular.

ELT changed the order:

1. Extract
2. Load
3. Transform inside the target platform

Instead of transforming everything before loading, enterprises started moving raw or near-raw data into a unified storage layer first, then using the target platform's compute power for downstream modeling and analytics.

ELT solved several ETL limitations:

• It reduced upfront processing complexity.
• It preserved more original data.
• It gave analysts and modeling teams more flexibility later.

But ELT also created a new problem.

If all transformation is delayed until after loading, then dirty source data, schema drift, type mismatches, CDC events, privacy fields, and format inconsistencies all arrive directly in the target system.

That might be acceptable in simple batch scenarios. It becomes much more expensive in real-time synchronization, CDC, multi-table sync, lakehouse ingestion, SaaS API ingestion, and AI-oriented data engineering.

That is where a third pattern becomes more useful:

EtLT

Why EtLT matters​

EtLT is not just a compromise between ETL and ELT.

A more useful way to understand it is:

Extract -> lightweight transform -> Load -> semantic Transform

That means:

• Extract the data
• Apply the minimum engineering transformations required to make the data usable
• Load it into a unified data foundation
• Apply business-level and semantic transformation later

The key idea is the distinction between lowercase t and uppercase T.

Lowercase t is not heavy business modeling. It is the engineering work that must happen before data enters the platform safely and consistently, such as:

• Field projection
• Type mapping
• Format normalization
• Primary key or partition field handling
• Sensitive field masking
• CDC event conversion
• Multi-table routing
• Schema evolution handling
• Pre-ingestion quality validation
• One-read, multi-write patterns
• Rate limiting and parallelism control

These transformations should not always be postponed to the target system. Otherwise, the lakehouse or warehouse becomes full of inconsistent, weakly governed, and semantically unclear raw data.

At the same time, lowercase t should not try to absorb all business logic.

Complex business definitions, KPI semantics, subject-area modeling, and cross-domain aggregation still belong to uppercase T, which should happen in the warehouse, lakehouse, semantic layer, or metric layer.

That is the value of EtLT:

Standardize the data engineering layer before loading, then apply business semantics after loading.

This is exactly the place where SeaTunnel fits naturally.

Its Source, Transform, and Sink architecture is well suited for the lowercase t in EtLT. It can connect heterogeneous systems, apply lightweight transformation during movement, handle CDC, adapt schemas, route multiple tables, and write the result into the target platform.

In an EtLT architecture, SeaTunnel is not just a data mover. It becomes the data integration runtime that prepares data before it enters the unified data foundation.

Why traditional ETL starts to struggle​

Traditional ETL is built for relatively stable pipelines.

You write the rules, draw the DAG, schedule the tasks, and fix failures when they happen.

But modern enterprise data environments are no longer that simple.

Today a single enterprise may operate across:

• OLTP databases
• Kafka streams
• CDC pipelines
• SaaS APIs
• Object storage
• Logs and events
• Lakehouse platforms
• Real-time OLAP systems
• Vector databases
• AI interaction logs
• Model output datasets

The problem is not only that there is more data. The data is also more fragmented, more heterogeneous, and more real-time.

Pipeline length is another issue.

A single business metric may depend on dozens of tables, multiple layers of wide tables, several business domains, and a long chain of definition changes. At that point, many enterprises no longer struggle with "Can we build the workflow?" They struggle with "Can anyone still explain the whole pipeline end to end?"

This is where traditional ETL shows a structural limitation.

• One renamed field can break hundreds of tasks.
• One changed enum can silently shift multiple core metrics.
• One incorrect incremental logic branch can pollute an entire downstream analysis chain.

The scheduler can tell you that a task failed. It usually cannot tell you why that task matters.

The sync tool can move the data. It usually cannot tell you which business metric is now at risk.

The engineer can fix the script. But only if that engineer can first rebuild the missing context.

So the real weakness of traditional ETL is not just performance or reliability.

It is that:

It can execute the process, but it does not understand the system.

Why Copilot is not enough​

Many teams first bring AI into data engineering through Copilot-style workflows:

• Generate SQL
• Complete Spark code
• Draft YAML
• Produce test samples

These capabilities are useful. They improve local productivity.

But they do not solve the deepest problem in enterprise data engineering.

Because the hardest part of data engineering is rarely just code generation.

It is system understanding.

Copilot can help generate a SQL statement, but it does not know the real business meaning of the field.

It can help draft a synchronization task, but it does not know which downstream metrics will be affected by a schema change.

It can help generate a scheduler config, but it does not know whether the change breaks historical consistency or recovery semantics.

What enterprises actually struggle with includes:

• Lineage reasoning
• Dependency analysis
• Semantic understanding
• Metric governance
• Risk estimation
• Impact analysis
• Incremental recovery

These are not just autocomplete problems.

So enterprises do not only need an AI IDE. They increasingly need an agentic data engineering system that can understand the target, decompose tasks, call engineering tools, and verify the result.

The real shift: from pipeline to agent​

If we keep only one conclusion, it is this:

Traditional ETL is "people define the process, systems execute the process." Agentic data engineering is "people define the goal, systems generate the process."

That is not a slogan. It is a change in how work is organized.

In the traditional model, engineers design the task chain first, configure Source, Transform, and Sink, and then let the scheduler execute the pipeline.

The system faces a fixed process.

In the agent model, the input may only be a business goal.

For example:

Add a new gross margin metric for orders and keep it aligned with the finance definition.

Traditionally, the engineer must:

• Identify relevant data sources
• Read table schemas
• Inspect lineage
• Design transformation logic
• Configure sync and scheduling jobs
• Add quality checks
• Run regression validation

In an agent-oriented workflow, the system should be able to generate a sequence of actions around the goal:

• Identify the affected business entities
• Discover candidate data sources
• Analyze upstream lineage
• Decide whether the job belongs to ETL, ELT, or EtLT
• Generate or update the SeaTunnel synchronization task
• Configure full-load or CDC mode
• Apply lightweight transformation
• Write the result into the warehouse or lakehouse
• Trigger data quality validation
• Evaluate downstream impact
• Present the result for human confirmation

That is the real difference.

The breakthrough is not "AI wrote a SQL statement for me."

The breakthrough is:

The system starts generating engineering actions from a business goal.

But this immediately raises a critical question:

When the agent plans a data action, who executes it reliably?

That is exactly where SeaTunnel becomes essential.

SeaTunnel in the agent era: the data integration execution layer​

An agent cannot stop at reasoning and recommendations.

If a Data Engineering Agent decides that a table should be synchronized, a CDC job should be adjusted, a broken pipeline segment should be replayed, or a data slice should be reloaded into the target system, it needs a stable and observable execution layer to carry out that decision.

That execution layer needs several core capabilities.

1. It must connect to many kinds of data sources​

Enterprise data systems are inherently heterogeneous.

An agent cannot live in a world with only one database or one file system. It needs to connect to MySQL, Oracle, PostgreSQL, SQL Server, Kafka, Hive, Iceberg, Doris, ClickHouse, StarRocks, Elasticsearch, S3, HDFS, MongoDB, and many other systems.

SeaTunnel's connector architecture is designed for exactly this kind of environment. It abstracts Source, Transform, and Sink through a consistent plugin model so heterogeneous systems can be integrated in a unified way.

2. It must support batch, streaming, CDC, and large-scale synchronization​

The agent era does not run on a single data movement pattern.

It needs:

• One-time full migration
• Continuous CDC
• Offline batch movement
• Real-time synchronization
• Single-table sync
• Multi-table or database-level sync

SeaTunnel is valuable here because it is not just a script wrapper for ETL. It is a real data integration runtime that can support full load, incremental sync, real-time processing, CDC, and multi-table movement in the same ecosystem.

3. It must handle the lowercase t in EtLT​

Agentic systems do not need every business transformation to happen inside the sync layer.

But they do need the sync layer to complete the minimum engineering transformation required to make the data trustworthy and usable before it lands in the platform.

SeaTunnel's Transform layer is a strong fit for:

• Field mapping
• Type conversion
• Filtering
• Column projection
• Data masking
• Routing
• Simple reshaping

That is exactly the role of the lowercase t in EtLT:

Do not overload the movement layer with heavy business modeling, but make the data governable and ready for the next stage.

4. It must provide consistency, fault tolerance, and recovery​

An agent can decide that a broken link should be replayed.

But replay only matters if the underlying system can recover correctly.

The execution layer still needs checkpointing, failure recovery, state handling, restart behavior, and strong delivery guarantees where needed.

A reasoning layer without a reliable execution layer becomes a planner without hands.

That is why execution quality still matters as much as intelligence.

What the future stack starts to look like​

If we look one step ahead, enterprise data engineering increasingly resembles a layered operating system rather than a collection of disconnected pipelines.

In that stack:

• The semantic layer defines the business model.
• Metadata provides structure and context.
• Memory accumulates operational experience.
• The planning layer turns goals into actions.
• The execution layer performs synchronization, CDC, movement, replay, and recovery.

SeaTunnel belongs to this execution layer.

That placement is important.

The future is not "put a large language model on top of ETL."

The future is a coordinated system where reasoning and execution are separated clearly:

• The agent decides what should happen.
• SeaTunnel ensures that it actually happens in a reliable way.

The evolution in one sentence​

ETL built data pipelines.

ELT moved raw data into a unified platform first.

EtLT rebalanced pre-load engineering standardization and post-load semantic modeling.

The agent era pushes data engineering one step further:

From fixed pipelines to goal-driven systems.

In that world, SeaTunnel is not just a synchronization tool.

It becomes a practical execution foundation for agentic data engineering.

Agents make data systems understand goals.

EtLT makes ingestion more controllable.

SeaTunnel turns those goals into reliable data engineering actions.

That is the deeper change now happening across enterprise data engineering.

In the field of data integration and synchronization, Apache SeaTunnel is undoubtedly one of the most popular tools today. This series will dive deep into its advanced usage.

The first article starts with one of SeaTunnel’s core concepts: Data Flow. It analyzes the underlying principles, such as how data flows and is transformed, and explains how this concept applies to complex scenarios through examples.

One-Sentence Summary: The Conclusion First​

SeaTunnel is not a linear “source → sink” tool.

• It is a DAG execution engine driven by DataStream / DataFlow.

In SeaTunnel's Zeta engine, multiple upstream branches converging on one downstream stage is a direct manifestation of this model.

1. SeaTunnel’s Core Concept: Data Flow​

Inside SeaTunnel, everything revolves around data flow.

What is a data flow?​

Data flow = a stream of Records with a consistent structure and Schema.

It is not a table, not a file, and not a SQL result set.

It is:

Record1 → Record2 → Record3 → ...

Every plugin “operates on data flow”​

In practice, a plugin either produces a stream, consumes a stream, or does both when it sits in the middle of a pipeline.

2. The Real Meaning of plugin_output / plugin_input​

You may have been “using” them for a long time, but now it is time to truly “understand” them.

1️⃣ plugin_output​

plugin_output = "source_data_output_1"

Its meaning is not simply “a name”.

It means:

Assign a unique ID to the data flow produced by the current plugin.

You can understand it as:

DataStream<ID = source_data_output_1>

2️⃣ plugin_input​

plugin_input = "source_data_output_1"

Its meaning is:

This plugin wants to consume a specific data flow.

In one sentence​

plugin_output / plugin_input = the “connection ports” of data flow

3. SeaTunnel’s DAG Model: You Are Already Using It​

Under SeaTunnel's Zeta engine and its LogicalDag model, the successful experiment you ran is essentially this:

SourceA ┐
        ├──► Downstream Stage
SourceB ┘

Internally, SeaTunnel builds a DAG like this:​

DataStream A ┐
             ├──► Downstream Vertex
DataStream B ┘

Key question: Why can they converge?​

Because:

In Zeta's logical graph, one downstream vertex can be connected to multiple upstream data flows.

That is a graph-level capability first, not a blanket guarantee that every runtime plugin instance accepts multiple plugin_input values directly.

In Zeta, SeaTunnel internally does the following:

• Takes multiple input streams;
• Represents them as a multiple-input vertex in the LogicalDag;
• Splits the graph into executable pipelines during physical planning.

A precise caveat is important here:

• Zeta's DAG model can express multiple-input vertices.
• The current Flink and Spark starter implementations still reject multiple plugin_input values in a single plugin instance.
• So treat this as a Zeta / logical-graph concept, not as a universal sink-plugin contract.

4. How Is This Fundamentally Different from “SQL / ETL” Thinking?​

This is where many people get confused.

The SQL world​

SELECT * FROM A
UNION ALL
SELECT * FROM B

• This is result-set semantics.

The SeaTunnel world​

Record stream from A
Record stream from B
↓
A downstream stage processes the records according to the configured graph

• This is stream semantics.

In Zeta, as long as the graph and Schema assumptions are valid, the downstream stage can be planned from those upstream streams.

5. The Role of Schema in Data Flow: Must Remember​

Data flow = Record + Schema

Prerequisites for data flow convergence in SeaTunnel:​

• The number of fields must be consistent.
• Field types must be compatible.
• Field names must be aligned, or at least mappable.

Otherwise:

• The job may fail at runtime.
• Or the Sink may fail to write the data.

When you said “the target fields are definitely aligned”, that is exactly why your experiment succeeded.

6. Formal Definition of SeaTunnel’s Data Flow Model​

You can directly use the following standard wording in future design discussions, solution explanations, or documentation:

SeaTunnel uses DataStream as its core abstraction.

Source plugins generate data streams, Transform plugins process data streams and output new data streams, and Sink plugins consume upstream data streams and write the data into external systems.

In SeaTunnel Zeta's LogicalDag, multiple data streams can converge at one downstream vertex. Whether that convergence is expressed directly by a specific sink plugin or through an intermediate stage depends on the engine/runtime implementation.

7. Direct Impact on Your Builder / Strategy Design​

Now you can be very certain about three things:

1️⃣ The Builder must support N Sources → M Sinks​

It is not a 1→1 model. It is a graph model.

2️⃣ plugin_output is a “first-class citizen”​

If someone does not set plugin_output in your Builder:

• You should automatically generate one for them.

This is a platform-level capability.

3️⃣ In Zeta, the logical graph may contain multiple-input vertices​

Even if the DSL shows only one:

plugin_input = "s1"

Your Builder should model upstream relationships as:

Set<DataStream>

instead of hard-coding every downstream step as a single linear String-to-String hop.

8. Key Facts You Have Already Verified​

Here are the conclusions you have already validated through practice:

✅ SeaTunnel is a DAG, not a linear ETL tool.
✅ In Zeta's DAG model, multiple upstream branches can converge on one downstream stage.
✅ That convergence is a graph-level concept, not automatically a direct multi-input sink contract in every engine implementation.
✅ Schema alignment is the prerequisite.
✅ The DSL describes data flow, not SQL.

9. Summary​

SeaTunnel has only three core roles​

Source     →   Transform   →   Sink
(produces)     (modifies)      (consumes)
data flow      data flow       data flow

How are data flows “connected”?​

Just remember this connection rule:

The connection depends on two things:

• plugin_output: What is the name of the data flow I produce?
• plugin_input: Which upstream data flow am I consuming?

For example, two upstream branches converging on one downstream stage in Zeta:

SourceA ┐
        ├──► Downstream Stage
SourceB ┘

One Source → two Sinks:

         ┌──► SinkA
Source ──┤
         └──► SinkB

Two independent flows in one conf file:

SourceA ───► SinkA

SourceB ───► SinkB