SeaTunnel With Spark
Apache Spark is the right choice when your team already runs Spark and wants SeaTunnel jobs to fit into that batch or mixed workload environment. If you are evaluating SeaTunnel from scratch and do not need Spark specifically, start with SeaTunnel Engine first.
Start Here
Use this path if you want to run SeaTunnel on Spark:
SeaTunnel provides separate Spark 3 starters so existing Spark 3.3 deployments can remain on their current runtime while Spark 3.5 deployments use the Spark 3.5.8 starter:
seatunnel-spark-3-starter.jarfor Spark 3.3.xseatunnel-spark-3.5-starter.jarfor Spark 3.5.x
For an existing Spark 3.4 deployment, retain the existing seatunnel-spark-3-starter.jar
and start-seatunnel-spark-3-connector-v2 launcher. This change does not add a separate
Spark 3.4 starter or extend its compatibility coverage.
The Spark 3.5 starter reuses the translation layer compiled against Spark 3.3. Row-encoder, batch-write and bounded micro-batch sink tests exercise selected paths on Spark 3.5.8, but do not prove compatibility of every Catalyst or DataSource V2 path. Other binary incompatibilities may surface only when those paths execute on the target runtime. Validate your connectors and deployment mode before upgrading.
The public Spark 3.5 starter uses the batch read() and write().save() pipeline;
use a batch job configuration with this launcher. The micro-batch test runs a finite
Spark file stream through the row encoder and SeaTunnel streaming sink adapter. It
does not validate the SeaTunnel streaming source, checkpoint restore, or streaming
jobs submitted through the public starter. Spark 3.5 connector-container tests are
scheduled-only (testInPR=false); normal PR CI does not run that runtime matrix.
The Spark 3.5 Windows .cmd launcher still passes a command string to spark-submit.cmd;
it does not use the Unix launcher's argument-file protocol. Arbitrary arguments
containing embedded quotes or Windows command-shell metacharacters are not
supported, and native Windows launcher behavior has not been validated. Do not
pass untrusted values through this launcher; use the Unix launcher when exact
argument preservation is required.
The packaged Windows tests exercise help, JVM startup failure, and opening quoted
configuration paths containing spaces, !, and &. The path tests stop at a
configuration parse error, before Spark submission; they do not establish exact
argument preservation by spark-submit.cmd. Native Windows CI must pass on the
revision being validated before treating these launcher cases as verified.
The Unix packaged test runs the real starter JVM and checks argument preservation
for paths containing spaces and both quote characters. It captures the arguments
at spark-submit; it does not submit a Spark job.
When To Choose Spark
Spark is usually the right engine when:
- your organization already runs Spark clusters in production
- the surrounding workloads are mainly batch-oriented
- you want SeaTunnel to align with an existing Spark ecosystem and deployment model
Spark-Specific Configuration
Spark-specific job options live in the env block and use the spark. prefix.
Example:
env {
spark.app.name = "example"
spark.sql.catalogImplementation = "hive"
spark.executor.memory = "2g"
spark.executor.instances = "2"
spark.yarn.priority = "100"
spark.dynamicAllocation.enabled = "false"
}
Command Line Example
Spark on YARN cluster mode:
./bin/start-seatunnel-spark-3-connector-v2.sh --master yarn --deploy-mode cluster --config config/example.conf
Spark on YARN client mode:
./bin/start-seatunnel-spark-3-connector-v2.sh --master yarn --deploy-mode client --config config/example.conf
For Spark 3.5, use start-seatunnel-spark-3.5-connector-v2.sh with the same options.
Minimal Example Job
The example below runs on Spark and prints generated records to the console.
env {
parallelism = 1
spark.app.name = "example"
spark.sql.catalogImplementation = "hive"
spark.executor.memory = "2g"
spark.executor.instances = "1"
spark.yarn.priority = "100"
spark.dynamicAllocation.enabled = "false"
}
source {
FakeSource {
plugin_output = "fake"
row.num = 16
schema = {
fields {
name = "string"
age = "int"
}
}
}
}
transform {
FieldMapper {
plugin_input = "fake"
plugin_output = "fake1"
field_mapper = {
age = age
name = new_name
}
}
}
sink {
Console {
plugin_input = "fake1"
}
}
If you need more transform options, see Transforms Catalog and Transform Common Options.
Running From A Source Checkout
If you are running examples from the repository source tree, the example module is:
seatunnel-examples/seatunnel-spark-connector-v2-example
The example entry point is:
org.apache.seatunnel.example.spark.v2.SeaTunnelApiExample
Next Steps
- Quick Start With Spark
- Spark Translation Layer
- Transforms Catalog
- SeaTunnel Engine if you want to compare against the default engine