Skip to main content
Version: Next

SeaTunnel With Spark

Apache Spark is the right choice when your team already runs Spark and wants SeaTunnel jobs to fit into that batch or mixed workload environment. If you are evaluating SeaTunnel from scratch and do not need Spark specifically, start with SeaTunnel Engine first.

Start Here​

Use this path if you want to run SeaTunnel on Spark:

SeaTunnel provides separate Spark 3 starters so existing Spark 3.3 deployments can remain on their current runtime while Spark 3.5 deployments use the Spark 3.5.8 starter:

  • seatunnel-spark-3-starter.jar for Spark 3.3.x
  • seatunnel-spark-3.5-starter.jar for Spark 3.5.x

For an existing Spark 3.4 deployment, retain the existing seatunnel-spark-3-starter.jar and start-seatunnel-spark-3-connector-v2 launcher. This change does not add a separate Spark 3.4 starter or extend its compatibility coverage.

The Spark 3.5 starter reuses the translation layer compiled against Spark 3.3. Row-encoder, batch-write and bounded micro-batch sink tests exercise selected paths on Spark 3.5.8, but do not prove compatibility of every Catalyst or DataSource V2 path. Other binary incompatibilities may surface only when those paths execute on the target runtime. Validate your connectors and deployment mode before upgrading.

The public Spark 3.5 starter uses the batch read() and write().save() pipeline; use a batch job configuration with this launcher. The micro-batch test runs a finite Spark file stream through the row encoder and SeaTunnel streaming sink adapter. It does not validate the SeaTunnel streaming source, checkpoint restore, or streaming jobs submitted through the public starter. Spark 3.5 connector-container tests are scheduled-only (testInPR=false); normal PR CI does not run that runtime matrix.

The Spark 3.5 Windows .cmd launcher still passes a command string to spark-submit.cmd; it does not use the Unix launcher's argument-file protocol. Arbitrary arguments containing embedded quotes or Windows command-shell metacharacters are not supported, and native Windows launcher behavior has not been validated. Do not pass untrusted values through this launcher; use the Unix launcher when exact argument preservation is required.

The packaged Windows tests exercise help, JVM startup failure, and opening quoted configuration paths containing spaces, !, and &. The path tests stop at a configuration parse error, before Spark submission; they do not establish exact argument preservation by spark-submit.cmd. Native Windows CI must pass on the revision being validated before treating these launcher cases as verified.

The Unix packaged test runs the real starter JVM and checks argument preservation for paths containing spaces and both quote characters. It captures the arguments at spark-submit; it does not submit a Spark job.

When To Choose Spark​

Spark is usually the right engine when:

  • your organization already runs Spark clusters in production
  • the surrounding workloads are mainly batch-oriented
  • you want SeaTunnel to align with an existing Spark ecosystem and deployment model

Spark-Specific Configuration​

Spark-specific job options live in the env block and use the spark. prefix.

Example:

env {
spark.app.name = "example"
spark.sql.catalogImplementation = "hive"
spark.executor.memory = "2g"
spark.executor.instances = "2"
spark.yarn.priority = "100"
spark.dynamicAllocation.enabled = "false"
}

Command Line Example​

Spark on YARN cluster mode:

./bin/start-seatunnel-spark-3-connector-v2.sh --master yarn --deploy-mode cluster --config config/example.conf

Spark on YARN client mode:

./bin/start-seatunnel-spark-3-connector-v2.sh --master yarn --deploy-mode client --config config/example.conf

For Spark 3.5, use start-seatunnel-spark-3.5-connector-v2.sh with the same options.

Minimal Example Job​

The example below runs on Spark and prints generated records to the console.

env {
parallelism = 1

spark.app.name = "example"
spark.sql.catalogImplementation = "hive"
spark.executor.memory = "2g"
spark.executor.instances = "1"
spark.yarn.priority = "100"
spark.dynamicAllocation.enabled = "false"
}

source {
FakeSource {
plugin_output = "fake"
row.num = 16
schema = {
fields {
name = "string"
age = "int"
}
}
}
}

transform {
FieldMapper {
plugin_input = "fake"
plugin_output = "fake1"
field_mapper = {
age = age
name = new_name
}
}
}

sink {
Console {
plugin_input = "fake1"
}
}

If you need more transform options, see Transforms Catalog and Transform Common Options.

Running From A Source Checkout​

If you are running examples from the repository source tree, the example module is:

  • seatunnel-examples/seatunnel-spark-connector-v2-example

The example entry point is:

  • org.apache.seatunnel.example.spark.v2.SeaTunnelApiExample

Next Steps​