Set Up with Locally
Prepare
Before you getting start the local run, you need to make sure you already have installed the following software which SeaTunnel required:
- Java (Java 8 or 11, other versions greater than Java 8 can theoretically work as well) installed and
JAVA_HOME
set. - Download the engine, you can choose and download one of them from below as your favour, you could see more information about why we need engine in SeaTunnel
- Spark: Please download Spark first(required version >= 2 and version < 3.x ). For more information you could see Getting Started: standalone
- Flink: Please download Flink first(required version >= 1.12.0 and version < 1.14.x ). For more information you could see Getting Started: standalone
Installation
Enter the seatunnel download page and download the latest version of distribute
package seatunnel-<version>-bin.tar.gz
Or you can download it by terminal
export version="2.1.0"
wget "https://archive.apache.org/dist/incubator/seatunnel/${version}/apache-seatunnel-incubating-${version}-bin.tar.gz"
tar -xzvf "apache-seatunnel-incubating-${version}-bin.tar.gz"
Install connectors plugin
Since 2.3.0-beta, the binary package does not provide connector dependencies by default, so when using it for the first time, we need to execute the following command to install the connector: (Of course, you can also manually download the connector from [Apache Maven Repository](https://repo.maven.apache.org/maven2/org/apache/seatunnel/ to download, then manually move to the connectors directory).
sh bin/install_plugin.sh
If you need to specify the version of the connector, take 2.3.0-beta as an example, we need to execute
sh bin/install_plugin.sh 2.3.0-beta
Usually we don't need all the connector plugins, so you can specify the plugins you need by configuring config/plugin_config
, for example, you only need the flink-assert
plugin, then you can modify plugin.properties as
--flink-connectors--
seatunnel-connector-flink-assert
--end--
Run SeaTunnel Application
Configure SeaTunnel: Change the setting in config/seatunnel-env.sh
, it is base on the path your engine install at prepare step two.
Change SPARK_HOME
if you using Spark as your engine, or change FLINK_HOME
if you're using Flink.
Run Application with Build-in Configure: We already provide an out-of-box configuration in the directory config
which
you could find when you extract the tarball. You could start the application by the following commands
- Spark
- Flink
cd "apache-seatunnel-incubating-${version}"
./bin/start-seatunnel-spark.sh \
--master local[4] \
--deploy-mode client \
--config ./config/spark.streaming.conf.template
cd "apache-seatunnel-incubating-${version}"
./bin/start-seatunnel-flink.sh \
--config ./config/flink.streaming.conf.template
See The Output: When you run the command, you could see its output in your console or in Flink UI, You can think this is a sign that the command ran successfully or not.
- Spark
- Flink
Hello World, SeaTunnel
Hello World, SeaTunnel
Hello World, SeaTunnel
...
Hello World, SeaTunnel
The content printed in the TaskManager Stdout log of flink WebUI
, is two columned record just like below(your
content maybe different cause we use fake source to create data random):
apache, 15
seatunnel, 30
incubator, 20
...
topLevel, 20
Explore More Build-in Examples
Our local quick start is using one of the build-in example in directory config
, and we provide more than one out-of-box
example you could and feel free to have a try and make your hands dirty. All you have to do is change the started command
option value in running application to the configuration you want to run, we use batch
template in config
as examples:
- Spark
- Flink
cd "apache-seatunnel-incubating-${version}"
./bin/start-seatunnel-spark.sh \
--master local[4] \
--deploy-mode client \
--config ./config/spark.batch.conf.template
cd "apache-seatunnel-incubating-${version}"
./bin/start-seatunnel-flink.sh \
--config ./config/flink.batch.conf.template
What's More
For now, you are already take a quick look about SeaTunnel, you could see connector to find all source and sink SeaTunnel supported. Or see deployment if you want to submit your application in other kind of your engine cluster.