User Contribution: Apache SeaTunnel Web Deployment Guide

Thanks to community user “Compass” for his contribution, which has injected new vitality into the joint construction of users and the community. I hope his experience can be passed on!

Preparations

Java Environment

Requires Java 8 or Java 11, and set JAVA_HOME. If JAVA_HOME is invalid, it needs to be set to a valid state.

echo ${JAVA_HOME}

Create Installation Directories

sudo mkdir -p /opt/seatunnel/backend
sudo mkdir -p /opt/seatunnel/web

Download Software

Download Apache-SeaTunnel-2.3.3-bin.tar.gz from https://seatunnel.apache.org/download/.

And also apache-seatunnel-web-1.0.0-bin.tar.gz.

Place them in the prepared directories.

Install SeaTunnel

Unzip the Software

sudo tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz -C /opt/seatunnel/backend
sudo tar -zxvf apache-seatunnel-web-1.0.0-bin.tar.gz -C /opt/seatunnel/web/

Configure Environment Variables

Configure environment variables in /etc/profile.d/seatunnel.sh, then execute source /etc/profile.

Download JAR Packages

Before downloading JAR packages for Apache SeaTunnel, check its download logic in apache-seatunnel-2.3.3, viewing vim bin/install-plugin.sh.

Main logic includes:

1. Using mvnw for downloading.
2. Downloading seatunnel-hadoop3–3.1.4-uber to the lib folder.
3. Downloading connectors specified in config/plugin_config to connectors/seatunnel.

Edit config/plugin_config to include or exclude connectors as needed.

View all supported connectors in vim connectors/plugin-mapping.properties.

Create Required Directories

Execute mkdir -p flink flink-sql spark seatunnel to create multiple folders.

Download Automatically

Execute sh bin/install-plugin.sh for automatic download. This is not recommended due to slow download speeds from the official website. Alternatively, modify the code for manual acceleration.

Manual Download

Modify the code for faster downloading through Aliyun’s Maven source, then copy the relevant JAR files to the respective directories.

For detailed instructions, refer to this article.

Note: Copy JAR files to both the lib folder and connectors/seatunnel.

Start Running

Test

Navigate to the main folder /opt/seatunnel/backend/apache-seatunnel-2.3.3 and execute:

sudo ./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

Logs displaying the following information indicate a successful test run.

Start the Service

Execute the command nohup sh bin/seatunnel-cluster.sh 2>&1 &. To check the log, use tail -f logs/seatunnel-engine-server.log. If similar information is printed, the service has started successfully.

Note that Apache SeaTunnel’s Server must be started for the Web service to function properly.

Install SeaTunnel-Web

Preliminary Steps

If SeaTunnel’s Web and Server are on the same machine, you can skip this step. Otherwise, deploy the SeaTunnel client on the machine where the Web service will be hosted.

For more details, visit this article: Configuring SeaTunnel Engine Client

Unzip the Software

Execute sudo tar -zxvf apache-seatunnel-web-1.0.0-bin.tar.gz -C /opt/seatunnel/web/.

Initialize the Database

Modify Configuration

Change the script/seatunnel_server_env.sh file to match your database information.

Consider prefixing variables like HOSTNAME, PORT with STWEB_ to avoid conflicts with other system variables.

Prefix variables in script/init_sql.sh with STWEB_.

Execute Database Initialization Command

Run sh init_sql.sh. If there are no errors, the initialization is successful.

Configure Backend Service

Modify Port and Data Source

Edit conf/application.yml to adjust the port number and data source information.

Configure Client Information

Copy the hazelcast-client.yaml file from the SeaTunnel backend to the web configuration directory.

Configure Supported Plugins

Copy the plugin-mapping.properties file from the SeaTunnel backend to the web configuration directory.

Configure JAR Packages

Configuring JAR packages correctly is crucial to avoid various issues, such as:

1. No data source available for creation.
2. No Source or Sink available for selection.
3. Configured tasks not executing.

Configure Metadata MySQL JAR Package

Copy mysql-connector-java-8.0.15.jar to /opt/seatunnel/web/apache-seatunnel-web-1.0.0-bin/libs.

Configure Data Source JAR Packages

Copy relevant JAR packages to /opt/seatunnel/web/apache-seatunnel-web-1.0.0-bin/libs.

Important Notes

Data source JAR packages should also be copied to the backend lib folder /opt/seatunnel/backend/apache-seatunnel-2.3.3/lib.

For downloading data source JAR packages, refer to the relevant section in this article.

Start the Service

Execute sudo sh bin/seatunnel-backend-daemon.sh start. Visit ip:8081/ui to access the web interface, using the port configured in conf/application.yml.

The default username and password are admin/admin.

After logging in, create a data source and set up tasks as required.

View Data Source

Configure tasks according to your needs.

Configure Tasks

Select the role table from the official SeaTunnel as the Source.

For the Sink, create a similar table in another database, seatunnel_bak.

Run Instance

After successful task execution, verify the data in the destination database.

select *, 'org' flag from seatunnel.`role`
union all
select *, 'dest' flag from seatunnel_bak.`role`

This concludes the setup and test of Apache SeaTunnel and its Web interface. If there are any errors, feel free to correct them. The author is a user in the Apache SeaTunnel community and welcomes everyone to contribute to the community!