Figure out the task scheduling process of Apache SeaTunnel Zeta Engine once and for all

7 min readApr 12, 2023

Written by Liu Guangdong(liugddx)

Introduction to Zeta

Zeta Engine is a data synchronization engine specially designed and developed for data synchronization scenarios. It is faster, more stable, more resource-efficient, and easier to use. In the comparison of various open-source synchronization engines around the world, Zeta’s performance is far ahead. It has undergone several R&D versions, and the beta version was released in October 2022. After discussion, it was decided to name it Zeta (the fastest star in the universe, and the community believes that this fully reflects the characteristics of the engine), whose properties include:

Simple and easy to use, the new engine minimizes the dependence on third-party services and can realize cluster management, snapshot storage, and cluster HA functions without relying on big data components such as Zookeeper and HDFS. This is very useful for users who do not have a big data platform or are unwilling to rely on a big data platform for data synchronization.
More resource-saving, at the CPU level, Zeta Engine internally uses Dynamic Thread Sharing (dynamic thread sharing) technology, in the real-time synchronization scenario, if the number of tables is large but the amount of data in each table is small, Zeta Engine will Synchronous tasks run in shared threads, which can reduce unnecessary thread creation and save system resources. On the read and data write side, the Zeta Engine is designed to minimize the number of JDBC connections. In the CDC scenario, Zeta Engine will try to reuse log reading and parsing resources as much as possible.
More stable. Zeta Engine uses Pipeline as the minimum granularity of Checkpoint and fault tolerance for data synchronization tasks. The failure of a task will only affect the tasks that have upstream and downstream relationships with it. Try to avoid task failures that cause the entire Job to fail or rollback. At the same time, for scenarios where the source data has a storage time limit, Zeta Engine supports enabling data cache to automatically cache the data read from the source, and then the downstream tasks read the cached data and write it to the target. In this scenario, even if the target end fails and data cannot be written, it will not affect the normal reading of the source end, preventing the source end data from being deleted due to expiration.
Faster, Zeta Engine’s execution plan optimizer will optimize the execution plan to reduce the possible network transmission of data, thereby reducing the loss of overall synchronization performance caused by data serialization and deserialization, and completing faster Data synchronization operations. Of course, it also supports speed limiting, so that sync jobs can be performed at a reasonable speed.
Full scene data synchronization support, Apache SeaTunnel’s goal is to support full synchronization and incremental synchronization under offline batch synchronization and support real-time synchronization and CDC.

Naming origin

Zeta Ophiuchi (ζ Oph) is a hot blue subgiant star 366 light-years away in the constellation Ophiuchus. With an apparent magnitude of 2.57, it is the third-brightest star in the constellation, after Rasalhague and Sabik. Zeta Ophiuchi is the closest O-type star to the Sun, and one of the bright stars that outlines the polygonal constellation of the Serpent of Fortune. It is an unusually fast spinner, which is predicted to spin at speeds of up to 400 km/s. The Apache SeaTunnel engine (zeta) is the dream star, with fast as the most obvious feature.

Overall structure

Overview

Service initialization

TaskExecutionService

TaskExecutionService is a task execution service that will run one instance per node. It receives TaskGroup from JobMaster and runs Task in it, and maintains TaskID->TaskContext, the specific operations on Task are encapsulated in TaskContext. The Task holds the OperationService inside, which means that the Task can remotely call other Tasks or JobMaster through the OperationService to communicate.

CoordinatorService

CoordinatorService is a service that acts as a coordinator. It is mainly responsible for processing commands submitted by clients and restoring tasks after switching masters. When the client submits a task, it will find the master node and submit the task to the CoordinatorService.It will cache the task information and wait for the task to complete. After the task is completed, the task is archived.

SlotService

SlotService is a slot management service used to manage the available Slot resources of the cluster. It runs on all nodes and periodically reports resource information to the master.

Task submission process

Step 1: Convert jobconf to LogicDag

We define the job process through the job configuration file, so the first thing Apache SeaTunnelClient needs to do is to parse the job configuration file and generate an action list. Action is similar to the operator in Flink, which is the encapsulation of Apache SeaTunnel API. An action contains an instance of SeaTunnelSource or SeaTunnelTransform or SeaTunnelSink . Every action needs to know its upstream.

public interface Action extends Serializable {
@NonNull String getName();
void setName(@NonNull String name);
@NonNull List<Action> getUpstream();
void addUpstream(@NonNull Action action);
int getParallelism();
void setParallelism(int parallelism);
long getId();
Set<URL> getJarUrls();
}

Currently, Apache SeaTunnel supports three types of actions: SourceAction, SinkAction, and TransformAction. If there is only one Source and one Sink, and one more Transform, we only need to simply parse it as follows:

If there are multiple sources or multiple transforms or multiple sinks, we will rely on source_table_name and result_table_name to build the action pipeline. So in this case result_table_name is required for source action and both result_table_name and source_table_name are required for transform action. Finally, source_table_name is required for sink actions.

Step 2: Convert LogicPlan to PhysicalPlan

The Apache SeaTunnel engine will receive the logical plan sent by the client, and the engine needs to convert it into a physical plan that can be directly executed. Therefore, it is necessary to process the logical execution plan and generate a physical plan through conversion. The specific process is as follows:

Logical plan picture

Once receiving the logical plan, we need to remove redundant Actions and verify the Schema (Transform2 and Transform 5 should be the same)

Execution Plan

When converting to an execution plan:

(1) Transforms need to be merged, and the basis for merging is whether the data will be split after Transform (if there is no shuffle, the transform will be merged). (2) Convert the shuffle action into a queue. (3) Split multiple pipelines.

Physical plan

We split the Pipeline into separate executable tasks according to the degree of parallelism, and at the same time need to add SourceSplitEnumerator and SinkAggregatedCommitter tasks, which can send tasks to executionService. Then the task will run normally.

Step 3: Schedule the taskGroup to the specified node and wait for it to run

On the master node, the physical plan is split into pipelines, and the pipeline is split into taskGroups further, which are scheduled and executed on different nodes.

physicalPlan: The job submitted by the user is parsed into a runnable execution plan.
pipeline: The tasks in the pipeline only have the upstream and downstream operators of the pipeline, and there are no associated operators for different pipelines.
taskGroup: Each execution plan vertex will create a taskGroup, a taskGroup contains one or more tasks, and each taskGroup requires a unit of computing resources. taskGroup is the smallest unit of task allocation and execution. as follows:

TaskGroup#1: {
    task#1: {source->transformation#1->transformation#2->queue1},
    task#2: {queue1->sink}
}

📌📌Welcome to fill out this survey to give your feedback on your user experience or just your ideas about Apache SeaTunnel:)

About Apache SeaTunnel

Apache SeaTunnel (formerly Waterdrop) is an easy-to-use, ultra-high-performance distributed data integration platform that supports real-time synchronization of massive amounts of data and can synchronize hundreds of billions of data per day in a stable and efficient manner.

Why do we need Apache SeaTunnel?

Apache SeaTunnel does everything it can to solve the problems you may encounter in synchronizing massive amounts of data.

Data loss and duplication
Task buildup and latency
Low throughput
Long application-to-production cycle time
Lack of application status monitoring

Apache SeaTunnel Usage Scenarios

Massive data synchronization
Massive data integration
ETL of large volumes of data
Massive data aggregation
Multi-source data processing

Features of Apache SeaTunnel

Rich components
High scalability
Easy to use
Mature and stable

How to get started with Apache SeaTunnel quickly?

Want to experience Apache SeaTunnel quickly? SeaTunnel 2.1.0 takes 10 seconds to get you up and running.

https://seatunnel.apache.org/docs/2.1.0/developement/setup

How can I contribute?

We invite all partners who are interested in making local open-source global to join the Apache SeaTunnel contributors family and foster open-source together!

Submit an issue:

https://github.com/apache/incubator-seatunnel/issues

Contribute code to:

https://github.com/apache/incubator-seatunnel/pulls

Subscribe to the community development mailing list :

dev-subscribe@seatunnel.apache.org

Development Mailing List :

dev@seatunnel.apache.org

Join Slack:

https://join.slack.com/t/apacheseatunnel/shared_invite/zt-1kcxzyrxz-lKcF3BAyzHEmpcc4OSaCjQ

Follow Twitter:

https://twitter.com/ASFSeaTunnel