Apache SeaTunnel: Discover the next generation of high-performance distributed data integration tools
Good afternoon everyone, my name is Liu Guangdong, and I am a Committer from the Apache SeaTunnel community. Today, I am excited to share insights into the next-generation, high-performance, distributed tool for massive data integration: Apache SeaTunnel. This presentation is aimed at providing a developer’s perspective on Apache SeaTunnel, with a focus on its intricate design elements.
Topic Overview
- Introducing the Apache SeaTunnel tool
- Core architectural designs of Apache SeaTunnel
- Introduction to the custom-built Zeta engine in Apache SeaTunnel
- Features of Apache SeaTunnel Web
- Community’s recent planning and roadmap
SeaTunnel Introduction
Let’s start by discussing the design and objectives of Apache SeaTunnel. As a next-generation data integration platform, SeaTunnel offers a one-stop solution with several key features:
- Ease of Use: Utilizing SeaTunnel is straightforward, requiring only a config file and a shell script to launch a task.
- Monitorable and Quantifiable: During task execution, data read/write and latency metrics are collected and summarized.
- Rich Data Ecosystem: Over 100 data sources, including databases, message queues, cloud storage, and data lake components, are seamlessly integrated.
- Full-Scenario Support: Offline, real-time, full and incremental, and CDC synchronization, along with features like DDL changes and dynamic table addition, are all supported.
- Low Resource Consumption: Optimizations for memory and CPU usage, multiple table synchronization, and shared JDBC resources.
The Evolution of SeaTunnel
Before joining the Apache Incubator and being named SeaTunnel, our project was known as Waterdrop, open-sourced in 2017. It entered the Apache Incubator in December 2021 and released its first Apache version in March 2022. Significant milestones include supporting cross-engine designs and the integration of a large number of connectors by October 2022.
The Zeta Engine, specifically designed for data synchronization, was conceptualized to optimize resource usage. By December 2022, SeaTunnel incorporated over 100 connectors, including CDC connectors. Presently, Zeta Engine supports CDC, multi-table synchronization, and is developing features like automatic table creation.
Overall Architecture
The architecture of SeaTunnel can be divided into four components: Source and Sink data sources on the left and right, the Web interface on top, and the core engine in the center. This core part includes three key elements: Source, Transform, and Sink.
Decoupled Engine Design
One of the significant contributions to the community has been the decoupling of the engine, simplifying the integration of connectors.
This design alleviates the complexity of adapting connectors for different engines like Flink or Spark and supports various versions, streaming and batch tasks, and JDBC resource sharing.
Core Design and Architecture of SeaTunnel
The workflow starts with reading the configuration file into the project, followed by the catalog parsing the data into Tables, Schemas, and Options. Using SPI, SeaTunnel launches connectors to integrate this data, which are then translated into engine-specific connectors. The Source-Transform-Sink process executes the task.
CDC Design
Despite the existence of tools like FlinkCDC, Debezium, and Canal, SeaTunnel developed its CDC component to ensure smooth migration of CDC tasks across engines, support parallel snapshotting of historical data without locks, dynamic table addition, multi-database reading, schema evolution, and checkpoint processes.
The enumerator generates multiple SnapshotSplits of a table and assigns them to readers.
When the SnapshotSplit read completes, the reader reports the split’s high-water mark to the enumerator. When all SnapshotSplits report the high water mark, the enumerator begins the increment phase.
Zeta Engine
The Zeta Engine is a data synchronization engine built for speed, stability, and efficiency. It features cluster autonomy, resource efficiency, stability with pipeline-based fault tolerance, speed through execution plan optimization, and support for various data synchronization scenarios.
The first core component is the core component of Zeta, which is the CoordinatorService. If you know Zeta, it is actually a CS architecture, that is, the client. Then configure the data in the Config file, and then after processing it, throw it to us In the Service, the Service is based on the resources on our Worker nodes, and then passes these tasks through the DAG, and then splits them into some operators, and then throws these operators to different Workers, and then schedules them. implement.
The first core component, CoordinatorService, is the Master service of the cluster. It provides the generation process of each job from LogicalDag to ExecutionDag, and then to PhysicalDag, and finally creates the job’s JobMaster for job scheduling, execution and status monitoring.
It contains the following four parts.
- JobMaster is responsible for the generation process of a single job from LogicalDag to ExecutionDag and then to PhysicalDag, and is scheduled and run by PipelineBaseScheduler.
- CheckpointCoordinator is responsible for the Checkpoint process control of the job.
- ResourceManager is responsible for the application and management of job resources. It currently supports Standalone mode and will support On Yarn and On K8s in the future.
- Metrics Service is responsible for the statistics and summary of job monitoring information.
The next core component is TaskExecutionService, which is the cluster’s Worker service and provides the real runtime environment for each Task in the job. TaskExecutionService uses Dynamic Thread Sharing technology to reduce CPU usage.
What does that mean? That is, we don’t need to occupy an exclusive thread when each task comes, and then let it finish running.
If some tasks are for large tables or small tables, they can be completed through this thread. If there are large tables, this resource may be monopolized, that is, we may run to the end, and all the tasks will be monopolized by the larger tables, and then the small tables may not get the resources to run.
So through this technology, we can let those tasks that have relatively large resources and take a long time to exclusively occupy a thread and let them run in another place. As for other small tables, all resources can be shared in this thread. Ensure that we can get the CPU running time for each task.
The third component is a component of resource application. SlotService runs on each node of the cluster and is mainly responsible for the division, application and recycling of resources on the node.
Its main work contents are as follows:
The master node and worker threads are started, the worker thread registers information with the resource manager, and the status is monitored by regular heartbeats.
JobMaster requests resources from ResourceManager.
If resources are sufficient, the corresponding worker information will be provided to the job supervisor. If not, the jobmaster will not have enough resources.
After JobMaster receives feedback, it will determine whether to start scheduling tasks or trigger exceptions, and then set the thresholds we use. For operation and maintenance personnel, we also provide some better monitoring platforms.
Web Feature Release
The newly released SeaTunnel Web interface simplifies the creation of data synchronization tasks. As it is still in its early stages, more functionalities are expected to be added soon.
Recent Planning
If you’re interested in joining the Apache open-source community, SeaTunnel offers many opportunities to contribute, especially with features like multi-table support in Sink tasks and multi-table synchronization.
My Open Source Journey
Finally, I’d like to share my journey into open source. I was initially drawn to SeaTunnel after researching data synchronization products and finding a vibrant community actively discussing and contributing to the project. Starting with simple documentation changes, I gradually became more involved. I encourage everyone to start with “good first issue” tasks and contribute to this thriving community.
Thank you all for listening to my presentation!
