Apache SeaTunnel 2.3.8 Officially Released!

The Apache SeaTunnel 2.3.8 version has officially been released! With this update, users can now enjoy the much-anticipated Docker image support, the Job-level logging feature, and other enhancements and optimizations. This article will detail the key updates in Apache SeaTunnel 2.3.8. We encourage more developers and users to join our open-source community.

• Download SeaTunnel 2.3.8: https://seatunnel.apache.org/download/
• Release Notes: https://github.com/apache/seatunnel/releases/tag/2.3.8

Key Updates

Job-Level Logging

In this update, we’ve optimized the logging functionality. In previous versions, logs from multiple tasks were written into a single file. When running multiple tasks simultaneously, the logs were intertwined, making troubleshooting difficult.

The new update supports two configurations for more efficient log querying.

The first option is adding a JobId to each log entry, allowing you to filter logs by JobId.

The second option is to split log files by JobId. Modifying the log configuration file allows you to create separate log files for each JobId.

New Docker Image

This update includes official Docker image support. The image comes with all connectors pre-installed, enabling users to run SeaTunnel more conveniently without downloading additional packages. This reduces deployment complexity and streamlines the process for users deploying SeaTunnel with Kubernetes.‘

For users requiring customization or additional development, the new version also offers a one-click command to build customized images:

Multi-Table Support for Flink/Spark Engines

In previous versions, the multi-table read and write feature was only supported on the Zeta engine. With this update, Spark and Flink engines now also support multi-table read and write operations.

Prometheus Integration for Cluster Monitoring

Previously, users needed to access API endpoints to retrieve cluster and task metrics. Now, users can export these metrics to Prometheus. Prometheus will regularly pull the status of SeaTunnel clusters and tasks, providing a visual interface to monitor the cluster and identify issues quickly.

Typesense Connector Support Added

Support for the Typesense connector has been added in this release.

Improvements and Optimizations

Embedding Transform Added

With the Embedding transform, SeaTunnel can embed machine learning models into the data transformation process. This allows raw fields to be converted into vector values and stored in the corresponding machine-learning databases. Currently, SeaTunnel supports models from vendors like Doubot, Qianfan, and OpenAI, with more model providers to be added in the future.

Kafka Support for Protobuf Read/Write

The Kafka connector now supports reading and writing Protobuf data formats. A Protobuf data type definition has been added under the Kafka connector for data reading and writing.

File Support for Reading Compressed Archives

Support for reading compressed file formats has been added, eliminating the need for manual extraction steps.

Finer-grained resource Loading Isolation

ClassLoader isolation has been improved from task group level to task level, avoiding dependency conflicts when using the same ClassLoader for Source/Sink.

Other optimizations include:

• Paimon stream read and dynamic bucket write support.
• SQL nested query support for querying Map structure fields.
• Iceberg-related optimizations.
• Kerberos authentication support.
• SaveMode adds IGNORE type.
• Optimization for reading across Redis versions.
• Adaptation for MySQL 8.1/8.2/8.3 versions.
• Support for TiDB CDC reads.
• Removal of system table restrictions in JDBC connectors.
• Event notification support for all connectors.
• Multi-table read support for the ES connector.
• Multi-table write support for the HBase connector.

Key Bug Fixes

• Fixed Hazelcast networking issues with non-TCP connections.
• Resolved resource isolation issues.
• Fixed Paimon dynamic bucket table and decimal precision loss issues.
• Fixed resource leakage at the end of Iceberg tasks.

For more detailed updates, please refer to the Release Notes: https://github.com/apache/seatunnel/releases/tag/2.3.8

Thanks to Our Contributors

Special thanks to @liunaijie for guidance and assistance with this release. Additionally, we’d like to thank the following community members for their efforts in making this release a success:

hailin0, hawk9821, cl0924, sunxiaojian, dailai, corgy-w, Hisoka-X, liunaijie, chl-wxp, zhangshenghang, ISADBA, loustler, chenqianwen, FuYouJ, xxsc0529, EricJoy2048, ZhangWeike2000, jw-itq, kevinjmh, Carl-Zhou-CN, FlechazoW, PeppaPage, liugddx, Cheun99, happyboy1024, CosmosNi, Anush008, BruceWong96, zqr10159, cloud456, Gxinge, xxsc0529, luzongzhu, jiamin13579, Zuhdan, yujian225.

About Apache SeaTunnel

Apache SeaTunnel is an easy-to-use, ultra-high-performance distributed data integration platform that supports real-time synchronization of massive amounts of data and can synchronize hundreds of billions of data per day stably and efficiently.

Welcome to fill out this form to be a speaker of Apache SeaTunnel: https://forms.gle/vtpQS6ZuxqXMt6DT6 :)

Why do we need Apache SeaTunnel?

Apache SeaTunnel does everything it can to solve the problems you may encounter in synchronizing massive amounts of data.

• Data loss and duplication
• Task buildup and latency
• Low throughput
• Long application-to-production cycle time
• Lack of application status monitoring

Apache SeaTunnel Usage Scenarios

• Massive data synchronization
• Massive data integration
• ETL of large volumes of data
• Massive data aggregation
• Multi-source data processing

Features of Apache SeaTunnel

• Rich components
• High scalability
• Easy to use
• Mature and stable

How to get started with Apache SeaTunnel quickly?

Want to experience Apache SeaTunnel quickly? SeaTunnel 2.1.0 takes 10 seconds to get you up and running.

https://seatunnel.apache.org/docs/2.1.0/developement/setup

How can I contribute?

We invite all partners who are interested in making local open-source global to join the Apache SeaTunnel contributors family and foster open-source together!

Submit an issue:

https://github.com/apache/seatunnel/issues

Contribute code to:

https://github.com/apache/seatunnel/pulls

Subscribe to the community development mailing list :

dev-subscribe@seatunnel.apache.org

Development Mailing List :

dev@seatunnel.apache.org

Join Slack:

https://join.slack.com/t/apacheseatunnel/shared_invite/zt-1kcxzyrxz-lKcF3BAyzHEmpcc4OSaCjQ

Follow Twitter:

https://twitter.com/ASFSeaTunnel

Join us now!❤️❤️