Apache SeaTunnel 2.3.9 released: multiple new features and optimizations to comprehensively improve data integration capabilities

The Apache SeaTunnel community has officially released the latest version, 2.3.9. This update introduces several exciting new features, including Helm cluster deployment, multi-table support in Transform, Zeta new APIs, table schema conversion, task submission queue, database and table partition merging, and column-to-row transformations!

As an open-source, distributed data integration platform, this version brings comprehensive support for developers and enterprise users through new features, performance optimizations, and bug fixes.

📥 Download SeaTunnel 2.3.9: Download Link

📕 Release Notes: Release Notes Link

👇👇 For a detailed explanation of the key features, you can watch the video below 👇👇

Key Highlights of Version 2.3.9

Enhanced Data Processing and Monitoring Capabilities

Task and Data Monitoring:

• The new Zeta API allows task submissions via REST API and retrieval of task logs. Task log eviction is also introduced, along with a real-time visualization feature via the Zeta Manager UI.
• Zeta now supports task queuing, REST API submissions for Hocon job configurations, and pausing/resuming during DDL operations.

Schema Evolution:

• Several connectors, such as Oracle, StarRocks, and Paimon, now support Schema Evolution, lowering the integration barrier for complex data sources.

Feature Enhancements:

• Transform now supports multi-table operations, dynamic type handling, and database/table partition merging, providing more flexibility for complex data transformation scenarios.
• Support for renaming tables and columns in Transform.
• Support for the RowKindExtractor transformation operation type.
• Support for column-to-row transformations in SQL.

Multi-source Adaptability:

• Expanded support for new data sources, including Milvus dynamic schema, multi-table reads, and Redis data deletion operations.

Significant Improvements in Task Execution Efficiency

Resource Scheduling and Thread Pool Management:

• Optimized the Zeta engine’s CoordinatorService thread pool configuration, reducing potential memory overflow issues.

Task Execution Mode:

• Spark, Flink, and Zeta now support multi-table Transform operations, reducing task dependencies and improving concurrency performance.
• Oracle-CDC now supports DDL reading.
• Debezium has been enhanced to support sending data to message queues for buffering and handling special data types and time types. It also supports multi-table/table-to-one-topic mapping.

Data Stream Transfer:

• Improved write modes for connectors like ClickHouse and JDBC, enhancing stability in high-volume data processing scenarios.

New DDL Sink Support

Connector PR Author

StarRocks #8082 jw-itq

Paimon #8211 dailai

Oracle #7908 dailai

Doris #8250 deng-jeffer

PostgreSQL #8276 hawk9821

Elasticsearch #8412 zhangshenghang

Bug Fixes and Reliability Enhancements

• Helm deployment now supports quick cluster setup.
• Fixed several connector-related issues, such as exception handling and feature failures in MongoDB, Kafka, Hive, and other connectors.
• Improved Avro format support for null values and resolved formula and numeric parsing issues in Excel data reading.
• Optimized Docker deployment and CI processes to ensure consistency across different environments.

Acknowledgments

Special thanks to Tyrantlucifer for guiding and assisting with this release, and to the following community contributors for their joint efforts in making this release successful:

The release of Apache SeaTunnel 2.3.9 marks another milestone in its development as a leading open-source data integration tool. Whether for real-time data synchronization, batch-stream integrated task processing, or managing complex schemas and performance optimizations, SeaTunnel now offers even more powerful features for users.

As a rapidly growing Apache top-level project, SeaTunnel continues to follow the spirit of community-driven development and open-source collaboration. We welcome developers and enterprise users to contribute to the community and help refine and promote this robust data integration platform.