November Report on Apache SeaTunnel Community Development

Hi, Apache SeaTunnel community, the monthly report for November has been updated! This report will document the significant updates in the SeaTunnel community every month, so stay tuned!

Monthly Merge Star

We would like to express our gratitude to the following contributors for their outstanding contributions to Apache SeaTunnel in November (listed in no particular order):

“@CosmosNi,@BilwaST,@liunaijie,@corgy-w,@Carl-Zhou-CN,@davidzollo,@hawk9821,@lm-ylj,@sunxiaojian,@zhdech,@dailai,@XuyiK,@boandai,@YOMO-Lee,@shashwatsai,@eyys,@wengys,@hailin0,@jw-itq,@Hisoka-X,@zhangshenghang,@XenosK,@a110q,@arshadmohammad”

apache/seatunnel

fix

• [Fix][Doc]fix backup-count config @corgy-w
• [Fix][Doc]Fixed unreachable links @corgy-w
• [hotfix][gbase]Fix gbase driver link @zhangshenghang
• [fix]fix get job metrics get NPE @liunaijie
• [Hotfix][Transforms-v2]DynamicCompile Plugin compatibility fix @hawk9821
• [Fix][CI]time-consuming module optimization @hawk9821
• [Hotfix][Docs]Correct errors in the FAQ doc @davidzollo
• [hotfix][connector-v2][hive]Fix hive3 e2e dependencies @zhangshenghang
• [Bug][connectors-v2]fix mongodb bson convert exception @hawk9821
• [Fix][API]Fix column length can not be long @Hisoka-X
• [Fix][CI]Fix schedule test be canceled @Hisoka-X
• [Hotfix][E2E]Fix OracleCDCIT compile error @hailin0
• [Fix][Connecotr-V2]Fix clickhouse sink does not support composite primary key @eyys
• [Fix][Doc]Fix key description error in redis zh doc @lm-ylj
• [Hotfix][Zeta]Fix the dependency conflict between the guava in hadoop-aws and hive-exec @dailai
• [Fix][Zeta]Remove duplicate LIFECYCLE_WRITER_CLOSE events @Hisoka-X
• [Fix][Zeta]Fix job client print error msg even job finished @Hisoka-X
• [Fix][Doc]Fix klaviyo doc 404 link @YOMO-Lee
• [Fix][Zeta]Fix apply resource again for another pipeline when restore one pipeline @zhangshenghang
• [Bug][Connector-v2]MongoDB CDC Set SeatunnelRow’s tableId @hawk9821
• [Fix][Kafka]Fix in kafka streaming mode can not read incremental data @Carl-Zhou-CN

improve

• [Improve][Connector-V2]Save data savemode operation when schema savemode created new table @zhangshenghang
• [Improve]Remove unused code @hailin0
• [Improve][Connector-V2]Add table info when multi table sink error @Hisoka-X
• [improve][connector-v2][kafka-connector]Add e2e code for Kafka Kerberos authentication @zhangshenghang
• [Improve][Jdbc]Improve ddl write validate @hailin0
• [Improve][Connector-V2]Add pre-check for table enable cdc @Hisoka-X
• [Improve][Connector-V2]Reduce the request times of iceberg load table @Hisoka-X
• [Improve][Kafka]Support custom topic for debezium compatible format @hailin0
• [Improve][doc]Add doc name check @zhangshenghang
• [Improve][dist]add shade check rule @hawk9821
• [Improve][Jdbc]Refactor ddl change @hailin0
• [Feature][Jdbc]Add Jdbc default dialect for all jdbc series database without dialect @sunxiaojian
• [Improve][Excel]Support read blank string & auto type-cast @hailin0
• [Feature][File]Support config null format for text file read @hailin0
• [Feature][E2E][Kerberos]Support for Kerberos in e2e @zhangshenghang
• [Improve][API]Unified tables_configs and table_list @hawk9821
• [Improve]Allow nullable for SeaTunnelRowType convert to Column @hailin0
• [Feature][Iceberg]Support custom delete sql for sink savemode @hailin0
• [Improve][Doc]Add the way of configure jvm options in client @Hisoka-X
• [Improve][Connector-V2]Fix SqlServer cdc memory leak @Hisoka-X
• [Improve][E2E]Improve Druid E2E Case @hawk9821
• [Improve][Zeta]Fixed issues related to log deletion during ci @corgy-w
• [Feature][Core]Rename result_table_name/source_table_name to plugin_input/plugin_output @hailin0
• [Improve][Transform]Optimize transform structure @corgy-w
• [Improve][Zeta]Remove default jvm memory option @Hisoka-X
• [Improve][POM]Bump the version of maven-shade-plugin to 3.4.1 @dailai
• [Improve]Improve AlterTableSchemaEventHandler @hailin0
• [Feature][Connector-v2]Support S3 filesystem of paimon connector @dailai
• [Feature][E2E]Add E2E test case time analysis @hawk9821
• [Feature][Connectors]LocalFile Support reading gz @zhdech
• [Improve][Connector-V2]Improve schema evolution on column insert after for mysql-jdbc @jw-itq
• [Feature][Transform]Sql transform support array and split function @CosmosNi
• [Feature][E2E]Add hive3 e2e test case @zhangshenghang
• [Feature][RestAPI]Support submit job with seatunnel style hocon format config @CosmosNi
• [Improve][CI]Optimizing CI time cost @Hisoka-X
• [Improve][Connector-V2]RocketMQ Sink add message tag config @wengys
• [Feature] [Connector-Redis]Redis connector support delete data @lm-ylj
• [Feature][Zeta]Add jar path precheck when job submit on master @zhangshenghang
• [Feature][transform]transform support explode @CosmosNi
• [Feature][Connector-v2]Support schema evolution for Oracle connector @dailai
• [Feature][Transform-v2]Add metadata transform @hawk9821
• [Feature][Connector-V2]Jdbc chunk split add snapshotSplitColumn config #7794 @XenosK
• [Feature][Zeta]Support delete logs regularly @corgy-w
• [Improve][Transform]Auto mark column length with jsonpath transform @hailin0
• [Feature][Transform-V2]Support transform with multi-table @Hisoka-X
• [Feature][Core]Support cdc task ddl restore for zeta @hailin0

others

• [doc][connector][hive]Add hive chinese doc @zhangshenghang
• [Doc][Format]Add protobuf doc @zhangshenghang

apache/seatunnel-website

improve

• [Improve]Change announcement bar to add star @Hisoka-X

others

• Fix translate error in about.md @XuyiK
• Update jdbc-mysql.md @a110q

apache/seatunnel-web

fix

• [Bug] [Seatunnel-web]No configuration setting found for key ‘where_c…ondition’ @arshadmohammad
• [Addendum][Bug] [Seatunnel-web]Error when conditional column is not used in SE @BilwaST
• [Bug] [Seatunnel-web]Error when conditional column is not used in SE… @BilwaST
• [Bug][Seatunnel-web]Fix fetch table fields for Oracle Jdbc Shashwat Saini

About Apache SeaTunnel

Apache SeaTunnel is an easy-to-use, ultra-high-performance distributed data integration platform that supports real-time synchronization of massive amounts of data and can synchronize hundreds of billions of data per day stably and efficiently.

Welcome to fill out this form to be a speaker of Apache SeaTunnel: https://forms.gle/vtpQS6ZuxqXMt6DT6 :)

Why do we need Apache SeaTunnel?

Apache SeaTunnel does everything it can to solve the problems you may encounter in synchronizing massive amounts of data.

• Data loss and duplication
• Task buildup and latency
• Low throughput
• Long application-to-production cycle time
• Lack of application status monitoring

Apache SeaTunnel Usage Scenarios

• Massive data synchronization
• Massive data integration
• ETL of large volumes of data
• Massive data aggregation
• Multi-source data processing

Features of Apache SeaTunnel

• Rich components
• High scalability
• Easy to use
• Mature and stable

How to get started with Apache SeaTunnel quickly?

Want to experience Apache SeaTunnel quickly? SeaTunnel 2.1.0 takes 10 seconds to get you up and running.

https://seatunnel.apache.org/docs/2.1.0/developement/setup

How can I contribute?

We invite all partners who are interested in making local open-source global to join the Apache SeaTunnel contributors family and foster open-source together!

Submit an issue:

https://github.com/apache/seatunnel/issues

Contribute code to:

https://github.com/apache/seatunnel/pulls

Subscribe to the community development mailing list :

dev-subscribe@seatunnel.apache.org

Development Mailing List :

dev@seatunnel.apache.org

Join Slack:

https://join.slack.com/t/apacheseatunnel/shared_invite/zt-1kcxzyrxz-lKcF3BAyzHEmpcc4OSaCjQ

Follow Twitter:

https://twitter.com/ASFSeaTunnel

Join us now!❤️❤️