Recap of the Community Development for July and August

Dear community members, the Apache SeaTunnel community monthly update for July and August is here! This document captures the significant updates within the SeaTunnel community during these two months, can you find yourself on the contributor list?

SeaTunnel Monthly Merge Stars

Thanks to the following contributors who made outstanding contributions to Apache SeaTunnel in July and August (in no particular order):

@chaos-cn,@baicie,@litiliu,@panpan2019,@zhangm365,@Thomas-HuWei,@whhe,@wuchunfu,@alextinng,@liugddx,@Hisoka-X,@BilwaST,@XIAOKUOAIyang,@chl-wxp,@loustler,@TyrantLucifer,@hailin0,@xxsc0529,@virvle,@BruceWong96,@kevinjmh,@jackyyyyyssss,@zhangshenghang,@Carl-Zhou-CN,@ZhangWeike2000,@zqr10159,@XenosK,@FuYouJ,@Gxinge,@asapekia,@cl0924,@SachinS10-lab,@Asura7969,@yujian225,@zhouhoo,@corgy-w,@FlechazoW,@MRYOG,@jw-itq,@Anush008,@arshadmohammad,@Vonng,@ISADBA,@dependabot[bot],@tcodehuber,@latch890727,@zhilinli123,@EricJoy2048,@chenqianwen,@luzongzhu,@liunaijie,@dailai,@Wudadada,@cloud456,@OswinWu,@hawk9821,@Xuzhengz,@bingquanzhao

apache/seatunnel

fix

• [Fix][E2E]Change gbase driver version to address certificate expired @hawk9821
• [Hotfix][Connector-V2]Fixed precision problem of decimal data types for paimon sink @dailai
• [Fix][doc]fix dead link @corgy-w
• [Hotfix][Connector-V2]Fix null not inserted in es @corgy-w
• [Fix][Core]Fix NPE when config value is null @hawk9821
• [Fix][Connector-V2]Fix jdbc query sql can not get table path @corgy-w
• [Fix][Zeta]Fix resource isolation not working on multi node @chenqianwen
• [Hotfix][Metrics]fix sporadic multi-metrics ci @corgy-w
• [Fixbug]doris custom sql work @zhangshenghang
• [Fix]Fix dead link on seatunnel connectors list url @Hisoka-X
• [FIX][E2E]Modify the OceanBase test case to the latest imageChange image @xxsc0529
• [Fix][Seatunnel-core]Fix syntax error in the execution script on Windows @cl0924
• [Hotfix][MySQL-CDC]Fix package name spelling mistake @ISADBA
• Revert “[Fix][Connector-V2]Fix doris primary key order and fields order are inconsistent (#7377)” @Hisoka-X
• [Hotfix][MySQL-CDC]Fix ArrayIndexOutOfBoundsException in mysql binlog read @hailin0
• [Bugfix][Doris-connector]Fix Json serialization, null value causes data error problem. @liugddx
• [Fix][Connector-V2]Fix doris primary key order and fields order are inconsistent @Hisoka-X
• [Hotfix][Connector]Fix jdbc compile error @hailin0
• [Fix][Zeta]Fix task can not end cause by lock metrics failed @Hisoka-X
• [Fix]Fix http e2e case @jackyyyyyssss
• [Fix]Fix oracle sample data from column error @Hisoka-X
• [FIX][SFTP]Fix username parameter error in sftp sink document @wuchunfu
• [Fix][Doris]Fix the abnormality of deleting data in CDC scenario. @liugddx
• [Hotfix][Connector]Fix kafka consumer log next startup offset @hailin0
• [Fix][Connector-V2][OceanBase]Remove OceanBase catalog’s dependency on mysql driver @xxsc0529
• [Fix][Doc]Fix miss sink-options-placeholders.md in sidebars @Hisoka-X
• [Fix][Doc]Fix hybrid cluster deployment document display error @Hisoka-X
• [Hotfix][Zeta]Fix task cannot be stopped when system is busy @hailin0
• [Hotfix][Zeta]Fix task cannot be stopped when system is busy @hailin0
• [Fix][Connector-V2]Fix doris TRANSFER_ENCODING header error @liugddx
• [Fix][Dist]Remove aws jar dependency in release package @Hisoka-X

improve

• [Improve][Connector-v2]Release resource in closeStatements even exception occurred in executeBatch in jdbc @dailai
• [Improve][Connector-v2]Support mysql 8.1/8.2/8.3 for jdbc @dailai
• [Feature][Core]Added rest-api for batch start and stop @zhangshenghang
• [Improve][Test]Remove useless code of S3Utils. @FlechazoW
• [Improve][Document]Fix paimon.apache.org deadlink @FuYouJ
• [Improve][Connector-V2]Fake supports column configuration @corgy-w
• [Feature][Core]Add event notify for all connector @zhangshenghang
• [Improve][Connector-V2]Remove hard code iceberg table format version @Hisoka-X
• [Improve]Update docker doc and build tag @Hisoka-X
• [Feature][Connector-V2][Tablestore]Support Source connector for Tablestore #7448 @cloud456
• [Improve]Improve some connectors prepare check error message @zhangshenghang
• [Feature][Connector-V2]Add decimal_type_narrowing option in jdbc @Hisoka-X
• [Improve]Added OSSFileCatalog and it’s factory @loustler
• [Improve]Refactor S3FileCatalog and it’s factory @loustler
• [Feature][Connector-V2]Support typesense connector @zhangshenghang
• [Improve][Connector-V2]update vectorType @corgy-w
• [Improve][API]Add IGNORE savemode type @jw-itq
• [Improve][API]Move catalog open to SaveModeHandler @Hisoka-X
• [Improve]Update snapshot version to 2.3.8 @wuchunfu
• [Feature][Connector-V2]Support opengauss-cdc @dailai
• [Improve][Doc]Update the doc structure @Hisoka-X
• [Feature][Connector-V2]Fake Source support produce vector data @corgy-w
• [Improve][Connector-V2]Remove system table limit @Hisoka-X
• [Improve][Connector-V2]Close all ResultSet after used @Hisoka-X
• [Improve][Document]Update the guide and add example @yujian225
• [Improve]Update pull request template and github action guide @Hisoka-X
• [Feature][Connector-V2][OceanBase]Support vector types on OceanBase @xxsc0529
• [Improve]Skip downloading transitive dependency when install plugins @kevinjmh
• [Improve][Connector-V2]Support multi-table sink feature for email @corgy-w
• [Improve][Document]Change deprecated connector name in setup.md @Hisoka-X
• [Feature][Rabbitmq]Allow configuration of queue durability and deletion policy @zqr10159
• [Feature][connector-v2]Add Kafka Protobuf Data Parsing Support @zhangshenghang
• [Improve][Connector]Add multi-table sink option check @hailin0
• [Improve][API]Check catalog table fields name legal before send to downstream @Hisoka-X
• [Improve][Connector-V2][Console]Add existing rules to the optionRule @chaos-cn
• [Feature][Connector-V2]add Aliyun SLS connector #3733 @XenosK
• [Feature]add dockerfile @liunaijie
• [Improve][Connector-V2]Improve doris error msg @Hisoka-X
• [Improve][Connector-V2]Reuse connection in StarRocksCatalog @Hisoka-X
• [Feature][Zeta]Added other metrics info of multi-table @corgy-w
• [Improve][Connector-V2][Jdbc]Remove MysqlType references in JdbcDialect @whhe
• [Improve][E2E]Support windows for the e2e of paimon @hawk9821
• [Improve][Transform]Improve DynamicCompile transform @jackyyyyyssss
• [Improve][Jdbc]Merge user config primary key when create table @hailin0
• [Improve]Update snapshot version to 2.3.7 @Hisoka-X
• [Feature][Transform]Add LLM transform @Hisoka-X
• [Feature][Connector-V2]Support Qdrant sink and source connector @Anush008
• [Improve][SQL-Transform]Remove escape identifier from output fields @hailin0
• [Improvement]jdbc support dialect.name to choose dialect. @liunaijie
• [Improve][Jdbc]Skip all index when auto create table to improve performance of write @dailai
• [Feature][Transforms]Support cast to bytes function of sql @hailin0
• [Feature][Spark]Support multiple tables read and write @Carl-Zhou-CN
• [Improve]Improve thrift shade packages in doris jar @luzongzhu
• [Improve][Core]Improve base on plugin name of lookup strategy @corgy-w
• [Improve][Transform]Improve DynamicCompile transform @jackyyyyyssss
• [Feature][Jdbc]Support hive compatibleMode add inceptor dialect @Xuzhengz
• [Improve][Connector-v2]Optimize the way of databases and tables are checked for existence @dailai

others

• [Document]Remove a fs.oss.credentials.provider option @loustler
• [Hotifx][Jdbc]Fix MySQL unsupport ‘ZEROFILL’ column type @hailin0
• [ASF]Add new collaborator @Hisoka-X
• [Core][Flink]optimize method name @TyrantLucifer
• [Core][Flink]refactor flink proxy source/sink @TyrantLucifer
• [DOC][Oss]fix document configuration is rectified when the oss is selected as the checkpoint base @Carl-Zhou-CN
• [E2E]Enable JdbcPostgresIdentifierIT @hailin0
• [E2E]Enable fakesource e2e of spark/flink @hailin0
• [E2E]Remove unused code @hailin0
• Bump org.apache.activemq:activemq-client from 5.14.5 to 5.15.16 in /seatunnel-connectors-v2/connector-activemq @dependabot[bot]
• Update Oracle-CDC.md @virvle
• [Docs]translate event-listener doc into chinese @OswinWu

apache/seatunnel

fix

• [Hotfix][Zeta]Fix taskgroup failed log lost @EricJoy2048
• [Fix][Zeta]Fix release slot resource twice @Hisoka-X
• [Fix][Zeta]Fix hybrid deployment can not get worker when init @Hisoka-X
• [Fix][Connector kafka]Fix Kafka consumer stop fetching after TM node restarted @litiliu
• [Fix][E2E]Fix classloder number @jackyyyyyssss
• [HotFix] [Remote Debug]Description When remote debug is enabled, the Seatunel Server 5801 port is not listening properly @hawk9821
• [Fix][Connector-V2][Maxcompute]Fix Maxcompute sink can’t map field @panpan2019
• [Fix][Connector-V2]Fix Hana type converter decimal scale is 0 convert to int error @Hisoka-X
• [Bugfix][RestApi]Fix hazelcast/rest/maps/submit-job return an wrong jobId @liugddx
• [hotfix][connector-v2-hbase]fix and optimize hbase source problem @zhangshenghang
• [Fix][Connector-V2]Fix kafka batch mode can not read all message @Hisoka-X
• [Fix][Connector-V2][Elasticsearch]Fix sink configuration for DROP_DATA @Wudadada
• [bugFix][Connector-V2][CDC]SeaTunnelRowDebeziumDeserializationConverters NPE @Asura7969
• [Hotfix][connector-v2-rabbit]fix rabbit checkpoint exception in Flink mode @zhangshenghang
• [Hotfix][Zeta]Fix batch tasks cannot be stopped after retrying @hailin0
• [Hotfix][Hive Connector]Fix Hive hdfs-site.xml and hive-site.xml not be load error @EricJoy2048
• [Fix][Flink-starter]Hotfix flink checkpoint default value @zhangshenghang
• [Fix][Zeta]Fix env jars not working on zeta @Hisoka-X
• [fix] [connector-tdengine]fix sql exception and concurrentmodifyexception when connect to taos and read data @alextinng

improve

• [Improve][API]Make sure the table name in TablePath not be null @Hisoka-X
• [Feature] [Activemq]Added activemq sink @asapekia
• [Improve][Connector-v2]Optimize the count table rows for jdbc-oracle and oracle-cdc @dailai
• [Improve][Zeta]Handle user privacy when submitting a task print config logs @hawk9821
• [Feature][Connector-V2][Iceberg]Support Iceberg Kerberos @luzongzhu
• [Improve][Core]Move MultiTableSink to seatunnel-api module @Hisoka-X
• [Improve][Connector-V2]Add the nullable attribute of the field for paimon @hawk9821
• [Improve][Doris Connector]Unified serialization method,Use RowToJsonConverter and TextSerializationSchema @liugddx
• [Feature]Split transform and move jar into connectors directory @liunaijie
• [Feature][Zeta]Added the metrics information of table statistics in multi-table mode @hawk9821
• [Improve][Hive]Close resources when exception occurs @hailin0
• [Improve][Restapi]Add ip and port to monitoring information @liugddx
• [Improve][CI]Add milvus to label @Hisoka-X
• [Improve]Improve error message when can not parse datetime value @Hisoka-X
• [Feature][Connector-V2]Suport choose the start page in http paging @Gxinge
• [Improve][Jdbc]Remove user info in catalog-table options @hailin0
• [Improve][CI]Add auto label action @Hisoka-X
• [Feature][RestAPI]overview support tag filter @liunaijie
• [Feature][Transform]DynamicCompile add transform @jackyyyyyssss
• [Feature][Connector-V2]Support multi-table sink feature for HBase @BruceWong96
• [Improve][Jdbc]Support write unicode text into sqlserver @hailin0
• [Feature][Connector-V2][Milvus]Support Milvus source & sink @Thomas-HuWei
• [Improve][connector-v2-jdbc-mysql]Add support for MySQL 8.4 @wuchunfu
• [Feature][Connector-V2][Clickhouse]Add clickhouse.config to the source connector @loustler
• [Improve][Jdbc]Change catalog table log to debug level @Hisoka-X
• [Feature][Core]Support using upstream table placeholders in sink options and auto replacement @hailin0
• [Feature][Core]Add a test case for dynamic custom parameters @liugddx
• [feature][connector-v2-hbase-sink]Support Connector v2 HBase sink TTL data writing @zhangshenghang
• [Improve][Document]Fix Postgre to PostgreSQL in Postgre-CDC.md file @Vonng
• [Improve]Fix plugin_config @hailin0
• [Feature] [Github CI]module runs repeatedly @hawk9821
• [Improve][Redis]Redis reader use scan cammnd instead of keys, single mode reader/writer support batch @FuYouJ
• [feature][connector-file-local]add save mode function for localfile @chl-wxp
• [Improve][E2E][Doris]Add no schema test case for connector-doris-e2e @bingquanzhao
• [Improve][Doc]Improve DingTalk doc @MRYOG
• [Improve][Zeta]Add retry when request slot on resource manager @Hisoka-X
• [Improve][Doc]Add IMap OSS config jars into document @zhouhoo
• [Improve][Zeta]Add check for submit duplicate job id @Hisoka-X
• [Feature][Connector-V2]Support jdbc hana catalog and type convertor @Hisoka-X
• [Feature][Zeta]Add Master and Worker split mode deployment @EricJoy2048
• [Improve][Connector-V2]Support schema evolution for mysql-cdc and mysql-jdbc @dailai
• [Improve][hbase]The specified column is written to the specified column family @zhilinli123
• [Feature][connector][kafka]Support read Maxwell format message from kafka #4415 @zhilinli123
• [Feature][Connector-V2] [Hudi]Add hudi sink connector @liugddx

others

• [Chore][NOTICE]Correct year in NOTICE file @wuchunfu
• [Docs]Update engine related docs info @tcodehuber
• [Doc]hdfs file doc correct @liunaijie
• [E2E]Fix iceberg e2e image tag @hailin0
• [Docs]Update contribution and readme related docs info @tcodehuber
• [Docs]Update transform about and faq related docs info @tcodehuber
• [Docs]Update concept related docs info @tcodehuber
• [Docs]Update quick start v2 related docs info @tcodehuber
• Bump org.xerial.snappy:snappy-java from 1.1.8.3 to 1.1.10.4 in /seatunnel-connectors-v2/connector-hudi @dependabot[bot]
• [CI]Fix k8s engine it not run when only test case changed @Hisoka-X
• [Docs]update rest-api doc @liunaijie
• [Chore]remove useless codes @TyrantLucifer
• [CI][Test]Fix engine-k8s-it failed @Hisoka-X
• [typo]typo the hint of savemode execution location @liunaijie
• [Doc]correct the document @liunaijie
• [Docs][connector-sink]add Docs connector v2 sink @zhangshenghang
• [Docs]translate clickhousefile,phoenix,rabbitmq,starrocks sink doc into chinese @latch890727

apache/seatunnel-website

fix

• [Fix][Doc]Fix miss sink-options-placeholders.md in sidebars @Hisoka-X
• [Fix][Doc]Fix hybrid cluster deployment document display error @Hisoka-X

others

• add new committer @liunaijie
• Add release docs 2.3.7 fix @wuchunfu
• Add release docs 2.3.7 @wuchunfu
• Revert old version doc @Hisoka-X
• fix version script @baicie
• Add release docs 2.3.6 @Hisoka-X
• fix: sync image @baicie

apache/seatunnel-web

fix

• [Bug] [Seatunnel-web]GET /seatunnel/api/v1/job/executor/resource API is not using logged-in user. @arshadmohammad
• [Bug] [Seatunnel-web]When job execution initialization fails, the job execution status remains unchanged. @arshadmohammad
• [Bug] [Seatunnel-web]Create log directory if it does not exist. @arshadmohammad
• [Bug] [Seatunnel-web]Validation messages are wrong @arshadmohammad
• [Fix]Fix the run_time calculation to account for missing days. @XIAOKUOAIyang
• [Bug][Seatunnel-web]Jwt Algorithm is hardcoded. Use the configured j… @BilwaST
• [Bug] [Seatunnel-web]seatunnel-web start and plugin download scripts by default do not have execute permission @arshadmohammad
• [Bug] [Seatunnel-web]Seatunnel-web always requires the cluster name to be seatunnel @arshadmohammad
• [Bug] [Seatunnel-web]gc.log, nohup.out is created outside of logs directory @arshadmohammad
• [Improve][feature][Bug]Add Tooltip and Fix Design Issue In the Task Definition @SachinS10-lab

improve

• [Improvement] [Seatunnel-web]Improve code in DatasourceServiceImpl,SeaTunnelEngineMetricsExtractor etc. @arshadmohammad
• [Improvement] [Seatunnel-web]Add API to get and update job definition. @arshadmohammad
• [Improvement] [Seatunnel-web]Add API to get job execution status. @arshadmohammad
• [Improvement] [Seatunnel-web]Add support to provide reason for job failure @arshadmohammad
• [Feature] [Seatunnel-web]Add support to execute job with parameter through API. @arshadmohammad
• [Improvement] [Seatunnel-web]Bump seatunnel version from 2.3.6 to 2.3.7 @arshadmohammad
• [Feature][Seatunnel-web]Add support to create job with single API @arshadmohammad
• [Improvement][Seatunnel-web]Execute job REST API should be of type POST, not GET @arshadmohammad
• [Improvement][Seatunnel-web]Add support to execute seatunnel-web REST API e2e @arshadmohammad
• [Improvement][Seatunnel-web]Bump frontend-maven-plugin from 1.11.3 to 1.15.0 and slf4j.version from 1.7.25 to 1.7.36 @arshadmohammad
• [Feature][Seatunnel-web]Add FakeSource and Console data sources into seatunnel-web @arshadmohammad
• [Improve] [Seatunnel-Web]Add support for Seatunnel 2.3.6 in Seatunnel-Web @arshadmohammad

others

• [Chore]Code clean @ZhangWeike2000
• [Build]Update snapshot version to 1.0.2 @Hisoka-X
• Revert “Update README.md for fix the seatunnel.sh cmd” @Hisoka-X
• Update .asf.yaml @Hisoka-X
• Update README.md for fix the seatunnel.sh cmd @zhangm365
• add icon @liunaijie

About Apache SeaTunnel

Apache SeaTunnel is an easy-to-use, ultra-high-performance distributed data integration platform that supports real-time synchronization of massive amounts of data and can synchronize hundreds of billions of data per day stably and efficiently.

Welcome to fill out this form to be a speaker of Apache SeaTunnel: https://forms.gle/vtpQS6ZuxqXMt6DT6 :)

Why do we need Apache SeaTunnel?

Apache SeaTunnel does everything it can to solve the problems you may encounter in synchronizing massive amounts of data.

• Data loss and duplication
• Task buildup and latency
• Low throughput
• Long application-to-production cycle time
• Lack of application status monitoring

Apache SeaTunnel Usage Scenarios

• Massive data synchronization
• Massive data integration
• ETL of large volumes of data
• Massive data aggregation
• Multi-source data processing

Features of Apache SeaTunnel

• Rich components
• High scalability
• Easy to use
• Mature and stable

How to get started with Apache SeaTunnel quickly?

Want to experience Apache SeaTunnel quickly? SeaTunnel 2.1.0 takes 10 seconds to get you up and running.

https://seatunnel.apache.org/docs/2.1.0/developement/setup

How can I contribute?

We invite all partners who are interested in making local open-source global to join the Apache SeaTunnel contributors family and foster open-source together!

Submit an issue:

https://github.com/apache/seatunnel/issues

Contribute code to:

https://github.com/apache/seatunnel/pulls

Subscribe to the community development mailing list :

dev-subscribe@seatunnel.apache.org

Development Mailing List :

dev@seatunnel.apache.org

Join Slack:

https://join.slack.com/t/apacheseatunnel/shared_invite/zt-1kcxzyrxz-lKcF3BAyzHEmpcc4OSaCjQ

Follow Twitter:

https://twitter.com/ASFSeaTunnel

Join us now!❤️❤️