SeaTunnel Plugin Development Experience: Embracing CatalogTable and SaveMode

In the latest plug-in development of Apache SeaTunnel, the connector-v2 maxcompute connector implements a new version based on CatalogTable + SaveMode.

This article mainly shares with you the key changes on the source side, including the abandonment of outdated methods and the replacement of data transfer through CatalogTable. Huiduan has added the implementation of multi-table sink and saveMode interfaces, and requires manual management of auto service annotations and other advanced experience methods! **

In addition, specific catalogs were developed to handle public parameters and client operations.

The connector-v2 maxcompute connector has embraced a significant evolution with the integration of CatalogTable and SaveMode, marking a milestone in the SeaTunnel plugin’s journey.

Core Changes on the Source Side

The SeaTunnelSource now enriches its lifecycle with SatunnelPluginLifeCycle interface’s prepare method — a shift aimed at initializing source configuration parameters alongside pre-job actions like data deletion and SQL execution.

Key modifications include:

• The discontinuation of two deprecated methods.
• A new constructor approach that crafts a CatalogTable, paving the way for data transmission through the List<CatalogTable> getProducedCatalogTables() method.
• A refined SeatunnelRowType retrieval directly from catalogTable, enhancing adaptability for existing connectors and streamlining the transfer of common parameters.

Catalog Insights

Harnessing getTable to fetch CatalogTable, the connector adapts to custom columns (column) configured for DataX compatibility, generating CatalogTable based on these specified columns.

Sink Side Enhancements

The MaxcomputeSinkFactory has broadened its horizon with an extra createSink method. It now fetches catalog and ReadonlyConfig directly from the context, stepping away from the auto service annotation to welcome multi-table sink and saveMode interface implementations.

Reflecting on the Removal of auto_service Annotation
Emphasize critical changes with bold text or another pull quote.

The removal of the auto_service annotation signals a decisive move towards a streamlined, context-aware constructor that adeptly handles catalogtable for internal seatunnelRowType adaptation.

Closing Thoughts

As the SeaTunnel plugin continues to evolve, these foundational changes not only refine its current capabilities but also set the stage for future innovations that will further empower data management and processing.