How to Seamlessly Migrate Data From MongoDB to Doris?

3 min readDec 24, 2024

If you need to use Apache SeaTunnel to synchronize data from MongoDB to Doris, you can follow these steps, which are based on the official documentation of Apache SeaTunnel and best practices provided by the community:

I. Environment Preparation

Download and Install SeaTunnel:

Visit the official GitHub page of SeaTunnel and download the latest stable version.
Unzip the downloaded file and configure the necessary environment variables (e.g., JAVA_HOME).
Ensure that MongoDB and Doris databases are installed and running.
Obtain the connection information for MongoDB and Doris, including host addresses, ports, database names, usernames, and passwords.

II. Writing SeaTunnel Configuration File

Create a Configuration File:

In the conf directory of SeaTunnel, create a new configuration file, for example, mongodb_to_doris.conf.

Configure MongoDB Source:

In the configuration file, specify MongoDB as the data source (Source).
Configure MongoDB’s connection information, database name, collection name, and conditions for reading data.

env {
  execution.parallelism = 1
  spark.app.name = "MongoDBToDoris"
  spark.sql.shuffle.partitions = 2
  spark.driver.memory = "1g"
  spark.executor.memory = "1g"
}
source {
  MongoDB {
    host = "your_mongodb_host"
    port = your_mongodb_port
    database = "your_database"
    collection = "your_collection"
    # Other MongoDB connection configurations...
  }
}

Configure Doris Sink:

In the configuration file, specify Doris as the data destination (Sink).
Configure Doris’s connection information, database name, table name, and the format of the written data.

sink {
  Doris {
    jdbc.url = "jdbc:mysql://your_doris_fe_host:your_doris_fe_port/your_database"
    jdbc.user = "your_doris_user"
    jdbc.password = "your_doris_password"
    table = ["your_table"]
    # Other Doris connection configurations...
    column = ["column1", "column2", ...] # Fill in according to the actual table structure
    write_mode = "replace" # Or "append", choose according to your needs
  }
}

III. Running SeaTunnel Task

Submit the Configuration File:

Use SeaTunnel’s command-line tool to submit the configuration file and start the data synchronization task.

./bin/start-seatunnel-spark.sh --config ./conf/mongodb_to_doris.conf

Monitor Task Execution:

Monitor the task execution through SeaTunnel’s log output or Web UI (if enabled).
Ensure that the task can read data from MongoDB and write it to Doris normally.

IV. Notes

Data Format Matching:

Ensure that the data structure in MongoDB matches the table structure in Doris. If there are differences, you may need to perform field transformation or mapping in SeaTunnel’s configuration file.

Performance Tuning:

Adjust SeaTunnel’s configuration parameters, such as parallelism and memory allocation, according to the data volume and synchronization requirements to improve data synchronization performance.

Error Handling:

Configure error handling mechanisms, such as retry policies and log recording, to deal with potential data synchronization errors.

By following these steps, you can use SeaTunnel to synchronize data from MongoDB to Doris. In actual operations, you may need to further configure and adjust according to the specific environment and requirements.

About Apache SeaTunnel

Apache SeaTunnel is an easy-to-use, ultra-high-performance distributed data integration platform that supports real-time synchronization of massive amounts of data and can synchronize hundreds of billions of data per day stably and efficiently.

Welcome to fill out this form to be a speaker of Apache SeaTunnel: https://forms.gle/vtpQS6ZuxqXMt6DT6 :)

Why do we need Apache SeaTunnel?

Apache SeaTunnel does everything it can to solve the problems you may encounter in synchronizing massive amounts of data.

Data loss and duplication
Task buildup and latency
Low throughput
Long application-to-production cycle time
Lack of application status monitoring

Apache SeaTunnel Usage Scenarios

Massive data synchronization
Massive data integration
ETL of large volumes of data
Massive data aggregation
Multi-source data processing

Features of Apache SeaTunnel

Rich components
High scalability
Easy to use
Mature and stable

How to get started with Apache SeaTunnel quickly?

Want to experience Apache SeaTunnel quickly? SeaTunnel 2.1.0 takes 10 seconds to get you up and running.

https://seatunnel.apache.org/docs/2.1.0/developement/setup

How can I contribute?

We invite all partners who are interested in making local open-source global to join the Apache SeaTunnel contributors family and foster open-source together!

Submit an issue:

https://github.com/apache/seatunnel/issues

Contribute code to:

https://github.com/apache/seatunnel/pulls

Subscribe to the community development mailing list :

dev-subscribe@seatunnel.apache.org

Development Mailing List :

dev@seatunnel.apache.org

Join Slack:

https://join.slack.com/t/apacheseatunnel/shared_invite/zt-1kcxzyrxz-lKcF3BAyzHEmpcc4OSaCjQ

Follow Twitter:

https://twitter.com/ASFSeaTunnel