[Roadmap] Source MongoDB to Certified

[Hesperide]

[Roadmap] Source MongoDB to GA

To get early access and share your feedback on proposed improvements to our MongoDB connectors, please fill out this form.

We're thrilled to bring you a host of robust updates to our MongoDB source connector. These changes significantly augment its scalability and reliability, especially for large databases:

• Unlimited Size Initial Snapshots: We're introducing support for syncing MongoDB tables larger than 1 TB. This means no matter how large your MongoDB database is, Airbyte has you covered.
• Change Data Capture (CDC): We’ve found that CDC leveraging a combination of Change Streams and the MongoDB OpLog is the most reliable method of maintaining an up-to-date copy of your MongoDB tables. We've also refined our ability to handle large incremental updates to your tables. Moving forward, CDC is the only supported method for connecting to MongoDB clusters.
• GA Connector Reliability: We’ve resolved many of the top issues you’ve encountered replicating data from MongoDB, particularly with discovering MongoDB schemas.

Behind the scenes, these changes close the gap with our most reliable database sources, such as Postgres and MySQL. Together, your syncs are now always resumable, and shielded against network hiccups that can arise when syncing massive data volumes - as is now the norm for all certified Airbyte database connectors.

If you are an existing user of our Mongo connector (source-mongodb-v2), you will now be prompted to upgrade and reset your Mongo connector in the next 60 days to benefit from these new improvements.

This new iteration of the MongoDB connector leaves a small number of workloads unsupported. These are:

• A minority of MongoDB collections with very dynamic schemas.
• MongoDB clusters running on versions below 4.4 (deprecated versions)
• MongoDB clusters of type ‘standalone’ (no replica sets)
• Replicating data from system collections (system., replset., oplog.*)
• Support for replicating data from views, capped or clustered collections
• Other MongoDB clusters that do not support the OpLog / Change Data Streams

---

We're seeking early access users to test these new features and share their feedback as they become available. You can sign for the beta here. You may also leave an emoji response on this description. A 👍 indicates you generally approve of this plan, and a 👎 indicates you do not. Thank you for your ongoing support, as always.

For additional context, refer to some of the top open MongoDB issues:

• Source MongoDB: CDC/Log based replication #2881
• 🐛 Mongo to BigQuery: Long (Int64) is serialized as float-like (scientific) number #9590
• MongoDB Source Connector Fails Schema Fetch On Empty keys #19206

[haithem-souala]

Any ETA, as the current mongodb source version is not working for databases with medium to large set of collections.
Unknown error (http.504.iiBiWsjYibLWn3dGRhuYUd) when fetching schema.

[Hesperide]

@haithem-souala - the team is starting to work on these improvements imminently, and we expect to ship some of the above updates (including the error you've mentioned) in the next 1-2 months.

[nikileshsa]

will this include fixes to the destination connector as well? @Hesperide #29107

[gititdom]

I am getting error "Target MongoDB instance is not a replica set cluster." when attempting to connect to a Sharded Cluster. When will the sharded cluster be supported?

[renatodossantosleal]

Is this error fixed in any version? I'm getting the same problem with large schema on BigQuery.

[postamar]

Source-mongodb-v2 is certified now, closing.

[losblancoo]

Are there plans for reintroducing support for MongoDB clusters of type ‘standalone’ (no replica sets). As it was available in prior connector versions(< 1.0.0)?