[Mongo Source] Mongo Sharded Support

[whanata]

I was wondering if there is any plans to support Mongo Shards?

Right now, I'm treating each Mongo Shard as a Replica Set and have 3 seperate connections for my 3 shards.

It works fine, but I'm seeing overtime small discrepancies on the end data such as:

• When a specific document gets updated, in the airbyte internal table it was signified as it's been deleted
• In Airbyte Internal, even though a specific document existed, it wasn't pushed to the final tables

I was hoping if there was any plans to support Mongo Shards since I believe some of these problems are attributed to listening to the actual Mongo Servers vs the Mongos Router and the result of the change streams that come out of it.

I'm using OSS currently using Docker Compose, and planning to move to using kind to run the Kubernetes version of Airbyte in a EC2 Box using Docker Containers.

[JerbiAhmed]

Starting with MongoDB 8.0, the listCollections command is no longer supported when connecting directly to each replica set shard causing airbyte connection to fail —unless the user has the directShardOperations role. However, as per the documentation, granting this role is strongly discouraged:

https://www.mongodb.com/docs/manual/reference/supported-shard-direct-commands/
https://www.mongodb.com/docs/manual/sharding/#connecting-to-a-sharded-cluster
https://www.mongodb.com/docs/manual/reference/built-in-roles/#mongodb-authrole-directShardOperations

If you’re using MongoDB Atlas, directShardOperations role isn’t available, and so far,we have no viable workaround.