How to add a MongoDB running inside Docker as a source?

[HansiMcKlaus]

Hi everyone,

I'm currently running Airbyte, MongoDB and and PostgreSQL as containers inside Docker. Adding the PostgreSQL database as a destination in Airbyte was no problem, but when attempting to add the MongoDB as a source, I get the following error message:

Configuration check failed:
Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake}, caused by {java.io.EOFException: SSL peer shut down incorrectly}}]

I can successfully connect to the MongoDB locally with the MongoDB Compass via the connection string: mongodb://localhost:27017/. The database has no password.

GUI when attempting to add the MongoDB as a source in Airbyte:
Cluster Type: Self-Managed Replica Set
Connection String: mongodb://localhost:27017/
Database Name: <DATABASE NAME>
Username:
Password:
Authentication Source: admin

On a separate attempt, I explicitly created the MongoDB database as a replica set by including command: ["--replSet", "replica", "--bind_ip_all", "--port", "27017"] in the docker-compose.yml and later initiating it via rs.initiate(); in the mongo shell. The connection string changed to mongodb://localhost:27017/?directConnection=true, but it resulted in the same error as above.

Are there any additional settings I have to change in the MongoDB database to add it as a source, or is it a network problem?

[marcosmarxm]

Did you try to use different local address to connect to the container? (localhost, 127.0.01 or host.docker.internal)
If these doesn't solve the problem you probably need to create a network both Airbyte and your local Mongo are connected.

[HansiMcKlaus]

Changing localhost to 127.0.01 or host.docker.internal still results in the same error. When using host.docker.internal, I also can't connect via the MongoDB Compass anymore.

In the docker-compose.yml, I tried to add the MongoDB container to the networks airbyte_public and airbyte_internal, by including:

networks:
  airbyte_public:
    name: airbyte_public
    external: true
  airbyte_internal:
    name: airbyte_internal
    external: true

Result stays the same with the same error message.

I already attempted to add the MongoDB container and Airbyte (There are 9 containers running, so I added them all) to a new network, which also didn't work, though I don't remember whether it was the same error.

Is it enough to simply add the containers to a network, or are there additional options? Also, does one have to add all the containers from Airbyte, or is there one specific container that needs to be included?

[marcosmarxm]

Did you try to connect using the IP for the container?
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' YOUR_CONTAINER_NAME

[HansiMcKlaus]

Using the IP for the container results in the following error: Configuration check failed Failed to run connection tests.

I also recreated the network with all Airbyte containers and the MongoDB container inside. Connecting via mongodb://localhost:27017/?directConnection=true, mongodb://127.0.0.1:27017/?directConnection=true or mongodb://172.19.0.2172.18.0.2:27017/?directConnection=true (Container IP) results in the same error in Airbyte.

I also can't connect via MongoDB Compass using the Container IP, as it results in a connection timed out.

EDIT: Turns out Airbyte didn't run properly. When using the Container IP, I get the same "Configuration check failed" as in the initial post. However, upon including all the containers in the same network, I get the Configuration check failed Failed to run connection tests.

Edit 2: Adding the Airbyte Containers to a shared network seems to break Airbyte, as other connections that previously worked, now return the same Configuration check failed Failed to run connection tests. and start working again after deleting Airbyte from Docker and restarting it using ./run-ab-platform.sh.

[hmtrii]

Any idea? I faced the same issue

[quguangyu0321]

I faced same issue, tried docker network, also tried different mongo version and so on, it not works.

[sjtutim]

I faced the same issue，who can help？ Thanks

[daviibf]

I am facing the same issue. Any updates on this ?

[daviibf]

Actually, my error is slightly different:

Configuration check failed
Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongodb:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb}, caused by {java.net.UnknownHostException: mongodb}}]

I've verified the network are the same from airbyte services and the source. Even if I access the airbyte container and try to ping the mongodb service, it works. But whenever I try to connect it as a source, it doesn't work.

[cagdaskarademir]

I faced the same or similiar issue and found a workaround solution. First, I downgraded the version of source-mongodb-v2 (as I received an SSL handshake error and upon checking the code, found this version to be the cause).

Although I have 3 nodes, I couldn't connect using the multiple host method, so I selected the secondary host and tried that instead.

It's done. Perhaps this approach can help you too.

[briankariuki]

Using host.docker.internal solved this issue for me

[saadqc]

Same issue, the credentials are valid and I have mongo running on LAN server. Which is available even on router level.
I can connect from my laptop to the server mongo instance over lan using compass or python mongoclient but airbyte connector fails

[briankariuki]

I had to look for another solution, as I couldn't get airbyte to work with mongo.