Mongo replicas are not created on production deployment

[miyuru-vs1bv]

I am setting up our production env and I have set for 3 replicas.

But in the docker swarm its only showing 1 mongo service.

Is this correct or is there should be 3 mongo services when replicas set for 3?

[pankaj-pant]

Hi @miyuru-vs1bv. May I ask how have you configured the mongo replicas?

For Mongo the docker compose replicas variables doesn’t work but instead it needs to configured so each “replica” is a separate entity in the compose file

As a reference point, in the Farajaland repo, the mongo1 replica is configured here

To add a second replica mongo2 in prod for example, it needs to be configured in your country-config repo like so. If you want to add a third “replica” you’d add a new entry mongo3: … below the mongo2 service configuration

The mongo2 (...and other mongo repilcas) configured in this file docker-compose.production-deploy.yml are deployed with this file docker-compose.deploy.yml - that has the mongo1 replica configured. The reason for this is that the mongo “replicas” need to work together and know about each other, and it’s a level more sophisticated than spawning 3 replicas of the auth service for instance

[miyuru-vs1bv]

thank you, it seems that country config repo does not have that file updated.

https://github.com/opencrvs/opencrvs-countryconfig/blob/develop/infrastructure/docker-compose.production-deploy.yml

i'll try the Farajaland file and update you on the progress

[pankaj-pant]

@euanmillar could you please update the docs regarding adding mongo replicas?

[miyuru-vs1bv]

I am using the same config as Farajaland and getting the above error.

[pankaj-pant]

Ok, this seems like an error related to an incorrect label used on the second mongo replica. Could you confirm this by inspecting the labels of the nodes using the following command?

docker node inspect <id-of-node> --format '{{ json .Spec.Labels }}'

If both of them print the same thing: {"data1":"true"} - then that needs to be fixed.

As you can see here, the second mongo replica in Farajaland has its specific data label which corresponds to the label of a separate machine. The labels force the mongo replicas to be always on different machines.

[miyuru-vs1bv]

looks like the labels are not set on worker nodes

[pankaj-pant]

Ok, then you need to set distinct labels on the worker nodes as is written in the documentation. So prod-node1 has data_label set to data1, please provide labels to the worker nodes as well in similar pattern. prod-node2 should have data_label set to data2, so on.

And after that we need to make sure that the mongo replicas are also pointed to the respective worker node using this rule

[euanmillar]

That's right. This is a step that requires running the Provision action again to create the labels associated with worker nodes.

[miyuru-vs1bv]

Thanks @pankaj-pant its working now.