scaling the database

[gpicavet]

Hello,

I'm having some performance concerns with huge number of clients and a single MongoDB instance.
on heavy load there are a lot of waiting read commands in the mongo queue (almost 200)

So my first question : what would be the best way for scaling out ?
-using replication
-using sharding.

Firstly I'm not used to Mongo, and i dont know if replication would completely remove blocking read !
Secondly, we'd need to supply a list of mongo instances URL and some ReadPreference to the java driver in MongoBootstrap class

Would sharding be a more flexible solution ? (if room activity is correctly distributed among shards, blocking read would drastically decrease ?)

Kind regards

[vsellier]

Hello,

The addon must be updated to allow to change the hostnames and the readPreference, you are absolutely right.
We should add the write concern configuration too to be totally flexible.

Without this features (hostnames and read concern), the only way I see to increase your read capacity is to use sharding with a mongos instance on your chat server.
You'll have to configure manually the sharding for each collection (It's better to use the version 1.2 of the addon to avoid to have one collection per room :) )

The replication could have been a solution for read capacity problem but the replica lags must be carefully monitored to avoid the clients to retrieve old room content from an outdated replica (Mongodb 3.2 add some features to ensure fresh data to be returned on replica's reads but the chat addon is still not validated with this version).

Regards,

[gpicavet]

Thanks ! I'll give a try to the sharding solution !
Is it the solution used by the exo tribes community site ?

By the way, as bottleneck seems to be read messages of a room, i've just noticed that each client get all messages each time it refreshes the view . Would we gain something using a timestamp to get only the diff ?

[thomasdelhomenie]

I open an issue to improve the configuration capabilties to allow to connect to a replica set or to a bunch of mongos servers : https://jira.exoplatform.org/browse/CHAT-386
Do not hesitate to comment if necessary.

By the way, as bottleneck seems to be read messages of a room, i've just noticed that each client get all messages each time it refreshes the view . Would we gain something using a timestamp to get only the diff ?

I agree there is room for improvements here. Can you share more details about your performance tests ?

[gpicavet]

Hello Thomas!

Thanks for the Jira,

Yes i'm doing some load testing as described here, based on the estimated requirements in production.
i've attached the gatling script i'm using ChatSimulation.zip
, basically, it is a modified version of gatling script included in project.
I think the results could be very different depending on the dataset, so i think i'm as close as possible.

I couldn't reach requirements with default application, so i was looking for scaling database, but
i've noticed that adding a "roomId" index on the messages_room_{TYPE} collections considerably improved the response time.

Now i think clustering (replication or sharding) could also be great for higher throughput and high availabilty

[thomasdelhomenie]

Hello Gregory :)

Thanks for the details. I will open a Jira issue for adding the index on roomId (we manage all our issues in Jira, so it will be simplier).
Do you reach the required performance after adding the index on roomId ?

[gpicavet]

Yes ! i have now a constant 750 requests/seconds throughput on a 4 CPU Xeon @ 2,6GHz, which means about 750 users with default chat refresh time & user activity. We have actually 600 simultaneous users in production but we plan to double it.
So i'll have to study clustering the database on two physical servers for example

[dbeller]

We are using 4.3 Enterprise Edition with chat 1.2.

We are in a somewhat similar situation in that we would like to fully leverage a mongodb replica set but do not have a way of specifying in the exo client. We are currently using a virtual IP in front of the mongodb cluster, but that only provides us higher availability, not higher performance. I saw that https://jira.exoplatform.org/browse/CHAT-386 was opened but also recently closed due to inactivity. This is a feature that we were hoping would be in chat 1.4 (exo 4.4) and would definitely still like to see implemented.

[fdrouet]

Hello @dbeller ,
currently the mentioned jira issue is not on the short term roadmap (AKA next 3/4 months) but I will consider adding it to the mid-term one. Take note that we are currently working hard to deliver a new Chat 1.5 version which will use websockets instead of http polling. This will drastically reduce the number of mongodb requests/sec for the same number of concurrent users. This new chat 1.5 version will be compatible with eXo Platform 4.4.

if you are a customer don't hesitate to use your Customer Space channel for such request.

[gpicavet]

Hello,
Websocket will indeed optimize client/server communication, but it will not reduce read/write on the database.
It could also be interesting to split the storage into an in-memory hot storage, which would store unread messages, and a cold storage for messages history.
In-memory storage must be Really Fast and sharable (in case you want to use multiple servers). Something like Redis for example.
But this should also be optional because it's overkill for small organization.