[URGENT] How to monitor internal tasks of Hasura Graphql Engine tasks to debug cpu / memory usage problems

[Aarbel]

Hi Hasura team and community.

Since 2 days, we are facing crashes on our Hasura AWS ECS Instance without any insights about the crash.
The ECS instance reboots and re-crashes 2 mins after.

Hasura is like a blackbox for us, we are logging the Hasura tasks logs, buts it's not really helpful to track the origin of too high CPU and memory usage.

What is the best methodology to monitor well the Hasura docker image and Hasura internal processes ?

Cd bellow some screenshots about monitoring and logs on 1 day. Note that CPU maximum jumps to 100% or over for few seconds and are not visible in the charts. (moreover on the last screenshot, we can see we added an additional ECS task, to prevent some potential crashes for ours users, the time to find a solution to monitor well and debug).

[adas98012]

Hi,
Thanks for reaching out. We need some additional information about your project, to assist you better:

What is the Version of Hasura CE you are using ?
Are you using Hasura Cloud (free tier, standard tier or pro) ?

Thanks,

[Aarbel]

@adas98012 thanks a lot for your message

we fully removed graphql websocket subscriptions and it solved our CPU problems. Looks like Hasura has really deep problems with websockets, and also makes it very hard to debug in the same time.

Anyway, we are using Hasura 2.4.0 and not Hasura Cloud.

Recommendations about tools to monitor well theses kind of "weird" infrastructure problem would be absolutely welcome