Event trigger spends too much time in pending state

[mustafefe]

Hey all.

I have created an event trigger that sends a request to a node js web api when a data inserted to a specific table. It works and does it's job but the problem is, it spends too much time in pending state. I am kind of newbie with Hasura so could you please share with me if you have any recommendation about how to approach/analyze the issue?

Thanks in advance.

[meetzaveri]

Hey @mustafefe , thanks for reaching out here.

Problem

Reading your question, I can imagine what factors might play here for "event triggers remaining in pending state"

• Increase in event queue time. This means events are spending more time in queue before actual execution. Hence, they will be delivered slowly.
• Business logic in your webhook. This might be responsible where event trigger wait for actual webhook response. You can adjust timeout_sec in Retry Logic configuration as mentioned here

In relation to the 2nd point above, we would recommend you to check your webhook business logic code and find out where its taking too much time.

How to troubleshoot

Coming on to the 1st point above, if event queue time is increasing, that means average event processing rate is slowing down (avg. events processed per second). This might be the bottleneck to where you might need some improvements.

So as you'll see, we have a metric called Hasura event queue time. If you are using hasura enterprise and if you have enabled prometheus metrics, then you will be able to do event triggers observability.

Lets cut down to the solution:

• As mentioned in that above doc link, you can use Events Http Pool size flag or env var for improving event trigger processing. Reference - https://hasura.io/docs/latest/deployment/graphql-engine-flags/reference/#events-http-pool-size
• Say you are experiencing load on your app where tons of event triggers are being spawned simultaneously. Then tweaking this event http pool size parameter will definitely help to ensure your event triggers are delivered in time.
• Another way you can analyze is by manually invoking the trigger rather than for it to happen in flow. This manual invocation of trigger would help you to debug invocation time, invocation response and other test scenarios. Guide on how to invoke trigger manually - https://hasura.io/docs/latest/event-triggers/invoke-trigger-manually/

Ending note

I would like to know few things from your side too.

• How many event triggers on avg. are in pending state at a given time from your side ? Can you shoot me any approx. number ?
• Which platform you are using? Hasura CE (open source), Hasura cloud or Hasura enterprise ?

I hope above explanation helps to troubleshoot your issue. Awaiting response from your side, cheers!