rds read replica lags

[simonphughes]

Not sure whether this is the best place to post this, but I think i have an issue with our platform which is running Hasura Cloud.

We have an RDS PostgreSQL cluster running in AWS cloud with a single writer instance, and a set of read only replicas due to us having a large number of real time subscriptions running for each of our clients. In general our whole platform is running on AWS lambda functions using the serverless framework. Using the Events capability of Hasura we have a number of API calls being made on inserts into tables.

Recently I noticed a strange issue where we are inserting data into a table. The Event is triggering and that is correctly calling the api and I can see the data in the request being passed across correctly. The API call in this case then goes to the database to get the inserted object, and a whole bunch of related data which is needed to process. This is happening 1000s of times a day without issue, yet occasionally we get a failure and the system does not process the insert. Putting in some extra logging I can see that we are calling our Hasura query and its returning no data for the passed in parameters. If I manually run the query with exactly the same parameters it works correctly and returns the data. Its probably working correctly 99.99% of the time, but with such high volumes these failures are unacceptable.

The only explanation I can come to with this, is the original data is being written to the Writer instance of postgres, the Event is kicked off, and when the Lambda API runs the query against the Reader instance, the data hasn't fully replicated.

Does this sound plausible, and any suggestions for how to resolve this?
Thanks

[adas98012]

@simonphughes
Thanks for the update. Please create a support ticket here with all the relevant details : https://cloud.hasura.io/support/create-ticket
You may put a reference to this Github Discussion there as well.

[simonphughes]

Thanks @adas98012 I have raised a support ticket.

[kedarguy]

I ran into a similar issue when adding a read replica, was a solution found?
Should I open a separate support ticket?

[simonphughes]

A few suggestions were made by the support team so I would suggest you raise one. For me the almost all of our logic is set to run off the back of events firing, so I put in a few retries with a small 2 second gap between them and that seems to have resolved our issue.

[adas98012]

@kedarguy
Thanks for reaching out. Apart from the solution that @simonphughes mentioned, here is an alternate one (that we have shared with other users). This one does the retry logic for failed reads (assuming those are few) against a second datasource.
Here is an outline of the solution:

• You have datasource with writer db and one or more Read Replicas. Let's call it main_plus
  Refer Doc

• You can add a second datasource with just writer db (without Read Replicas). Let's call it main

  Note: You'll likely see untracked tables in "main". Track the ones you'll like to retry read from. You may have to add a custom name to resolve name match with previous queries.

• Now you'll see two distinct sets of queries in the console (one set against main_plus and the other against main). You can chose to use the main ones for the retry logic (assuming you have a very small subset of failed reads).