Postgres transactions aborted

[simonphughes]

I have hasura running in production against an PostgreSQL database which is hosted on an AWS Aurora Serverless cluster. The solution works really well in general with the platform scaling up resources on demand, and scaling back down in quiet times. What I am seeing almost daily now as the usage is ramping up is HTTP500 server errors coming from hasura.

I used the cloudformation script that is provided on github to create our environment which includes load balanacer at the front end, hasura running in 2 containers on a Fargate ECS cluster, and the serverless aurora cluster. The platform always records, and we only see a moment of downtime which so far I believe isn't affecting customers, but I would like to get a better understanding of what is going wrong.

In the cloudwatch logs, the first event out of the ordinary has the following info in it:
"internal": {
"statement": "BEGIN ISOLATION LEVEL REPEATABLE READ ",
"prepared": true,
"error": {
"exec_status": "FatalError",
"hint": null,
"message": "SET TRANSACTION ISOLATION LEVEL must be called before any query",
"status_code": "25001",
"description": null
},
"arguments": []
},

This is then followed by:
WARNING: there is already a transaction in progress

Then I get the following:
"internal": {
"statement": "ABORT",
"prepared": true,
"error": "cmd expected; tuples found",
"arguments": []
},

and finally this:
{
"type": "scheduled-trigger",
"timestamp": "2022-03-16T13:08:02.856+0000",
"level": "error",
"detail": {
"path": "$",
"error": "QErr {qePath = [], qeStatus = Status {statusCode = 500, statusMessage = "Internal Server Error"}, qeError = "postgres tx error", qeCode = postgres-error, qeInternal = Just (Object (fromList [("statement",String "BEGIN ISOLATION LEVEL REPEATABLE READ "),("prepared",Bool True),("error",Object (fromList [("exec_status",String "FatalError"),("hint",Null),("message",String "prepared statement \"5\" already exists"),("status_code",String "42P05"),("description",Null)])),("arguments",Array [])]))}",
"code": "unexpected"
}
}

After that, the AWS load balancer kills off the container hosting it, and spins a new one and carries on working.
Could this be related to scaling events on the aurora cluster blocking things? Any other suggestions?

Many thanks

[drojkind]

Bumping this question as we have been noticing this same issue recently, was not an issue previously. Also using the same setup ECS setup with Aurora Serverless Cluster.

[ReganL]

Bumping this as well, I'm having a similar issue