Error The pool is probably full in production

[mkevison]

I am getting the error "KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?"

I've already cleaned up my code and I believe it's not an error related to an unfinished transaction.

My system is well balanced, I have my redis cache server, two database servers, (write and read). Two microservice servers for scheduled actions, a load balancer server using nginx and the main api server.

This error only happens on the main api server

Database replication is done by postgres itself. And I'm using adonis' own replication to define the writing and reading database

My system is a multitenant and has the main database and two tenants on different databases. Depending on the user, my system searches the specific database. the 3 databases run on the same Postgres cluster

It always happens at the same time between 17:00 and 18:00 this fall

I don't know if this is happening due to some user spike or if it's another error in the system.

Whenever this happens I have to restart the servers manually. I use docker to manage my server instances

My system has around 500 users per day according to google analitycs. I don't believe it's peak users

Is there any way to catch this error and restart the database connection? Or any way to trace where this error is connected?

[RomainLanz]

Hey @mkevison! 👋🏻

You should be able to catch this error inside the global exception error.
Then you can call the DatabaseConnectionManager to close all connections to the database.

await Database.manager.closeAll(true)

📚 https://docs.adonisjs.com/reference/database/connection-manager#closeall

But it seems you have an issue with your application directly. If you would like some help, I would recommend creating a repository with the minimum amount of code to reproduce your issue.

[mkevison]

Thanks for your tip.

I installed new relic on my server, to try to monitor and find out where the problem is, I will wait for it to happen again.

I hope I can identify where this error is by this monitoring. If I can't do it, I'll do the suggestion you gave me, I prefer not to need it because that would just be a palliative for the problem

I'm back here to tell what happened

[mkevison]

With the help of New Relic I managed to find the problem.

It was where I least expected it, in the payments api webhook.

And in the end it was a losing transaction.

Turns out the payments api was notifying me of an unpaid payment that was expired, and I didn't know it did that.

And my security system always checked all the webhooks to see if it really was paid, and when it found that it wasn't paid, it returned an error and I didn't close the transaction.

I got to check this route a few times and didn't notice this transaction not ended.

But finally I solved my problem.