Laravel Horizon Default Supervisor Inconsistently Going Offline

[hyp-azam]

Laravel Horizon Supervisor Inconsistently Going Offline

Versions:

• Laravel version: v10.35.0
• Horizon version: v5.21.4

Environment:
Running horizon on an ECS container

Problem:
I recently upgraded Laravel from 8 to 9 and then finally 10. My horizon command is working and jobs are processing but on the dashboard, one of my supervisors (a default supervisor "supervisor-1") intermittently goes offline and then comes online again.

Upon investigating, I checked the horizon routes specifically {{host}}/horizon/api/masters API which has the supervisors array. There I saw that supervisor-1 was in running state with some id and then after some time when another API was called the supervisor was in an inactive state. Then after some time on another API, supervisor-1 was in running state but the PID had changed from what it was earlier. This indicates that it was somehow restarted. As for other supervisors which are not having the problem, the PID remains the same.

I have checked supervisor logs and ECS container logs but did not find anything unusual.

1. supervisor-1 running with PID: 20171

2. supervisor-1 now inactive

3. supervisor-1 running again with PID: 20517 (changed)

Supervisor Configuration:

'defaults' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'balance' => 'simple',
        'queue' => ['default'],
        'maxTime' => 3600,
        'maxJobs' => 1000,
        'minProcesses' => 5,
        'maxProcesses' => 10,
        'memory' => 32,
        'tries' => 1,
        'balanceMaxShift' => 5,
        'balanceCooldown' => 3
    ],
],

[hyp-azam]

I solved the issue. The problem was caused by the memory allocation for supervisor-1, which was set to 32MB, while the other supervisors had 64MB allocated. When I increased the memory to 64MB, it worked fine. I am just surprised that Laravel did not provide any logs regarding the supervisor crashing in the Horizon logs.

[siddiquiazam]

It seems we can listen to the following events provided by horizon:

1. MasterSupervisorOutOfMemory
2. SupervisorOutOfMemory

We can add listeners for these events and then log or alert however we want.