Laravel Horizon is scaling the wrong way (auto + max-processes)

[michabbb]

hi all,

i am very surprised i am unable to find anything about that issue.
laravel horizon is scaling - at least for me - completely wrong.
that means every supervisor starts as many workers as "max-processes" from the horizon config.

as seen here:
https://github.com/laravel/horizon/blob/d4e24a9cfff8cacfa44c75ca0ac759984f152d1c/src/Console/SupervisorCommand.php#L97

but that makes no sense at all. let´s say I have 5 supervisors, each with max-processes of 50 and an empty queue.
so why should I start 250 workers to scale them immediately down to 5 (min-processes).

pardon, for me, a scaler should always start as low as possible and scale up ONLY if needed, not the other way around.

so the specific line of code I am referring to should be like this:

  $supervisor->scale(1);

with that, horizon only starts 1 worker for each supervisor and only scales up to "max-processes" if really needed.

you can argue: what´s the problem? you should never run more workers than your host is able to run. so in the worst case, you are running a few seconds on max load of your host before everything gets scaled down.

but for me, this is more a thing of "good principals", and don´t waste any resources if you don´t need to.
and some people may over-provision their queues. means: the host is on limit when running all queues on max,
but this usually never should happen. but with the current (auto) "scale" concept of horizon: when horizon starts,
the host is always on max load even when the queue is empty, and that makes no sense at all.

i would be happy if we could change that. but I have no idea where I have to address that issue.
i think another option in the horizon config like always_start_with_one => true could solve that issue easily.

what do you think ?

thanks,
micha

[henzeb]

I agree. Makes no sense.

[henzeb]

Not much going on here. Maybe @driesvints @nunomaduro or @taylorotwell can elaborate on this? is there a reason why we start horizon full throttled?

[michabbb]

looks like nobody is using horizon 🤷

[jrafael29]

i think, you can set autoScalingStrategy = size, on horizon config file, in supervisor's config

[michabbb]

as long this code exists:

https://github.com/laravel/horizon/blob/295b4bcb713a62eb86d83951e291f7481bc24a28/src/Console/SupervisorCommand.php#L96C32-L98

        $supervisor->scale(max(
            0, $this->option('max-processes') - $supervisor->totalSystemProcessCount()
        ));

nothing helps. because at the moment horizon is stating it scales to max and then scales down after, that´s the issue.
every 3 year old understands that this is nonsense - but mr lambo himself thinks it´s a great idea.... nothing more to say 🙄 🤷 😞

[fragkp]

Maybe create a PR instead of complaning like an idiot? nothing more to say

[michabbb]

@fragkp It is no shame to be illiterate, but to flaunt it like that is embarrassing.

[fragkp]

Your answer to me is the reason that your "PR" is not merged.

[michabbb]

@fragkp Thanks for confirming that you’re unable to read anything, but at least you enriched the world with your meaningful comments, congratulations. 👏

[ghost]

@michabbb I saw your PR.

A reason might be that when you deploy in full load, you don't want a hiccup exactly after the deploy process.

Correct me if I am wrong, but that formula from there assures the already working processes are subtracted from the max amount and only the difference is started, resulting in total = max number (that is already auto for me).

There are 3 cases here.

Max 10
Already working 1 to 9
start 9 to 1 more

Max 10
Already working 0
start 10 more

Max 10
Already Working 10
Start 0 new.

Based on this if you are fully loaded at deploy, then 0 are started (I am not familiar with how the rest will be started after the old container shuts down). This might be considered more important than this other case:

You are with empty queue and you will start the difference until full, then scale down.

Now if the difference is closer to max => smaller load.
If the difference is farther away from max => bigger load.

[ghost]

@michabbb #49633 (comment) FYI supervisord is not aware of signals. I found an alternative.

[michabbb]

@bulletproof-coding sorry for the late answer.

here´s a snippet from my horizon config:

'supervisor-3' => [
            'connection'          => 'redis',
            'queue'               => ['spool'],
            'balance'             => 'auto',
            'autoScalingStrategy' => 'size',
            'processes'           => 1,
            'numprocs'            => 1,
            'maxProcesses'        => 100,
            'balanceMaxShift'     => 10,
            'balanceCooldown'     => 1,
            'maxTime'             => 0,
            'maxJobs'             => 10000,
            'memory'              => 128,
            'tries'               => 5,
            'timeout'             => null,
            'nice'                => 0,
        ],

'environments' => [
        'production' => [
            'supervisor-3' => [
                'minProcesses' => 1,
            ],
        ],
    ],

and there is a reason I was forced to make this "hack" into my CI process:

#!/usr/bin/env bash

FILE_PATH="vendor/laravel/horizon/src/Console/SupervisorCommand.php"


# Check if the replacement has already been made
replacement_line="        \$supervisor->scale(1);"
current_line_97=$(sed -n '97p' $FILE_PATH)

if [[ "$current_line_97" == "$replacement_line" ]]; then
    echo "Replacement already made. Exiting without changes."
    exit 0
fi

# Define the expected lines of code
expected_lines="        \$supervisor->scale(max(
            0, \$this->option('max-processes') - \$supervisor->totalSystemProcessCount()
        ));"

# Extract lines 97 to 99 from the file
current_lines=$(sed -n '97,99p' $FILE_PATH)

# Check if the extracted lines match the expected lines
if [[ "$current_lines" == "$expected_lines" ]]; then
    # Perform the replacement
    sed -i '97,99c\        $supervisor->scale(1);' $FILE_PATH
    echo "Code replaced successfully!"
else
    echo "The current lines do not match the expected code. No changes were made."
    echo "Current Lines: $current_lines"
    exit 1
fi

because the default behavior (at least) I see is that without that change horizon starts 100 workers... waits....
and then finally notices that nothing is in the queue and scales down to 1.

i took me a while to realize that mess because whenever the ci runs the server had a super high load and i was wondering where this is coming from. and the reason is exactly this - which makes no sense at all. but nobody cares, even Mr laravel himself things that this is a great idea and there is nothing to change.

again: nothing more to say. 🙄

[ghost]

@michabbb Until we understand the mechanism behind Horizon and supervisord we can't ask for nothing from them.
Like I said, supervisord is not aware of SIGTERM (graceful shutdown) so, it will restart the worker before the container is killed, leading to partially processed jobs (which in the documentation are solved by the need of each job to be idempotent).

The code you are referring to is called in an artisan command. For me that is not clear when it happens or how the scale function handles those x number of processes sent to it.

Also it should be noted that the queue worker is a long lived php process that is affected, just like octane, by static and singletons, limiting the php language and possibly leading to unwanted behavior and hard to debug bugs. There is the --once flag that solves this by booting the app for each job.

Until you understand fully how it works, production usage is risky from my point of view. I know this from experience. Taking for good what the community approved is not 100% safe, especially in corner cases.

[michabbb]

@bulletproof-coding I've explained this multiple times; it requires modifying a single line of code. I'm unsure why you're overcomplicating things. My CI process will retain this single-line change as it functions correctly with the default Horizon setup. I don't see the relevance of your points to the issue: Horizon should only launch one worker initially, not 100. That's the entire problem.
But honestly, I gave up long ago and no longer care. I change one line, and the world is fine, at least for me. if the rest of the world thinks it´s a great concept to start tons of workers without having anything in the queue, I don´t care 😏