[Bug] follow is affected by large number of jobs

[alippai]

Describe the bug

When there is a large number of jobs (thousands), then the pueue follow TASK_ID periodically prints text regarding large pueue network messages.

I'm not sure it's about the follow or if it's an internal healthcheck or actually written to the logfile as well.

This might be obvious from the code, but also I'll be able to create a small example later.

Steps to reproduce

NA yet

Debug logs (if relevant)

No response

Operating system

Linux

Pueue version

v4.0.1

Additional context

No response

[Nukesor]

Could you provide some log output? From the description alone, I have no idea where to begin debugging :D

[alippai]

@Nukesor: :D
Sorry, I know it was brief, I just cleaned the jobs and didn't saw the message again.

Now I have it again:

pueue follow JOB_ID
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.

The message is printed every few seconds. I have a feeling because of the large number of jobs not cleared in pueue. It's not part of the log file, likely it's printed by the pueue follow command

[Nukesor]

Could you give some more information, like number of jobs, amount of log output?
Do you use read_local_logs: false?
Does the issue also appear when calling pueue status?
Is that a full follow or just a partial follow?

Is the log output you sent exactly what you see on the terminal output? I.e. you call follow and the warnings pop up before the first bit of log output can be seen?

Also, what's your usecase :D How do you end with that many tasks in your queue?