Laravel supervisor queue jobs randomly stuck after a few hours

[1212087]

• Laravel Version: 7.6.2
• PHP Version: 7.4-fpm-alpine
• Database: MYSQL 8.0
• Queue Driver: Redis 6.0-rc3-alpine

Description

I'm developing a system that run a lot of jobs per day (about more than 300k jobs / day). This is my structure:

• Job A processed
• -> Job A fire an event (ex: JobADoneEvent) that it was processed, and pass job result to the event
• -> A listener catches JobADoneEvent, and dispatches job B (with data that passed from JobADoneEvent)
• Job B processed
• -> Job B fires an event (ex: JobBDoneEvent) that it was processed, and pass job result to the event
• -> A listener catches JobBDoneEvent, and dispatches job C (with data that passed from JobBDoneEvent)
• Job C processed
• -> Job C fires an event (ex: JobCDoneEvent) that it was processed, and pass job result to the event
• -> A listener catches JobCDoneEvent, and dispatches job D (with data that passed from JobCDoneEvent)
• Job D, E, F are the same as above

Each job has a queue handle that configured at supervisord.conf like this

[program:laravel-worker-JobA]
process_name=%(program_name)s_%(process_num)02d
command=/usr/local/bin/php /home/projects/artisan queue:work --queue=JobA --sleep=3 --tries=3 --timeout=60
numprocs=10
autostart=true
autorestart=true
user=nhoxtwi
redirect_stderr=true
startsecs=0

Job A call API to another host, then fire an event with responded data, Job B catch this data and insert/update them to DB, Job C catch Job B result then insert / update to DB (to another table)...

When I start docker container, everything was running fine without any error, but after some time (a few hours later), a random job was stuck without any exception (no log, no exception, no timeout error), the job is just displayed as "Processing job B (or C, or D...)" forever until I restart docker container manually. Because of stuck job, supervisor process was also stuck, that resulted in thousands of Job in the queue without any process to handle them (because all processes were stuck in previous jobs)
This is an example job:

    public $products;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(array $products)
    {
        $this->products = $products;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle(UpdateProductsFromCrawler $updateProductsFromCrawler)
    {
        $jobId = $this->job->getJobId();
        try {
            Log::info("Processing $jobId");
            $updateProductsFromCrawler->execute($this->products, $jobId);
            Log::info("Processed $jobId");
        } catch (PDOException $th) {
            Log::info("Retrying $jobId");
            self::dispatch($this->products)->onQueue('retryJobs');
        }
    }

And this is UpdateProductsFromCrawler execute function:

public function execute(array $products = [], string $jobId = '')
    {
        $this->logInfo("Step 1 $jobId");
        $productsForBulkInsert = [];
        $brands                = $this->getBrandsAndCache();
        $categories            = $this->getCategoriesAndCache();
        $this->logInfo("Step 2 $jobId");
        foreach ($products as $item) {
            $brand                                 = $this->firstOrInsertItemBrand($item, $brands);
            $product                               = $this->prepareProduct($item, $brand);
            $category                              = $categories[$item->catid];
            $product['category_id']                = $category->id;
            $productsForBulkInsert[$product['id']] = $product;
        }
        if (empty($productsForBulkInsert)) {
            return;
        }
        ksort($productsForBulkInsert);
        $this->logInfo("Step 3 $jobId"); # Stuck at this step
        DB::beginTransaction();
        try {
            DB::table('products')
                ->whereIn('id', array_keys($productsForBulkInsert))
                ->lockForUpdate()
                ->get(['id']);
            Product::bulkInsertOrUpdate($productsForBulkInsert, [
                'name',
                'slug',
                'site_url',
                'image',
                'rating_star',
                'rating_count',
                'like_count',
                'price',
                'discount',
                'price_before_discount',
                'shop_location',
                'is_verified',
                'is_official_shop',
                'stock_quantity',
                'sold_quantity',
                'updated_at'
            ]);
            DB::commit();
            $this->logInfo("Step 4 $jobId");
            event(new ProductsUpdated($productsForBulkInsert));
            $this->logInfo("Step 5 $jobId");
        } catch (PDOException $th) {
            DB::rollback();
            throw $th;
        } catch (\Throwable $th) {
            $this->logInfo('Error when updating products from crawler');
            $this->logWhenError($th);
        }
    }

As you can see, the job is stuck at DB::beginTransaction(); to the end of the script, I don't know why, because a lot of same jobs were processed successfully before stuck randomly (it could be stuck at job A, job C, D...or any job except Job A)
I also has a schedule command that restart queue after 15 minutes, but not work, queue jobs are still be stuck randomly:

$schedule->command('queue:restart')->everyFifteenMinutes();

Thanks for reading my long post, I'm glad to hear any news or direction from you. I tried to describe more detail but I'm not good at English, so please tell me if you need any further information. Please help

[1212087]

$this->logInfo("Step 3 $jobId"); # Stuck at this step

Now I'm not sure if the job stuck at this step or not :(

[shoemoney]

Maybe memory??

I gave up with supervisor for production... pm2 is the way!!!