RedisBatchRepository - a redis driver for batch jobs instead of database table job_batches

[cyppe]

Hi!

Work in progress of a driver for storing batch jobs in Redis including atomic locks.

It's fully compatible with Horizon and pruning of jobs (php artisan queue:prune-batches --hours=1 --unfinished=1 --cancelled=1)

Im not a package developer so don't have the experience to make one. But I think this is something really uiseful as I had massive performance issues when running 50+ workers on batches where table lock could add ~4 seconds delay on each job and blocking workers from fetching next job.

This this implementation I cut time in a 1000 job batch with 50 workers from 100+ seconds to 15 seconds.

Again - still work in progess. But please take a look and share some feedback. And if anyone want to build a package of it and also improve it it would be awesome! Just ping me so I can follow along.

My work so far is in this Gist: https://gist.github.com/cyppe/3a099fb97273f5ac08c6bb215792b538

[cyppe]

Created a package so it's easier to use. https://github.com/cyppe/laravel-batch-jobs-redis-driver

[borjajimnz]

Hello @cyppe just tested this driver and works perfectly, thanks for sharing. cc @themsaid

[ercsctt]

This is awesome, would love to see this supported by default in the framework.

[patrickomeara]

I also think this would be a great addition to the framework. Batching currently makes three calls to the database per job, a read, select for update and an update. IOPS through the roof on big migrations.

[macropay-solutions]

Database queue driver is not for production, but only for development.

[bhushan]

is it part of laravel now ?

[bdteo]

I don't think it is part of Laravel yet. But it is a worthy addition for sure. The db based batching is really quite a hard bottleneck for many big projects based on Laravel.