SQS queue with delay longer than 15 minutes

[V3ct0rek]

Hi,
Is there a need to create an additional feature for SQS integration, specifically for adding a delay longer than 15 minutes? This could be achieved using AWS Step Functions and Lambda. For instance, in the configuration, we could introduce an option like "enable_long_term_delay" or "enable_aws_delay." When set to true, this option would route all jobs first through a Step Function to accommodate the extended delay, and then add them to SQS via Lambda.

What advantages does it add?

-You can easily remove/abort long/short-term delayed jobs because you can just stop the Step Function from executing and nothing will be sent to SQS.
-Easy processes management - via the AWS panel, you can see all StateMachine-related processes so if you are attaching many jobs to one object, you can easily add an ID to the StateMachine and you will have all jobs in one place.

Do you find this approach useful?

[macropay-solutions]

You can implement delays longer than 15 min with subsequent dispatches using just php code.
Send in the message the total delay you need, for example 32 min. (and the start time).
When the job is processed, check if 32 minutes had passed already.
If not dispatch the a new job with same payload with delay = min (15, remaining difference).

Downside: using multiple messages for a single message, but sqs messages are cheap anyway.

[macropay-solutions]

On mysql does not work properly. On a messaging system like SQS is working fine.
Laravel requests a new message from SQS. SQS searches for an available message and marks it as not visible for 15 min (setup available in SQS). If in those 15 min laravel says I finished this job, SQS discards it. If laravel is not finishing the job, the message will be visible for other worker to pick it up after those 15 min.

[V3ct0rek]

That's normal sqs behavior, it should work exactly the same if you will dispatch batch on sqs connection and delay jobs in it for 32 or more minutes

[macropay-solutions]

Sqs has max 15 min delay. Only if you dispatch to your mysql and then a single worker on that mysql should dispatch from there (after 32 min) to sqs without delay, but bear in mind the 256 kb max message size.

[V3ct0rek]

256kb is a lot in context of jobs so I won't qualify it as a problem, and yeah, after these 32 minutes Laravel worker will execute code from batch and in this case push job to the sqs

[macropay-solutions]

Down side is that a single worker will be overwhelmed by big volume of data from mysql. Also the mysql server will be very busy doing queries in that batch table as a daemon.