feat: Support redis serverless (Cluster Mode)

[lordsoffallen]

Currently it's not possible to use redis serverless with prefect following this setup here: https://docs.prefect.io/v3/advanced/self-hosted

This PR adds support for redis cluster mode so it can work on aws serverless mode. These changes were needed to make sure background services can communicate to prefect api server via redis message broker. (hence the changes in messaging and ordering files)

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[zzstoatzz]

hi @lordsoffallen - can you update the PR title and body to explain what this PR intends to add and what the motivation is?

for example, in your comments you mention cluster mode but it seems that several other things would need to be updated for cluster mode in general

[lordsoffallen]

hi @lordsoffallen - can you update the PR title and body to explain what this PR intends to add and what the motivation is?

for example, in your comments you mention cluster mode but it seems that several other things would need to be updated for cluster mode in general

Hi there,

updated. Yes i add redis cluster support so aws redis serverless would work. That's what i have in my setup in the moment and it works afaik.

Reference origin: #16299 (comment)

[zzstoatzz]

updated. Yes i add redis cluster support so aws redis serverless would work. That's what i have in my setup in the moment and it works afaik.

sorry, @lordsoffallen I'm asking if you can please update the PR body with an explanation for why the specific changes you made are necessary. ie "im adding hash tags for pipeline operations" etc

i gather that you're trying to trying to support redis serverless in general, but this PR does not appear to address all that would be needed for full cluster mode support

i may not be thinking of everything here, but at least it seems we'd need:

• to solve this upstream issue in docket: Support for redis-cluster chrisguidry/docket#120
• (in several places) to use the RedisCluster specific client when appropriate (handles redirects)

[lordsoffallen]

Hi @zzstoatzz,

thanks, regarding the changes: Prior to this, i get redis cluster errors and for messaging these were the needed changes (with claude's help). Right now it works for me, i didn't need further changes but before this i couldn't get prefect background workers running. So I thought i can push them upstream.

I am not too familiar with docket so i can't say what's needed so feel free to push changes or close this PR if there are a lot more other changes needed.

[github-actions]

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.

[lordsoffallen]

Hi @zzstoatzz

I see you guys made updates to docket and so, I am guessing support for serverless coming soon? Any updates on this? 😄