Y25-549 - Implement a `bunny` based consumer in `unified_warehouse` that does not rely on Warren

[dasunpubudumal]

User story

As PSD, we need to implement a message consumer layer within unified_warehouse so that we can decouple and sunset warren library from unified_warehouse.

Who are the primary contacts for this story
@dasunpubudumal

Who is the nominated tester for UAT

PSD

Acceptance criteria

Consider if these features can be featured flagged to decouple testing and deployment.
To be considered successful the solution must allow:

• Messages are successfully picked up by the consumer handlers using the correct parameters (e.g., routing keys, headers) and from correct sources.
• A "delayed queue" is employed to handle message ordering problems (e.g., if there is a foreign constraint violation, changes are that the producer haven't produced the necessary messages to record the row pertaining to the foreign key).
• After a pre-defined TTL expires, the delayed queue should route messages back to the original queue by configuring the original queue as a DLQ (Dead-Letter Queue)¹ for the delayed queue.
• Erroneous messages are dead-lettered properly.
• Message re-queueing behaviour depends on different exception criteria with a pre-defined limit; for example, for a ActiveRecord::RecordNotFound, a message could be delayed for an $n$ number of times and when the number of delays exceeds $n$, the consumer should dead-letter it. See app/warren/subscriber/multi_lims_warehouse_consumer.rb for how it currently works (we don't have the notion of a re-queueing limit as of now) in unified_warehouse.
• The message consumption process must be resilient to exceptions, ensuring that consumer threads continue running even if a message-related error occurs.

Dependencies

This story is blocked by the following dependencies:

• Y25-543 - Create the schemas in RedPanda registry in Avro format for the models used in unified_warehouse General-Backlog-Items#627
• Y25-550 - Integrate the avro gem and implement a fault-tolerant mixin for schema validation #825

References
This story has a non-blocking relationship with:

• #<issue_no.>
• sanger/#<issue_no.>

Additional context

Deployment notes

Add any details that should be noted when deploying this feature. For example, is a database migration needed, a specifc rake task or other script that needs to be run, a dependency on another application or service that needs to be released first etc?
These notes are intended to ensure the developers releasing the feature know what needs to be done.

Add any other context or screenshots about the feature request here.

Footnotes

1. For an example, see queue parameters for psd.multi-lims-warehouse-consumer.delay in ware-uat.psd.sanger.ac.uk. ↩