Modify workflow service logic to fetch and process SQS events #8101
Description
Summary
We want to use our new SQS client to fetch SQS events from a queue, pass them off to the event handler logic and handle the result. As part of this, we'll need to convert the SQS message from JSON into our pydantic model for events. If this errors, we'll need to recover gracefully, but can't process that event and should leave it alone (for now, might change that approach in the future to delete, but keep it simple for the moment).
| Scenario | Behavior |
|---|---|
| Event processed successfully | Event is deleted from SQS queue |
| A non-retryable error is raised | Event is deleted from SQS queue |
| A retryable error is raised | Event is NOT deleted from SQS queue |
| Any other exception | Event is NOT deleted from SQS queue + log an error |
All scenarios should have logging to make clear which scenario occurred. A log message should exist that we can easily use to make a metric from (says which scenario it hit for all 4).
If a retryable/general exception occurs, we should do a rollback, this is so that if any error did occur partially through updating the DB, we don't keep that state.
Acceptance criteria
- Convert the SQS message into our event format with Pydantic models - if this errors, log an error, don't delete the message.
- 4 scenarios above are covered and tested