[Issue]: FailedNotifications keep retrying forever #155
Description
Describe the Issue
When a webhook notification fails to apply, it is saved into the notification_info_entity table.
What we are seeing is that the retry_counter is always 1
This seems to be for a couple of scenarios we know about so far:
- Hyperwallet has protection on their API to avoid use, since the connector tries all retries in quick succession - we sometimes see 429 errors when calling the Hyperwallet API
- The webhook does not exist in Hyperwallet -> we have some webhooks that 404 when calling Hyperwallet for some reason (it may be that they no longer exist?)
- It looks like genuine retries still fail and don't increment the counter (unknown on this one at the moment - as we cant access the prod file on the fly due to no H2 console and exclusive connect mode)
Environment
Live/Production
Version
5.2.0
Expected Behavior
Retries increment when they fail, regardless of error / response code fail.
Actual Behavior
All retry counters are one for all webhooks/notifications in notification_info_entity and the retry jobs (that run every 15 minutes) continually retry the same webhooks and they never expire.
Steps to Reproduce
Add an entry into the notification_info_entity table (of a webhook that doesn't exist in the hyperwallet side)
Observe that retry jobs retry, fail and never increment the counters.
Pre-conditions
No response