Service upgrades lead to corrupted Consumer state #73
Description
While deploying new versions with AWS ECS there are 2 instances running for some time. After that the consumer often gets "stuck" and does not process any new events. No info or feedback in logs about the reasons. Do not have logs of what happened during the update, but when I switched to RaiseErrorHandler I at least got this message in the logs:
ERROR Task exception was never retrieved
future: <Task finished name='Task-17'
coro=<PollingService.execute() done, defined at
/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.1
3/site-packages/logicblocks/event/processing/services/polling.py:21
> exception=ValueError('Cannot deserialise {"state": null,
"last_sequence_number": 653} to type <class
\'logicblocks.event.processing.consumers.state.EventConsumerState\'
>.')>
Traceback (most recent call last):
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/services/polling.py", line 23, in execute
await self._callable()
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/services/error.py", line 246, in execute
raise exception
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/services/error.py", line 241, in execute
return await self._do_execute()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/services/error.py", line 273, in _do_execute
return await self._callable()
^^^^^^^^^^^^^^^^^^^^^^
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/consumers/subscription.py", line 143, in consume_all
await delegate.consume_all()
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/consumers/source.py", line 37, in consume_all
state = await self._state_store.load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/consumers/state.py", line 146, in load
self._states[partition] = EventConsumerState.deserialise(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
event.payload
^^^^^^^^^^^^^
)
^
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/processing/consumers/state.py", line 32, in deserialise
return fallback(cls, value)
File "/srv/.poetry/some-service-2uZo5AhP-py3.13/lib/python3.13/site-packages/logicblocks/event/types/conversion.py", line 37, in raising_deserialisation_fallback
raise ValueError(f"Cannot deserialise {value} to type {klass}.")
ValueError: Cannot deserialise {"state": null, "last_sequence_number": 653} to type <class 'logicblocks.event.processing.consumers.state.EventConsumerState'>.