Consumer failed to restart on 404 error

[jeremycline]

Friday evening some consumers crashed with something like the following and did not restart:

[2026-03-06 16:07:09,599 fedora_messaging.twisted.consumer WARNING] The connection to the broker was lost (ConnectionDone()), consumer halted; the connection should restart and consuming will resume.
[2026-03-06 16:07:09,599 twisted INFO] <twisted.internet.tcp.Connector instance at 0x7fcee8529010 disconnected IPv4Address(type='TCP', host='rabbitmq.fedoraproject.org', port=5671)> will retry in 2 seconds
[2026-03-06 16:07:09,599 twisted INFO] Stopping factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.fedoraproject.org port=5671 virtual_host=/pubsub ssl=True>, confirms=True)
[2026-03-06 16:07:09,599 fedora_messaging.twisted.protocol INFO] Disconnect requested, but AMQP connection already gone
[2026-03-06 16:07:09,599 twisted INFO] <twisted.internet.tcp.Connector instance at 0x7fcee8529010 disconnected IPv4Address(type='TCP', host='rabbitmq.fedoraproject.org', port=5671)> will retry in 2 seconds
[2026-03-06 16:07:09,599 twisted INFO] Stopping factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.fedoraproject.org port=5671 virtual_host=/pubsub ssl=True>, confirms=True)
[2026-03-06 16:07:12,063 twisted INFO] Starting factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.fedoraproject.org port=5671 virtual_host=/pubsub ssl=True>, confirms=True)
[2026-03-06 16:07:12,063 twisted INFO] Starting factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.fedoraproject.org port=5671 virtual_host=/pubsub ssl=True>, confirms=True)
[2026-03-06 16:07:12,073 fedora_messaging.twisted.factory INFO] Re-registering the Consumer(queue=cloud-image-uploader-aws, callback=<fedora_image_uploader.handler.Uploader object at 0x7fcef034b4d0>) consumer
[2026-03-06 16:07:12,075 pika.channel WARNING] Received remote Channel.Close (404): "NOT_FOUND - queue 'cloud-image-uploader-aws' in vhost '/pubsub' process is stopped by supervisor" on <Channel number=2 OPEN conn=<pika.adapters.twisted_connection._TwistedConnectionAdapter object at 0x7fcee97f3390>>
[2026-03-06 16:07:12,076 fedora_messaging.twisted.factory ERROR] The connection failed with an unexpected exception; please report this bug: Traceback (most recent call last):
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/factory.py", line 132, in on_ready_connection_errback
    r = failure.trap(
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 431, in trap
    self.raiseException()
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 455, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/factory.py", line 124, in on_ready
    queue_name: str = yield client.declare_queue(record.queue)
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/protocol.py", line 346, in declare_queue
    names = yield self.declare_queues([queue])
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/protocol.py", line 329, in declare_queues
    raise BadDeclaration("queue", args, e) from e
fedora_messaging.exceptions.BadDeclaration: Unable to declare the queue object ({'queue': 'cloud-image-uploader-aws', 'durable': True, 'auto_delete': False, 'exclusive': False, 'arguments': {}, 'passive': True}) because (404, "NOT_FOUND - queue 'cloud-image-uploader-aws' in vhost '/pubsub' process is stopped by supervisor")

--- Logging error ---
Traceback (most recent call last):
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/protocol.py", line 326, in declare_queues
    yield channel.queue_declare(**args)
pika.exceptions.ChannelClosedByBroker: (404, "NOT_FOUND - queue 'cloud-image-uploader-aws' in vhost '/pubsub' process is stopped by supervisor")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
                     ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
        current.result, *args, **kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/factory.py", line 132, in on_ready_connection_errback
    r = failure.trap(
        pika.exceptions.AMQPConnectionError,
        error.ConnectionDone,
        error.ConnectionLost,
    )
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 431, in trap
    self.raiseException()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 455, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/factory.py", line 124, in on_ready
    queue_name: str = yield client.declare_queue(record.queue)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/protocol.py", line 346, in declare_queue
    names = yield self.declare_queues([queue])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/image-uploader/venv/lib64/python3.14/site-packages/fedora_messaging/twisted/protocol.py", line 329, in declare_queues
    raise BadDeclaration("queue", args, e) from e
fedora_messaging.exceptions.BadDeclaration: Unable to declare the queue object ({'queue': 'cloud-image-uploader-aws', 'durable': True, 'auto_delete': False, 'exclusive': False, 'arguments': {}, 'passive': True}) because (404, "NOT_FOUND - queue 'cloud-image-uploader-aws' in vhost '/pubsub' process is stopped by supervisor")

I'm not sure what happened on the broker side. The end result was that the consumer process didn't exit and didn't continue consuming.