Better error handling in base consumer implementation.

[hodgestar]

Currently vumi.service consumers don't handle errors very well. Errors should be logged , rejected and sent back to RabbitMQ for retrying (for a start).

[jerith]

Why does this need to return a deferred?

[hodgestar]

Ready for review. There are two equivalent implementations of _consume. One with inlineCallbacks one with out. I have a slight preference for the one with, @jerith prefers the one without. Other opinions welcome.

[jerith]

Since it turned out to be a little more convoluted than I'd thought to get rid of the @inlineCallbacks, I don't mind keeping it around for this.

[smn]

👍 from my side on this change. The only thing we may be overlooking is that previously the consume method could return False if the message was to be nackd, currently we'd need to raise an error. I don't think anything relies on that obscure behaviour, just noting it here for the odd-chance we see weird behaviour after it lands.

[smn]

What's stopping this from landing? Am I missing some major objections from people?

[hodgestar]

Although I like the errback support and plan to keep it, I think the real solution here is:

• Add a special exception that can be raised to indicate that a message should not be requeued.
• Catch that exception in the consumer.
• Add a configuration option for a dead-letter exchange.

Thoughts?

[hodgestar]

Modification (suggested by @jerith):

• Add support for not requeuing messages that the same worker has attempted to process N times in the past M minutes (where N and M are configurable and default to 2 and 1 respectively).

[hodgestar]

I've actually changed my mind on how this should be done initially. I think we should:

• dead-letter messages that result in uncaught exceptions straight away if the consumer has a dead-letter exchange (and requeue such messages if the consumer does not).
• add a tool for viewing dead-lettered exchanges, seeing queues and requeueing either individual messages or all messages in a queue back to their previous queue.
• for this PR I propose we write a small command-line tool and ticket an HTTP API and GUI for later.

[smn]

I think the existing rabbitmq management plugin already goes a long way to seeing exchanges, queues and has the ability to inspect queues & re-queue.