clarification or documentation about channel vs connection events

[soryy708]

From experimenting with amqplib@0.10.3 I noticed some undocumented behavior observing channel vs connection events. I don't know if I can rely on this behavior remaining consistent in future patch versions of amqplib, because it's currently undocumented.

It appears that observing channel.on('error', ...) changes the behavior of connection.on('error', ...), which also interacts with channel.on('close', ...)/connection.on('close', ...).
If this is an implementation detail, I'd like clarification in this issue. If it's intended as stable behavior, I believe this should be documented in the amqplib docs.

---

Observations:

1. Given
   connection.on('error', ...) and connection.on('close', ...)
   (and no observers on channel)

   Making a "soft" error (as defined by RabbitMQ docs) triggers both connection.on('error', ...) and connection.on('close', ...)

2. Given
   connection.on('error', ...), connection.on('close', ...), and channel.on('error', ...)

   Making a "soft" error triggers channel.on('error', ...), but neither connection.on('error', ...) or connection.on('close', ...)

3. Given
   connection.on('error', ...), connection.on('close', ...), and channel.on('close', ...)

   Making a "soft" error triggers connection.on('error', ...), channel.on('close', ...), and connection.on('close', ...)

4. Given
   connection.on('error', ...), connection.on('close', ...), channel.on('error', ...), and channel.on('close', ...)

   Making a "soft" error triggers channel.on('error', ...), channel.on('close', ...), but neither connection.on('error', ...) or connection.on('close', ...)

In tabular form:

---

This undocumented nuance is a problem in the following example situation:

Suppose the user has:

• connection.on('error', logConnectionError)
• connection.on('close', reconnectAndAcquireChannel)

So when a "soft" error occurs, it's logged, the connection is closed, and re-acquired. The system self-heals.

Later the user wants more granular logging, so the user adds also:

• channel.on('error', logChannelError)

Now when a "soft" error occurs, it's logged, the channel closes, but the connection remains open. Re-acquire isn't run, so the system is in a pathologic state.

[cressie176]

Hi @soryy708,

Thank you for putting this all together so clearly. I agree the the behaviour should be documented. I'll take some time to reproduce what you've done and confirm the behaviour.

This undocumented nuance is a problem in the following example situation:

The approach I've taken with Rascal is to

1. Listen for channel both error & close events, then attempt recovery using the existing connection.
2. Listen for connection error & close events, then attempt recovery using a new connection.

There are some complications though - it was at one point possible to get both an error and close event, and even multiple error events (I'm not 100% this is still the case), so the code should only recover once. However, you can't simply use or unbind the listeners because an unhandled error event will crash the node application.

My preferred approach since writing rascal is to add error and close event handlers which publish a third lost event. I bind the recovery handler to the lost event using emitter.once so it is only triggered once, no matter how many close or error events are triggered. I never unbind the error and close event handlers so a repeat error event won't crash the application.

[cressie176]

Sorry this has taken a while. Here are the results from my testing...

CONNECTION CLOSE ON	CONNECTION ERROR ON	CHANNEL CLOSE ON	CHANNEL ERROR ON	CONNECTION CLOSE FIRED	CONNECTION ERROR FIRED	CHANNEL CLOSE FIRED	CHANNEL ERROR FIRED	ERROR THROWN	ERROR UNHANDLED
FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	TRUE
TRUE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	TRUE
FALSE	TRUE	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE	TRUE	FALSE
FALSE	FALSE	TRUE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	TRUE
FALSE	FALSE	FALSE	TRUE	FALSE	FALSE	FALSE	TRUE	TRUE	FALSE
TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	FALSE	FALSE	TRUE	FALSE
TRUE	FALSE	TRUE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	TRUE
TRUE	FALSE	FALSE	TRUE	FALSE	FALSE	FALSE	TRUE	TRUE	FALSE
FALSE	TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	FALSE	TRUE	FALSE
FALSE	TRUE	FALSE	TRUE	FALSE	FALSE	FALSE	TRUE	TRUE	FALSE
FALSE	FALSE	TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	TRUE	FALSE
TRUE	TRUE	TRUE	FALSE	TRUE	TRUE	TRUE	FALSE	TRUE	FALSE
TRUE	TRUE	FALSE	TRUE	FALSE	FALSE	FALSE	TRUE	TRUE	FALSE
TRUE	FALSE	TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	TRUE	FALSE
FALSE	TRUE	TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	TRUE	FALSE
TRUE	TRUE	TRUE	TRUE	FALSE	FALSE	TRUE	TRUE	TRUE	FALSE

[cressie176]

Perhaps more clearer as follows...

[cressie176]

In the event of a soft error (e.g. checking for a queue that doesn't exist)...

1. The channel will emit both an error event and close event
2. If the channel error event is not handled, it will propagate to the connection, which will emit both a connection error event and connection close event
3. If the connection error event is not handled, it will propagate to the process as an unhandled error event

I think this behaviour is reasonable. You can recover from soft errors by obtaining a new channel, but if you don't handle them, the connection will break

[soryy708]

Yes that makes sense.

How would it look like for non-soft errors? And, is there a list somewhere of what are soft and what aren't?

[cressie176]

I'm not sure it's useful to differentiate between "hard" and "soft" errors. It makes more sense to think in terms of channel errors and connection errors.

• Any time the channel emits an error means that it's broken and will need to be replaced.
• Any time the connection emits an error (assuming you are handling channel errors), means the it's broken, and will need to be re-established, plus all channels will need to be replaced.