Getting a lot of Upstream error messages

[FlaviuRadulescu]

Hi, we run amqproxy from the official container, and for un unknown reason from time to time we find the proxy with these type of errors.

We don't know why they are triggered. The RabbitMQ cluster is working fine, we have multiple servers connected to it. What could trigger this, why is recovering just at a restart, and how can we get more details related to it?

I would appreciate a fast support as it is affecting our production

amqproxy-1  | 2025-04-19T14:49:42.841616Z ERROR amq_proxy.client[remote_address: "172.18.0.3:44884"] Upstream error
amqproxy-1  | Error writing to upstream (AMQProxy::Upstream::WriteError)
amqproxy-1  |   from tmp/src/amqproxy/upstream.cr:155:9 in 'finish_publish'
amqproxy-1  |   from tmp/src/amqproxy/client.cr:88:11 in 'read_loop'
amqproxy-1  |   from tmp/src/amqproxy/server.cr:90:7 in '->'
amqproxy-1  |   from usr/share/crystal/src/fiber.cr:143:11 in 'run'
amqproxy-1  |   from ???
amqproxy-1  | Caused by: Closed stream (IO::Error)
amqproxy-1  |   from usr/share/crystal/src/slice.cr:256:5 in 'write'
amqproxy-1  |   from tmp/lib/amq-protocol/src/amq/protocol/frames.cr:18:9 in 'to_io'
amqproxy-1  |   from usr/share/crystal/src/io.cr:914:5 in 'finish_publish'
amqproxy-1  |   from tmp/src/amqproxy/client.cr:88:11 in 'read_loop'
amqproxy-1  |   from tmp/src/amqproxy/server.cr:90:7 in '->'
amqproxy-1  |   from usr/share/crystal/src/fiber.cr:143:11 in 'run'
amqproxy-1  |   from ???

[viktorerlingsson]

Hi @FlaviuRadulescu, I haven't been able to reproduce this. Can you please provide some more information on how to reproduce?

• When does the error occur?
• amqproxy version
• RabbitMQ version
• RabbitMQ logs from when the error occurs
• Queue type
• Are you using any plugins in RabbitMQ?
• Any other relevant app/client information

[johanrhodin]

Also if you are hosting with CloudAMQP send us a support ticket (support@cloudamqp.com) so that we can see the server logs & behavior too.

[kochen]

we recently encountered the same issue and after some digging, we found that:

• publisher application was trying to publish a message to rabbitmq, connected through amqproxy
• amqproxy failed to connect to rabbitmq due to a temporary dns issue
• rabbitmq doesn't even know there was an attempt to connect
• amqproxy threw an error: Hostname lookup for rabbitmq.xxx.yyy failed: Name has no usable address (AMQProxy::Upstream::Error)
• this caused connection closed unexpectedly (and further exceptions) on the publisher application side

the expected behaviour would be that amqproxy retries the connection and the client doesn't "feel" anything happened...

this issue does not seem to be related to rabbitmq, or the client application.

[kochen]

This seems to happen again, even without the dns issue mentioned above:
INFO amq_proxy.upstream[upstream: "127.0.0.1:5672"] Connection error #<IO::EOFError:End of file reached>

(this also shouldn't have the INFO log level, right?)

RabbitMQ logs, do not show any disconnection and/or errors around the time amqproxy has shown the EOFError...

[viktorerlingsson]

Thanks for the extra information, that's very helpful!

I will try to work on this next week.

[FlaviuRadulescu]

Thanks for the extra information, that's very helpful!

I will try to work on this next week.

Hi, any update on this? We are randomly seeing the above errors and the only options we have is to restart the docker container. This is happening randomly and not all the servers in the same time (servers being in the same data center). What do you think could be the problem. It is very annoying because the underlying application is not aware that the proxy is down and pushing messages into a black hole... so the proxy consuming messages without erroring is very bad

[viktorerlingsson]

Thanks for the extra information, that's very helpful!
I will try to work on this next week.

Hi, any update on this? We are randomly seeing the above errors and the only options we have is to restart the docker container. This is happening randomly and not all the servers in the same time (servers being in the same data center). What do you think could be the problem. It is very annoying because the underlying application is not aware that the proxy is down and pushing messages into a black hole... so the proxy consuming messages without erroring is very bad

@FlaviuRadulescu I'm a little swamped at the moment, but I'll try to look into this later this week!

[FlaviuRadulescu]

not sure this is the same or a different error

2025-06-03T10:25:35.610209Z ERROR amq_proxy.client[remote_address: "172.18.0.4:53086"] Upstream error
Error writing to upstream (AMQProxy::Upstream::WriteError)
  from usr/src/amqproxy/src/amqproxy/upstream.cr:155:9 in 'send'
  from usr/src/amqproxy/src/amqproxy/upstream.cr:146:7 in 'write'
  from usr/src/amqproxy/src/amqproxy/records.cr:8:3 in 'write'
  from usr/src/amqproxy/src/amqproxy/client.cr:46:9 in 'finish_publish'
  from usr/src/amqproxy/src/amqproxy/client.cr:88:11 in 'read_loop'
  from usr/src/amqproxy/src/amqproxy/client.cr:57:5 in 'read_loop'
  from usr/src/amqproxy/src/amqproxy/server.cr:79:9 in 'handle_connection'
  from usr/src/amqproxy/src/amqproxy/server.cr:49:11 in '->'
  from usr/share/crystal/src/fiber.cr:143:11 in 'run'
  from usr/share/crystal/src/fiber.cr:95:34 in '->'
  from ???
Caused by: Closed stream (IO::Error)
  from usr/share/crystal/src/io.cr:121:5 in 'check_open'
  from usr/share/crystal/src/io/buffered.cr:142:15 in 'write'
  from usr/src/amqproxy/lib/amq-protocol/src/amq/protocol/frames.cr:18:9 in 'to_io'
  from usr/share/crystal/src/io.cr:914:5 in 'write_bytes'
  from usr/src/amqproxy/src/amqproxy/upstream.cr:151:9 in 'send'
  from usr/src/amqproxy/src/amqproxy/upstream.cr:146:7 in 'write'
  from usr/src/amqproxy/src/amqproxy/records.cr:8:3 in 'write'
  from usr/src/amqproxy/src/amqproxy/client.cr:46:9 in 'finish_publish'
  from usr/src/amqproxy/src/amqproxy/client.cr:88:11 in 'read_loop'
  from usr/src/amqproxy/src/amqproxy/client.cr:57:5 in 'read_loop'
  from usr/src/amqproxy/src/amqproxy/server.cr:79:9 in 'handle_connection'
  from usr/src/amqproxy/src/amqproxy/server.cr:49:11 in '->'
  from usr/share/crystal/src/fiber.cr:143:11 in 'run'
  from usr/share/crystal/src/fiber.cr:95:34 in '->'
  from ???

what we see is that some processes are going through the proxy but others are failing with this error. Wonder if there is some kind of multiplexing error

[kickster97]

Hi!
Could you clarify if your client attempts to reconnect to the proxy after encountering these IO::Error messages?

I have not been able to reproduce your issue, it works as expected for me.

What I do is:

1. start RabbitMQ, start Amqproxy, start a publishing client that publishes to Amqproxy.
2. turn off RabbitMQ -> Amqproxy fails with an IO::Error, same as you show here.
   (My publishing client has a reconnect loop, so amqproxy also loops the logs i added below)
3. I turn on RabbitMQ again -> Amqproxy reconnects and starts working again.

2025-06-04T09:20:30.621573Z ERROR amq_proxy.client[client: "127.0.0.1:57900"] Upstream error
Error connecting to 'localhost:5672': Connection refused (AMQProxy::Upstream::Error)
  from src/amqproxy/channel_pool.cr:50:7 in 'add_upstream'
  from src/amqproxy/channel_pool.cr:32:13 in 'get'
  from src/amqproxy/client.cr:74:30 in 'read_loop'
  from src/amqproxy/client.cr:57:5 in 'read_loop'
  from src/amqproxy/server.cr:79:9 in 'handle_connection'
  from src/amqproxy/server.cr:49:11 in '->'
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/fiber.cr:170:11 in 'run'
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/fiber.cr:105:3 in '->'
Caused by: Error connecting to 'localhost:5672': Connection refused (Socket::ConnectError)
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/socket/addrinfo.cr:87:9 in 'initialize'
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/socket/tcp_socket.cr:29:3 in 'initialize'
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/socket/tcp_socket.cr:29:3 in 'new'
  from src/amqproxy/upstream.cr:20:7 in 'initialize'
  from src/amqproxy/upstream.cr:19:5 in 'new'
  from src/amqproxy/channel_pool.cr:39:18 in 'add_upstream'
  from src/amqproxy/channel_pool.cr:32:13 in 'get'
  from src/amqproxy/client.cr:74:30 in 'read_loop'
  from src/amqproxy/client.cr:57:5 in 'read_loop'
  from src/amqproxy/server.cr:79:9 in 'handle_connection'
  from src/amqproxy/server.cr:49:11 in '->'
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/fiber.cr:170:11 in 'run'
  from /opt/homebrew/Cellar/crystal/1.16.2/share/crystal/src/fiber.cr:105:3 in '->'

The upstream connection itself is only triggered when a channel is opened. If an upstream's read loop terminates, it closes all associated client channels. There's no reason for the upstream to reconnect unless a client has initiated a reconnection.

I'm interested in if you get logs that indicate the add_upstream is being run or not.

[FlaviuRadulescu]

In our setup, we have long-lived processes connected to amqproxy. Randomly, when some of these processes push messages to amqproxy, the messages never reach RabbitMQ. However, other processes connected to the same amqproxy continue working normally. The processes experiencing the issue do not receive any error notifications from amqproxy; they push messages as usual, but the messages are never delivered to RabbitMQ.