Failed AMQP handshake with TLS

[trevorperrin]

I'm implementing a client which connects via TLS (using the software versions listed below). During testing, the TLS handshake succeeds, but the AMQP portion of the handshake fails. While attempting to establish the connection, the AMQP header ("AMQP0091") is successfully sent by the client, and the server replies with a Connection.Start message. At this point, the client is supposed to reply with a Connection.Start-Ok message, but it does not do so.

The data is being generated (connectionstartframe.h), and the connection is set to send the ConnectionStartOKFrame. I followed the path through the following:

ConnectionStartFrame::process
ConnectionImpl::send
PassthroughBuffer::flush
TcpConnection::onData
SslConnected::send

In SslConnected::send, _handler->monitor is called with (readable | writable), but the write never occurs and the connection times out.

The workaround I've used is to call write in the SslConnected::send function, right before the call to _handler->monitor:

virtual void send(const char *buffer, size_t size) override
{
...
Monitor monitor(this);
write(monitor); // flush(monitor) works as well
// comment...
_handler->monitor(_connection, _socket, readable | writable);
}

This "works"; the AMQP handshake succeeds, and data is sent and received in the client via TLS. I don't like this solution, as I assume the Monitor should be tracking the Connection object, not SslConnected. I'm also concerned about edge cases around the handling of the output buffer, and SSL error messages like SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE. I have a solution that adds a monitor object to the send call of every TcpState object, in the event that any of those state objects need to monitor the connection. If this sounds reasonable, I can issue a pull request.

What I'm using:
OS Version: CentOS 7.4
AMQP-CPP Version: AMQP-CPP Master (as of April 2, 2018)
openssl Version: 1.0.2k
RabbitMQ Version: 3.7.2
Event Handler: libboostasio.h

[EmielBruijntjes]

Have you tested this with the a libev-handler as well? In examples/libev.cpp you can find an almost working example, you just have to uncomment a couple of lines and change the address.

You mentioned that _handler->monitor() is called with readable|writable, but that this writable event never occurs. To me that looks like an error of this boost asio handler that is supposed to let amqp-cpp know when the socket becomes writable.

[trevorperrin]

I modified the client to use LibEvHandler instead of LibBoostAsioHandler, and it also failed - though the failure mode was different. In sending data from the client for say, 2 minutes, the client buffer appears to grow without sending anything over the wire. The RabbitMQ Management portal also shows no incoming or outgoing data. Once the client stops issuing send calls, the buffer starts to clear through the event handler, and the RabbitMQ Management portal shows publish/deliver statistics for the data being sent.

Some observations:

1. I was able to get this same behavior out of the ASIO handler when I played around with the code in SslConnected (modifying items like state or the readable/writable options).
2. LibEvHandler appears to work well over TLS when I add the write (or flush) function to SslConnected::Send.
3. LibBoostAsioHandler works great on unsecured connections, without modifications to the code. This uses TcpConnected instead of SslConnected.
4. Analyzing TcpConnected in relation to the rest of the architecture, it appears there are two code paths that send data over the wire; one with an empty buffer that sends immediately using ::send, and one with an already populated buffer that uses sendmsg in the buffer object. This second path appears to use the event handler, while the first appears to bypass it and send directly (unless there is already something in the buffer or the immediate send failed). Comments in the code suggest this to be the case (passthroughbuffer.h, and tcpconnected.h in the send function).

The issue appears to be that SslConnected isn't using this two part send architecture. Once an immediate write/flush call is issued in the SslConnected::send function, it behaves just like non-secured connections and everything seems to work with either event handler.

[EmielBruijntjes]

I made a couple of changes to the SslConnected class. I'm not sure if it fixes your problem, but it does fix a couple of issues that I found, and it optimizes a thing or two by not going back to the event loop all the time. Can you check if this solves your problem too?

[trevorperrin]

The new changes did not resolve my issue. I ran through a series of tests (6 test cases: 3 for each handler). In the write(monitor, <readable>) cases, I am adding the following into SslConnected::send as previously mentioned:

Monitor monitor(this);
write(monitor, <readable>);
... comment
_handler->monitor(_connection, _socket, readable | writable);

Here are the results I'm seeing:

With the LibBoostAsioHandler:

1. No modifications to the code: The client exhibits the same behavior as before. The AMQP handshake fails in the same location in the process (no Connection.Start-Ok message).
2. With write(monitor, true): The AMQP handshake gets further along in the process. The Connection.Start-Ok message is successfully sent by the client. The server sends the Connection.Tune message, and the client responds with Connection.Tune-Ok. The client is now supposed to send the Connection.Open message, but it does not. The connection then times out from the server side.
3. With write(monitor, false): The client successfully sends data to the server.

With the LibEvHandler:

1. No modifications to the code: The client exhibits the same behavior as before. The AMQP handshake succeeds, and the client queues up data in its buffer (with no data transmission appearing in the RabbitMQ Management portal) until it stops issuing send requests. Then it flushes the buffer and data is transmitted to the server.
2. With write(monitor, true): This behaves the same way as the LibBoostAsioHandler test case 2 above. The Connection.Open message does not get sent, and the connection times out from the server side.
3. With write(monitor, false): The client successfully sends data to the server.

[EmielBruijntjes]

What is your test case?

[trevorperrin]

The test cases I've been running are using a rather large proprietary test harness, using multiple queues, and has the ability to swap in different handlers. It has been working well with unsecured connections using the LibBoostAsioHandler. I have submitted a simplified version of the ASIO test case to the following repository.

https://github.com/trevorperrin/AMQP-TEST

Running this without modifications to the SslConnected implementation, you'll see that the AMQP handshake does not complete (in the way specified in the initial comment).

[EmielBruijntjes]

I have modified the libev.cpp example in the example/ directory to also use a timer to publish messages. With this handler I cannot reproduce the problem. Can you?

[trevorperrin]

I did not reproduce the issue with your new handler, and I had created a similar test based on a timer which also successfully sent data over a TLS connection.

I was able to reproduce the issue with a libev test that did not use a timer to publish messages. This version of the libev test case has been submitted to the following repository (same as above):

https://github.com/trevorperrin/AMQP-TEST

With this test, I can run on a non-secure port without noticing any issues in sending data. If I run over a TLS connection, it buffers up all transactions until the end, then emits them all at once (as noted in previous comments). If I modify sslconnected::send to issue a write command (also as noted above), then I don't notice any issues sending data.

[EmielBruijntjes]

In the example that you gave you're not running the event loop between the publish calls. This could indeed lead to internal buffering, but this is not necessarily a bug. In fact, if you run your code on a non-secure connection, you will also run into buffering issues if you use a (much) bigger data load on a slow system/network.

The network-handling code of the AMQP-CPP library has internal buffers that are used if the underlying transport layer can not keep up with sending all the data. This applies to both secure connections (which is built on top of openssl) and non-secure connections (which sets up non-blocking TCP connections). In your application code you must therefore always run the event loop in between calls to AMQP-CPP, and you must also consider that it might be a bad idea to send a lot of data at once, because that could lead to internal buffering. This is especially true for secure connections, because openssl specifies that each write operation may have to be repeated, so AMQP-CPP always copies all to-be-sent data before starting the actual write operation, just to be prepared to repeat the call later on.

There is indeed room for optimization, which you found out by starting the write operation directly from the SslConnected::send() method. This is a good idea as an optimization, but you also have to deal with error handling in case the write operation fails, so just adding the write() call is not enough. If you like, you can dig into this and send a pull request. But even with the optimization in AMQP-CPP, your application level code must still be prepared to deal with slow connections that build up internal buffers anyway.

A more serious issue seems to be that the boost asio handler does not seem to work. Maybe @zerodefect can help you with this?

[zerodefect]

@trevorperrin Are you making progress? What is the latest?

[trevorperrin]

@zerodefect I've been looking at modifying a small test harness to use the libev handler in a way that doesn't starve the ev_loop, while also not locking the publisher into pushing out messages on a timer.

When trying to use a TLS connection with the Boost ASIO handler, it fails in the TcpConnection constructor unless I force a write in the SslConnected::send method. I put a small ASIO test in the repository referenced above (https://github.com/trevorperrin/AMQP-TEST). Running that example after commenting out everything between the connection constructor and the start of the ioService, the connection fails as initially stated above.

Ideally, I'd like to solve that issue, as I prefer using Boost ASIO.

[trevorperrin]

I've put together a small test which runs the ev_loop in a separate thread, and uses ev_async structs and custom watchers to register events against the ev_loop. In this way, I can declare exchanges and issue publish commands from outside the loop, which in turn will wake up the loop and declare/publish/etc. This is being done without modifying the SslConnected class to force a write in the send function.

I have as of yet been unsuccessful in converting this to a working Boost ASIO implementation. The test fails in the same fashion as mentioned above.

[tilsche]

We also have trouble getting ASIO with SSL working. For testing we modified examples/libboostasio.cpp by adding initialization and changed the address to amqps. The declareQueue.onSuccess callback is never invoked, the program exits after 10 seconds. The examples/libev.cpp works as expected.