In CCF, the :term:`TLS` layer is implemented using OpenSSL 3.3.
CCF handles RPC requests by managing a collection of HTTPS sessions on each node. The HTTP session receives raw TCP bytes from the :term:`ring buffer` and passes it to the TLS implementation that tries to decrypt on read (after a successful handshake).
If there isn't enough information to decrypt on read, the TLS layer responds with a 'WANT_READ' status, meaning more packets are needed to complete the message, so the endpoint tries to get more data from the ring buffer.
Once all incoming data is decrypted, the HTTP session parses the plain text data and passes the parsed request to the application, which processes it and produces a plain text response. The session then serialises this response, encrypts it through the TLS layer, and writes the encrypted data to the :term:`ring buffer` to be sent to the original caller.
The TLS implementation in 'src/tls' has three main components:
- A 'Certificate Authority' (CA), which has a root certificate that can sign other certificates in the network. This CA certificate is generated by the network outside of the TLS implementation and is imported when clients and servers are created.
- A 'Certificate' (Cert), which is the server/client's own certificate, signed by the CA and unique to this server or client.
- A 'Context', which is the common implementation of both 'Client' and 'Server' and provides input (read and decrypt) and output (encrypt and write) over the ring buffer and its triggers. Clients and servers are basically just a Context with a certificate.
Both CA and Cert have internal logic to validate their certificates and private keys, and the Context has logic to complete a TLS handshake, read and write using encryption and to query the peer's certificate.
The next layer up is the 'TLSSession' (of which 'HTTPSession' owns an instance) that holds the ring buffer, the pending read and write buffers, and drives the I/O between the ring buffer and the Context.
The 'Context' uses a pair of in-memory BIO objects to exchange encrypted bytes with the peer. The TLS layer reads ciphertext from the 'read' BIO and writes ciphertext to the 'write' BIO.
The 'TLSSession' drives this I/O directly, without callbacks:
- On receive, it writes the bytes that arrived from the ring buffer into the read BIO (
Context::recv) before asking the TLS layer to decrypt them. - After every TLS operation, it drains any ciphertext the TLS layer produced from the write BIO (
Context::send) and forwards it to the ring buffer (viaRINGBUFFER_TRY_WRITE_MESSAGE(tcp_outbound, ...)). If the ring buffer is full, the bytes are retained and retried on the next operation.
When a node message is received through the ring buffer, the 'HTTPSession' recv_() method is called (as it was registered as a callback). That hands the bytes to TLSSession::recv_buffered(), which writes them into the Context's read BIO, and then to TLSSession::read() which calls ccf::tls::Context::read().
Context::read() calls SSL_read_ex, which reads the (still encrypted) bytes from the read BIO and tries to decrypt them. On success it returns the plain text and a status of 0. If there isn't enough data, it returns SSL_ERROR_WANT_READ so the endpoint can try to extract more information from the ring buffer.
The plain text result is returned all the way back to HTTPSession::recv_() that then sends it to the application to process.
When the application finishes, it returns a plain text response. That message is sent back to the client via the send() call, which ends up filling the pending_write buffer and calling flush() which itself calls ccf::tls::Context::write().
Context::write() calls SSL_write_ex which encrypts the data into the write BIO. The 'TLSSession' then drains the write BIO and sends the encrypted bytes through the ring buffer.
Different errors are treated at different levels (TLS status codes in Context, ring buffer errors in TLSSession and HTTP errors in HTTPSession).
Context::handshake, Context::read and Context::write return 0 on success and an OpenSSL SSL_ERROR_* status code otherwise (obtained from SSL_get_error). The number of bytes read or written is returned separately through an output parameter, so the return value is never overloaded to mean both a byte count and an error.
The caller (TLSSession) inspects the status code to decide whether to wait for more data (SSL_ERROR_WANT_READ / SSL_ERROR_WANT_WRITE), close the connection (SSL_ERROR_ZERO_RETURN), or treat it as an error. A certificate verification failure during the handshake is surfaced as a distinct status so it can be treated as an authentication failure.
The main reasons why we moved to OpenSSL are:
- We already use OpenSSL for our crypto library ('src/crypto').
- We wanted TLS 1.3 support and MbedTLS doesn't have it.
- We wanted to support QUIC, which doesn't work with MbedTLS.