loading large number of certificates is slow

[jmayclin]

Problem:

We are seeing that it takes ~30 seconds to load ~20,000 certificates. This means that it's taking more than 1 ms per cert chain.

Solution:

Unknown. We need to figure out what about this process is so slow, and make it not slow 🙂

Requirements / Acceptance Criteria:

We don't have any strict SLA on cert loading, but if I were to pull some numbers from thin air, it feels like loading a cert chain should be < 100 us, and preferably < 10 us.

[jmayclin]

Starting certificate load performance test with 1 certificates
create s2n_cert: 50.815825ms
adding to config: 6.461µs
build config: 5.474758ms
Starting certificate load performance test with 10 certificates
create s2n_cert: 75.557707ms
adding to config: 20.723µs
build config: 4.997674ms
Starting certificate load performance test with 100 certificates
create s2n_cert: 753.189038ms
adding to config: 138.455µs
build config: 5.483482ms
Starting certificate load performance test with 1000 certificates
create s2n_cert: 7.540376736s
adding to config: 1.323211ms
build config: 5.492289ms
Starting certificate load performance test with 10000 certificates
create s2n_cert: 75.428231449s
adding to config: 15.668788ms
build config: 5.385493ms

[jmayclin]

Even after turning off the public/private key agreement check, this is still very slow

Starting certificate load performance test with 1 certificates
create s2n_cert: 43.975074ms
adding to config: 7.308µs
build config: 5.492813ms
Starting certificate load performance test with 10 certificates
create s2n_cert: 7.815077ms
adding to config: 19.434µs
build config: 4.883167ms
Starting certificate load performance test with 100 certificates
create s2n_cert: 75.400414ms
adding to config: 115.501µs
build config: 5.490135ms
Starting certificate load performance test with 1000 certificates
create s2n_cert: 759.18109ms
adding to config: 1.21033ms
build config: 5.577552ms

[jmayclin]

It looks like the vast majority of our time is spent in actually creating the cert, e.g. s2n_cert_chain_and_key_load_pem_bytes. This is a little surprising because that's a relatively thin wrapper over the libcrypto d2i_X509. It's possible that we're doing something else nasty, but my current hypothesis is that this is case of AWS-LC's cert parsing being slow.

We need a flamegraph or direct benchmark of the AWS-LC function to confirm that.

Edit: I changed my mind. Because we are presumably calling those same functions in the handshake, where it's much faster than that. 🤔

[jmayclin]

s2n-tls/crypto/s2n_certificate.c

Lines 396 to 407 in 8b1ed69

	int s2n_cert_chain_and_key_load_pem_bytes(struct s2n_cert_chain_and_key *chain_and_key, uint8_t *chain_pem,
	uint32_t chain_pem_len, uint8_t *private_key_pem, uint32_t private_key_pem_len)
	{
	POSIX_ENSURE_REF(chain_and_key);
	POSIX_GUARD(s2n_cert_chain_and_key_set_cert_chain_bytes(chain_and_key, chain_pem, chain_pem_len));
	POSIX_GUARD(s2n_cert_chain_and_key_set_private_key_bytes(chain_and_key, private_key_pem, private_key_pem_len));
	POSIX_GUARD(s2n_cert_chain_and_key_load(chain_and_key));
	return S2N_SUCCESS;
	}

set chain bytes: 66 us
set key bytes: 630 us
set chain&key load: 50 us

[cpsource]

how about this idea:

Lazy key parsing

Don't parse the private key at load time. Store the raw PEM/DER bytes and only deserialize into EVP_PKEY on first use
(i.e., when the cert is actually selected during a handshake). Most multi-tenant setups load thousands of certs but only
a fraction are active at any given moment.

// Instead of parsing immediately in s2n_cert_chain_and_key_load_pem_bytes:
chain_and_key->raw_key_pem = copy_of_pem_bytes; // just stash it
chain_and_key->pkey = NULL; // parse later

// On first handshake that selects this cert:
if (!chain_and_key->pkey) {
chain_and_key->pkey = parse_key(chain_and_key->raw_key_pem);
}

This turns 12-second startup into near-instant, spreading the cost across actual traffic. The downside is you lose
fail-fast on bad keys — but you could offer both modes via a config flag.