Rare intermittent failure of DTLS 3d tests

[gilles-peskine-arm]

This issue tracks rare intermittent failures of DTLS 3D (drop, delay, duplicate) tests in ssl-opt.sh on the CI.

In the merge queue for #8443 (ephemeral links: build, log), for full_without_ecdhe_ecdsa:

https://mbedtls.trustedfirmware.org/job/mbed-tls-pr-head/job/gh-readonly-queue%252Fdevelopment%252Fpr-8443-fc31cb28eeda7fafef1bd392fccee3b99035105b/1/

Logs:
8443-full_without_ecdhe_ecdsa.zip

Also observed on https://mbedtls.trustedfirmware.org/blue/organizations/jenkins/mbed-tls-pr-head/detail/PR-8454-head/7 in all.sh full_without_ecdhe_ecdsa.

Also observed on https://mbedtls.trustedfirmware.org/job/mbed-tls-pr-head/job/PR-8612-head/2/
PR-8612-head-2.zip

[gilles-peskine-arm]

We observed a similar-looking failure on #9212 at c5274f9 in component_full_without_ecdhe_ecdsa_and_tls13.
renego-ClientKeyExchange-duplicated.zip

What seems to be happening here is, in the proxy log, we see that the ClientKeyExchange message for the renegotiation is duplicated:

  05026 dispatch S <- C ClientKeyExchange (125 bytes): forwarded
  05026 dispatch S <- C ClientKeyExchange (125 bytes): duplicated

In the server logs, we see that the server processes the ClientKeyExchange as expected, ending with

ssl_tls.c:7267: |2| 0x7ffcc1e08970: <= derive keys
ssl_tls12_server.c:4077: |2| 0x7ffcc1e08970: <= parse client key exchange
…
ssl_msg.c:5232: |3| 0x7ffcc1e08970: switching to new transform spec for inbound data
ssl_msg.c:5259: |2| 0x7ffcc1e08970: <= parse change cipher spec

Then the server reads the next record, which is the duplicated ClientKeyExchange. We can see in the log that it is indeed a duplicated packet. psa_aead_decrypt fails. What seems to be happening, based on the logs, is that the server is trying to decrypt-authenticate the duplicated packet with the new session keys. But that packet is encrypted with the old session keys. The server rejects the duplicated packet when it should ignore it.

[gilles-peskine-arm]

Observed on Connection ID, 3D+MTU: Cli+Srv enabled, CID on renegotiation in full_without_ecdhe_ecdsa_and_tls13 at baa51c2: https://mbedtls.trustedfirmware.org/blue/rest/organizations/jenkins/pipelines/mbed-tls-pr-head/branches/PR-10021-head/runs/6/nodes/679/steps/3676/log/?start=0

all_u16-full_without_ecdhe_ecdsa_and_tls13-o-srv-120.log.txt
all_u16-full_without_ecdhe_ecdsa_and_tls13-o-pxy-120.log.txt
all_u16-full_without_ecdhe_ecdsa_and_tls13-o-cli-120.log.txt

and again at c22e315 (on #10030)

[gilles-peskine-arm]

Observed on Mbed-TLS/TF-PSA-Crypto#369 (mbedtls at f2d4cbb) in full_without_ecdhe_ecdsa_and_tls13 on "Connection ID, 3D+MTU: Cli+Srv enabled, CID on renegotiation" on OpenCI:
https://mbedtls.trustedfirmware.org/job/mbed-tls-tf-psa-crypto-multibranch/job/PR-369-head/10/execution/node/4776/log/?consoleFull

[gilles-peskine-arm]

Observed on the mbedtls nightly at d103046 on "Connection ID, 3D+MTU: Cli+Srv enabled, CID on renegotiation" on OpenCI.