TLS 1.3, plaintext alert: ignore when expecting encrypted

[SparkiDev]

Description

In TLS 1.3, ignore valid unencrypted alerts that appear after encryption has started.
Only ignore WOLFSSL_ALERT_COUNT_MAX-1 alerts.

Fixes zd#20857

Testing

./configure --disable-shared
make
./tests/unit.test -test_tls13_plaintext_alert

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[devin-ai-integration]

🛟 Devin Lifeguard found 1 likely issues in this PR

• check-all-return-codes snippet snippet snippet: Capture and assert the return value of wolfSSL_SetIORecv, wolfSSL_SetIOSend and wolfSSL_SetIOReadCtx (e.g. ExpectIntEQ(wolfSSL_SetIORecv(ctx, MbRecv), WOLFSSL_SUCCESS);).

@SparkiDev
please take a look at the above issues which Devin flagged. Devin will not fix these issues automatically.

[rizlik]

If the goal of this PR is to protect against DoS I don't think it's a good idea:

• under TCP one can easily forge a RST packet and close the connection
• peer can still forge arbitrary packets that cause the handshake to error out (handshake, app_data) ecc

UDP is different but DtlsShouldDrop ignores plaintext packet after handshake keys are computed.

[SparkiDev]

DoS can be done by any message, it doesn't have to be a valid alert.
But yes that is the report.

Instead the PR is about skipping alerts that were sent by the client before it received anything from the server to indicate it should be encrypted. OpenSSL and others do this.

May make this a compile time option.

[SparkiDev]

retest this please

[julek-wolfssl]

I still don't agree with fixing it. IMO ignoring plaintext alerts is going to introduce edge cases in the transition period between plaintext and ciphertext. This in turn can hang clients/servers that think the connection is still alive when its really dead. The argument that its a DoS of a connection doesn't make any sense since injecting garbage would also kill the connection.

[SparkiDev]

Changed to be compile time option when WOLFSSL_TLS13_IGNORE_PT_ALERT_ON_ENC is defined.
Test is testing as many alerts as is required to cause an error or one less being OK.
Test is checking that without define that an error is returned.

[SparkiDev]

retest this please

[julek-wolfssl]

Please add WOLFSSL_TLS13_IGNORE_PT_ALERT_ON_ENC to os-check.yml. I think a good configuration would be one that explicitly enables TLS and DTLS both 1.2 and 1.3.

[SparkiDev]

rebased and add configuration to os-check.yml

[SparkiDev]

rebased and updated os-check.yml to use CPPFLAGS

[SparkiDev]

retest this please

generic:
wolf-linux-cloud-node-d3ayn7 was marked offline: Connection was broken

[SparkiDev]

retest this please