PMTU discovery at time of handshake

[ShadowJonathan]

One thing I'd like to implement, is a "smart" discovery mechanism of path MTU at handshake time.

While the normal behaviour for the client would be to back off and retransmit with smaller packets if the server hasn't replied yet, I'd like to add in an optional opt-in piggy-back mechanism which would have to be implemented on both the client and server level.

Basically, when the server receives flight 1, it derives the client-discovered MTU, and then performs a little experiment; while it responds with the ServerHello with packets in a similar MTU, it duplicates the first sequence of bytes it sends with a slightly increased MTU.

The client would then have to be configured to recognise overlapping packets, with one a higher MTU, and respond in that higher MTU, copying the first sequence in a slightly higher bound.

This could be bounced a few times, until the bigger-MTU packets go MIA, after which the increasing is stopped. This would expend some slightly more datagram packets to secure and same-time MTU discovery.

Then, the server and the client can give the discovered MTU time to the connection objects they spawn, as they do normally.

This would have to be optional and opt-in on both sides, the RFC tells that this behaviour would be allowed, and that it doesn't trigger any fatal alerts on either side.

[ShadowJonathan]

After a long while of rethinking, i believe this is best handled by a kickback mechanism from the IP/UDP layer (or any other layer) allowing delivery of "PTMU hints" to the DTLS layer, basically a three-part solution as per the following;

• Allow giving an initial "PMTU hint", with which DTLS will start from
• Allow giving additional "PTMU too-big hints", which will have the DTLS layer further fragment its internals
• Allow independent DTLS downsizing of the handshake, when it thinks that packets arent arriving at all (this is the only "smart" part, and is optional, enabled by default)

[ShadowJonathan]

I've also come to realise that I might be making a bigger problem of this than necessary, as IP fragmentation still takes place when MTU limits are exceeded.

So, the MTU hints to DTLS are going to be just that, hints, the handshake part of DTLS needs an explicit hint (defaulting to 1500) to start with, throttling down to more a lower size when handshaking fails (to deal with IP fragmentation breaking), while the session state is just going to allow any-size packets, with a suggested size of 1500 - overhead (obtainable via a function that calculates those overheads)