address undefined shift behavior and overflow #6133

Job	Run time
make check (--enable-asynccrypt --enable-all --enable-dtls13 CFLAGS="-pedantic -Wdeclaration-afte...	3m 8s
make check (--enable-asynccrypt-sw --enable-ocspstapling --enable-ocspstapling2 CFLAGS="-pedantic...	1m 34s
make check (--enable-ocsp CFLAGS="-DTEST_NONBLOCK_CERTS -pedantic -Wdeclaration-after-statement -...	1m 6s
	5m 48s

Provide feedback