tun: AMD64 optimized checksum

This adds AMD64 assembly implementations of IP checksum computation, one
for baseline AMD64 and the other for v3 AMD64 (AVX2 and BMI2).

All performance numbers reported are from a Ryzen 7 4750U but similar
improvements are expected for a wide range of processors.

The generic IP checksum implementation has also been further improved to
be significantly faster using bits.AddUint64 (for a 64KiB buffer the
throughput improves from 15,000MiB/s to 27,600MiB/s; similar gains are
also reported on ARM64 but I do not have specific numbers).

The baseline AMD64 implementation for a 64KiB buffer reports 32,700MiB/s
and the AVX2 implementation is slightly over 107,000MiB/s.

Unfortunately, for very small sizes (e.g. the expected size for an IPv4
header) setting up SIMD computation involves some overhead that makes
computing a checksum for small buffers slower than a non-SIMD
implementation. Even more unfortunately, testing for this at runtimen in
Go and calling a func optimized for small buffers mitigates most of the
improvement due to call overhead. The break even point is around 256
byte buffers; IPv4 headers are no more than 60 bytes including
extensions. IPv6 headers do not have a checksum but are a fixed size of
40 bytes. As a result, the generated assembly code uses an alternate
approach for buffers of less than 256 bytes. Additionally, buffers of
less than 32 bytes need to be handled specially because the strategy for
reading buffers that are not a multiple of 8 bytes fails when the buffer
is too small.

As suggested by additional benchmarking, pseudo header computation has
been rewritten to be faster (benchmark time reduced by 1/2 to 1/4).

Updates tailscale/corp#9755

Signed-off-by: Adrian Dewhurst <[email protected]>

Author: sailorfrag

tun/checksum.go

@@ -0,0 +1,20 @@
+package tun
+
+import "golang.org/x/sys/cpu"
+
+// checksum computes an IP checksum starting with the provided initial value.
+// The length of data should be at least 128 bytes for best performance. Smaller
+// buffers will still compute a correct result. For best performance with
+// smaller buffers, use shortChecksum().
+var checksum = checksumAMD64
+
+func init() {
+	if cpu.X86.HasAVX && cpu.X86.HasAVX2 && cpu.X86.HasBMI2 {
+		checksum = checksumAVX2
+		return
+	}
+	if cpu.X86.HasSSE2 {
+		checksum = checksumSSE2
+		return
+	}
+}

tun/checksum_amd64.go

@@ -0,0 +1,45 @@
+//go:build amd64
+
+package tun
+
+import (
+	"golang.org/x/sys/cpu"
+)
+
+var archChecksumFuncs = []archChecksumDetails{
+	{
+		name:      "generic32",
+		available: true,
+		f:         checksumGeneric32,
+	},
+	{
+		name:      "generic64",
+		available: true,
+		f:         checksumGeneric64,
+	},
+	{
+		name:      "generic32Alternate",
+		available: true,
+		f:         checksumGeneric32Alternate,
+	},
+	{
+		name:      "generic64Alternate",
+		available: true,
+		f:         checksumGeneric64Alternate,
+	},
+	{
+		name:      "AMD64",
+		available: true,
+		f:         checksumAMD64,
+	},
+	{
+		name:      "SSE2",
+		available: cpu.X86.HasSSE2,
+		f:         checksumSSE2,
+	},
+	{
+		name:      "AVX2",
+		available: cpu.X86.HasAVX && cpu.X86.HasAVX2 && cpu.X86.HasBMI2,
+		f:         checksumAVX2,
+	},
+}