all: add observability tests

[odeke-em]

Reported offline by @johanbrandhorst, there is an issue whereby Prometheus trips out when a value isn't recorded. The issue is with the Prometheus Go exporter but was exacerbated by a missing tag value when recording MRoundtripLatencyMs in

redigo/redis/pool.go

Line 380 in 6175cfa

mls = append(mls, observability.MRoundtripLatencyMs.M(observability.SinceInMilliseconds(dialStartTime)))

Test should be able to catch such problems but also to verify that we don't encounter missing tags.