provide option to enable annotation of spans with key-values

While examining latencies, it would be nice to know which keys and/or values caused bad behavior. We could provide an option to allow spans to be annotated with keys and values. However, this option reveals Personally Identifiable Information (PII) thus a security concern and should be opt-in.