enhancement(common): add stable hash function for restart-safe hashes

[tobz]

Summary

We already maintain an opaque "fast" hasher, meant specifically for high-performance hashing (hashmaps, and so on) where the stability of the hash output for a given input does not need to hold across process restarts or releases, and the underlying implementation is free to be changed between releases. However, we have a need to be able to hash a particular input in a stable way such that the resulting hash stays the same across restarts and releases, such as when adding a hash component to a file path.

This PR introduces StableHasher to provide this. StableHasher is based on SHA-3 (specifically, SHAKE128) which is a standardized cryptographic hash algorithm, which means the generated output is always stable with respect to identical inputs, and it can be recreated in other applications/languages that properly implement SHA-3.

We've additionally added a written rule around the stability of this usage, such that it will remain the "stable" hasher implementation until at least Saluki v2 (we're currently at v0) to allow for long-term stability.

One note is that we use StableHasher to implement std::hash::Hasher, which means it only outputs 64-bit hashes. This is sufficient for simple hashing regimes but is not suitable for actual cryptographic needs, despite SHA-3's cryptographic suitability: as we don't maintain a large-enough output block, there's not enough entropy to do actual secure cryptographic things.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

N/A

References

AGTMETRICS-233

[tobz]

• chore(datadog): generate a unique retry queue id for component ID/endpoint/API key combination #674
• enhancement(common): add stable hash function for restart-safe hashes #673 👈 (View in Graphite)
• chore(datadog): create separate events and service checks destinations to match metrics #668
• chore(datadog): add support for writing delimited payloads to RequestBuilder<E, O> #660
• chore(datadog): refactor metrics destination to use new generic request builder #659
• chore(datadog): refactor RequestBuilder<O> to be trait-driven to allow reuse between destinations #658
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Regression Detector (DogStatsD)

Regression Detector Results

Run ID: bab1880a-f9f5-4303-80d8-e50ec6ae0852

Baseline: 7.65.0-rc.9
Comparison: 7.65.0-rc.9

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gates_idle_rss	memory utilization	+1.14	[+1.01, +1.28]	1
➖	dsd_uds_512kb_3k_contexts	ingress throughput	+0.00	[-0.01, +0.01]	1
➖	dsd_uds_1mb_50k_contexts	ingress throughput	+0.00	[-0.00, +0.00]	1
➖	dsd_uds_1mb_3k_contexts_dualship	ingress throughput	-0.00	[-0.00, +0.00]	1
➖	dsd_uds_1mb_3k_contexts	ingress throughput	-0.00	[-0.00, +0.00]	1
➖	dsd_uds_1mb_50k_contexts_memlimit	ingress throughput	-0.00	[-0.00, +0.00]	1
➖	dsd_uds_40mb_12k_contexts_40_senders	ingress throughput	-0.00	[-0.00, +0.00]	1
➖	dsd_uds_100mb_250k_contexts	ingress throughput	-0.01	[-0.12, +0.10]	1
➖	dsd_uds_100mb_3k_contexts	ingress throughput	-0.01	[-0.11, +0.09]	1
➖	dsd_uds_10mb_3k_contexts	ingress throughput	-0.02	[-0.06, +0.02]	1
➖	dsd_uds_500mb_3k_contexts	ingress throughput	-0.44	[-0.50, -0.38]	1
➖	dsd_uds_100mb_3k_contexts_distributions_only	memory utilization	-0.64	[-0.85, -0.43]	1

Bounds Checks: ❌ Failed

perf	experiment	bounds_check_name	replicates_passed	links
❌	quality_gates_idle_rss	memory_usage	0/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[pr-commenter]

Regression Detector (Checks Agent Go)

Regression Detector Results

Run ID: 97256fc7-e9fc-4f5b-ba51-89e8d3831f1e

Baseline: f61d1f4e054b884cb1894254ab2714b84b4684cb
Comparison: f61d1f4e054b884cb1894254ab2714b84b4684cb
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gates_idle_rss	memory utilization	+0.38	[+0.33, +0.44]	1

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[pr-commenter]

Regression Detector (Checks Agent)

Regression Detector Results

Run ID: e5867c20-7b85-43dd-9c74-605b54cef89c

Baseline: b50e0dd
Comparison: 5375d2a
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gates_idle_rss	memory utilization	+0.04	[+0.02, +0.05]	1

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	quality_gates_idle_rss	memory_usage	10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[pr-commenter]

Regression Detector (Saluki)

Regression Detector Results

Run ID: 65ca9189-3a88-4b0a-a3ac-d8ea947d613d

Baseline: b50e0dd
Comparison: c5f93cc
Diff

❌ Experiments with missing or malformed data

This is a critical error. No usable optimization goal data was produced by the listed experiments. This may be a result of misconfiguration. Ping #single-machine-performance and we can help out.

• dsd_uds_1mb_50k_contexts_memlimit
• dsd_uds_50mb_10k_contexts_no_inlining
• dsd_uds_50mb_10k_contexts_no_inlining_no_allocs

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	dsd_uds_100mb_3k_contexts_distributions_only	memory utilization	+0.25	[+0.09, +0.41]	1
➖	dsd_uds_1mb_3k_contexts_dualship	ingress throughput	+0.01	[-0.01, +0.02]	1
➖	dsd_uds_100mb_250k_contexts	ingress throughput	-0.00	[-0.08, +0.08]	1
➖	dsd_uds_100mb_3k_contexts	ingress throughput	-0.00	[-0.07, +0.07]	1
➖	dsd_uds_512kb_3k_contexts	ingress throughput	-0.00	[-0.02, +0.01]	1
➖	dsd_uds_40mb_12k_contexts_40_senders	ingress throughput	-0.01	[-0.04, +0.03]	1
➖	dsd_uds_1mb_50k_contexts	ingress throughput	-0.01	[-0.02, +0.01]	1
➖	dsd_uds_10mb_3k_contexts	ingress throughput	-0.01	[-0.04, +0.03]	1
➖	dsd_uds_1mb_3k_contexts	ingress throughput	-0.01	[-0.02, +0.00]	1
➖	quality_gates_idle_rss	memory utilization	-1.06	[-1.08, -1.04]	1
➖	dsd_uds_500mb_3k_contexts	ingress throughput	-1.09	[-1.21, -0.98]	1

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	quality_gates_idle_rss	memory_usage	10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".