Refactor target label handling in target allocator for improved performance

[swiatekm]

Description:

Refactor target label handling in target allocator. The intent is to improve performance on both main and in #4547, which otherwise leads to major regressions.

This PR makes two major changes:

The target.Item struct no longer holds labels after relabeling

Instead, when relabelling, we calculate the hash directly from the label builder and pass it to the target.Item constructor. This way, we can avoid materializing the new labels and allocating memory for them. Processing labels using a builder rather than the raw labels data structure also makes this implementation independent of the underlying implementation of labels.Labels, which changes in recent Prometheus releases.

As a minor optimization, we also reuse xxhash instances by storing them in a sync.Pool.

We initially construct labels using a labels.ScratchBuilder

This is a straightforward optimization focused on minimizing memory allocations. I've left detailed comments in the code explaining the specific decisions.

Testing:

Benchstat comparing the results from this branch to main:

goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator
cpu: AMD Ryzen 9 7950X3D 16-Core Processor
                                                             │ bench_main.txt │ bench_main_relabelbuilder_claude_scratchbuilder.txt │
                                                             │     sec/op     │           sec/op             vs base                │
ProcessTargets/per-node/1000-32                                  1046.7µ ± 2%                   633.2µ ± 1%  -39.50% (p=0.000 n=10)
ProcessTargets/least-weighted/1000-32                            1067.9µ ± 4%                   635.4µ ± 1%  -40.50% (p=0.000 n=10)
ProcessTargets/consistent-hashing/1000-32                        1072.3µ ± 1%                   631.7µ ± 1%  -41.09% (p=0.000 n=10)
ProcessTargets/least-weighted/10000-32                            6.712m ± 1%                   3.995m ± 1%  -40.47% (p=0.000 n=10)
ProcessTargets/consistent-hashing/10000-32                        6.729m ± 1%                   4.020m ± 1%  -40.27% (p=0.000 n=10)
ProcessTargets/per-node/10000-32                                  6.763m ± 3%                   3.995m ± 1%  -40.93% (p=0.000 n=10)
ProcessTargets/least-weighted/100000-32                           64.60m ± 3%                   41.08m ± 2%  -36.41% (p=0.000 n=10)
ProcessTargets/consistent-hashing/100000-32                       63.96m ± 3%                   40.95m ± 7%  -35.98% (p=0.000 n=10)
ProcessTargets/per-node/100000-32                                 64.65m ± 4%                   41.16m ± 1%  -36.34% (p=0.000 n=10)
ProcessTargets/consistent-hashing/800000-32                       617.7m ± 7%                   445.7m ± 3%  -27.84% (p=0.000 n=10)
ProcessTargets/per-node/800000-32                                 631.0m ± 3%                   436.6m ± 3%  -30.80% (p=0.000 n=10)
ProcessTargets/least-weighted/800000-32                           628.8m ± 8%                   441.3m ± 3%  -29.81% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/1000-32           1652.8µ ± 1%                   894.4µ ± 1%  -45.89% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/1000-32       1673.6µ ± 2%                   900.9µ ± 1%  -46.17% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/1000-32                 1667.2µ ± 1%                   896.0µ ± 1%  -46.26% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/10000-32          12.855m ± 7%                   7.478m ± 2%  -41.83% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/10000-32      12.586m ± 2%                   7.424m ± 1%  -41.01% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/10000-32                12.711m ± 2%                   7.395m ± 1%  -41.82% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/100000-32         119.73m ± 2%                   67.92m ± 1%  -43.27% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/100000-32     120.15m ± 2%                   67.60m ± 1%  -43.74% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/100000-32               120.58m ± 2%                   67.33m ± 2%  -44.16% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/800000-32          940.3m ± 4%                   508.8m ± 1%  -45.89% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/800000-32      937.4m ± 4%                   510.7m ± 3%  -45.52% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/800000-32                961.6m ± 4%                   514.4m ± 2%  -46.51% (p=0.000 n=10)
geomean                                                           30.21m                        17.91m       -40.72%

                                                             │ bench_main.txt │ bench_main_relabelbuilder_claude_scratchbuilder.txt │
                                                             │      B/op      │            B/op              vs base                │
ProcessTargets/per-node/1000-32                                  4.205Mi ± 0%                  1.501Mi ± 0%  -64.31% (p=0.000 n=10)
ProcessTargets/least-weighted/1000-32                            4.205Mi ± 0%                  1.501Mi ± 0%  -64.31% (p=0.000 n=10)
ProcessTargets/consistent-hashing/1000-32                        4.205Mi ± 0%                  1.501Mi ± 0%  -64.31% (p=0.000 n=10)
ProcessTargets/least-weighted/10000-32                           41.98Mi ± 0%                  14.93Mi ± 0%  -64.43% (p=0.000 n=10)
ProcessTargets/consistent-hashing/10000-32                       41.98Mi ± 0%                  14.93Mi ± 0%  -64.43% (p=0.000 n=10)
ProcessTargets/per-node/10000-32                                 41.98Mi ± 0%                  14.93Mi ± 0%  -64.43% (p=0.000 n=10)
ProcessTargets/least-weighted/100000-32                          419.7Mi ± 0%                  149.0Mi ± 0%  -64.51% (p=0.000 n=10)
ProcessTargets/consistent-hashing/100000-32                      419.6Mi ± 0%                  149.0Mi ± 0%  -64.50% (p=0.000 n=10)
ProcessTargets/per-node/100000-32                                419.6Mi ± 0%                  149.0Mi ± 0%  -64.50% (p=0.000 n=10)
ProcessTargets/consistent-hashing/800000-32                      3.309Gi ± 0%                  1.185Gi ± 0%  -64.20% (p=0.000 n=10)
ProcessTargets/per-node/800000-32                                3.310Gi ± 0%                  1.185Gi ± 0%  -64.20% (p=0.000 n=10)
ProcessTargets/least-weighted/800000-32                          3.309Gi ± 0%                  1.185Gi ± 0%  -64.20% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/1000-32           5.215Mi ± 0%                  1.522Mi ± 0%  -70.81% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/1000-32       5.215Mi ± 0%                  1.522Mi ± 0%  -70.81% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/1000-32                 5.215Mi ± 0%                  1.522Mi ± 0%  -70.81% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/10000-32          52.10Mi ± 0%                  15.17Mi ± 0%  -70.88% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/10000-32      52.10Mi ± 0%                  15.17Mi ± 0%  -70.88% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/10000-32                52.10Mi ± 0%                  15.17Mi ± 0%  -70.88% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/100000-32         521.1Mi ± 0%                  151.6Mi ± 0%  -70.91% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/100000-32     521.1Mi ± 0%                  151.6Mi ± 0%  -70.91% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/100000-32               521.1Mi ± 0%                  151.6Mi ± 0%  -70.91% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/800000-32         4.085Gi ± 0%                  1.200Gi ± 0%  -70.63% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/800000-32     4.085Gi ± 0%                  1.200Gi ± 0%  -70.63% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/800000-32               4.085Gi ± 0%                  1.200Gi ± 0%  -70.63% (p=0.000 n=10)
geomean                                                          140.1Mi                       45.20Mi       -67.74%

                                                             │ bench_main.txt │ bench_main_relabelbuilder_claude_scratchbuilder.txt │
                                                             │   allocs/op    │          allocs/op           vs base                │
ProcessTargets/per-node/1000-32                                   4.395k ± 0%                   2.146k ± 0%  -51.17% (p=0.000 n=10)
ProcessTargets/least-weighted/1000-32                             4.395k ± 0%                   2.146k ± 0%  -51.17% (p=0.000 n=10)
ProcessTargets/consistent-hashing/1000-32                         4.395k ± 0%                   2.146k ± 0%  -51.17% (p=0.000 n=10)
ProcessTargets/least-weighted/10000-32                            43.77k ± 0%                   21.28k ± 0%  -51.39% (p=0.000 n=10)
ProcessTargets/consistent-hashing/10000-32                        43.77k ± 0%                   21.28k ± 0%  -51.39% (p=0.000 n=10)
ProcessTargets/per-node/10000-32                                  43.77k ± 0%                   21.28k ± 0%  -51.39% (p=0.000 n=10)
ProcessTargets/least-weighted/100000-32                           436.5k ± 0%                   211.4k ± 0%  -51.56% (p=0.000 n=10)
ProcessTargets/consistent-hashing/100000-32                       436.5k ± 0%                   211.4k ± 0%  -51.55% (p=0.000 n=10)
ProcessTargets/per-node/100000-32                                 436.5k ± 0%                   211.4k ± 0%  -51.56% (p=0.000 n=10)
ProcessTargets/consistent-hashing/800000-32                       3.494M ± 0%                   1.693M ± 0%  -51.55% (p=0.000 n=10)
ProcessTargets/per-node/800000-32                                 3.494M ± 0%                   1.693M ± 0%  -51.55% (p=0.000 n=10)
ProcessTargets/least-weighted/800000-32                           3.494M ± 0%                   1.693M ± 0%  -51.55% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/1000-32            7.888k ± 0%                   2.641k ± 0%  -66.52% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/1000-32        7.888k ± 0%                   2.641k ± 0%  -66.52% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/1000-32                  7.888k ± 0%                   2.641k ± 0%  -66.52% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/10000-32           78.74k ± 0%                   26.24k ± 0%  -66.68% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/10000-32       78.74k ± 0%                   26.24k ± 0%  -66.68% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/10000-32                 78.74k ± 0%                   26.24k ± 0%  -66.68% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/100000-32          786.3k ± 0%                   261.3k ± 0%  -66.77% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/100000-32      786.3k ± 0%                   261.3k ± 0%  -66.77% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/100000-32                786.4k ± 0%                   261.3k ± 0%  -66.77% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/least-weighted/800000-32          6.291M ± 0%                   2.091M ± 0%  -66.76% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/consistent-hashing/800000-32      6.291M ± 0%                   2.091M ± 0%  -66.76% (p=0.000 n=10)
ProcessTargetsWithRelabelConfig/per-node/800000-32                6.292M ± 0%                   2.091M ± 0%  -66.76% (p=0.000 n=10)
geomean                                                           175.5k                        70.63k       -59.77%

[atoulme]

LGTM