Skip to content

Use new classic histogram conversion algorithm #1919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 30 commits into
base: main
Choose a base branch
from
Draft

Use new classic histogram conversion algorithm #1919

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b433634
Generate histogram test case inputs
dricross Sep 8, 2025
d733736
Add expected values for each test case
dricross Sep 8, 2025
490ada2
Add expected min. Add visualization utility
dricross Sep 9, 2025
6644712
fixup 0/1 bucket test cases. add unbounded histogram case
dricross Sep 9, 2025
20754be
Add more test cases, validation on more percentiles
dricross Sep 11, 2025
9605818
Bring in exponential mapping function
dricross Sep 19, 2025
b66221d
Update exponential algorithm. add accuracy tests
dricross Sep 23, 2025
62cfbe6
New algorithm passes all unit tests. Added comparison tool
dricross Sep 26, 2025
043ac49
documentation and testing updates
dricross Sep 29, 2025
d573f11
some optimization for exponential mapping cw
dricross Sep 29, 2025
94002ac
replace math.Pow
dricross Sep 29, 2025
9495fba
spread remainder linearly
dricross Sep 29, 2025
4277870
test lognormal/webiull for all algorithms
dricross Sep 29, 2025
251179a
Add cloudwatch tester program
dricross Sep 29, 2025
7dcabc7
Remove intermediate entries array
dricross Sep 29, 2025
40af2aa
adjust overflow safe average calc
dricross Sep 29, 2025
e22a2ed
Remove unnecessary logic
dricross Oct 1, 2025
c096a46
Remove logarithm and replace thresholds
dricross Oct 1, 2025
3889b5a
Add some comments
dricross Oct 1, 2025
e9cb8e2
Update test data
dricross Oct 1, 2025
a3bc06c
change density detection. fix maximum value count
dricross Oct 6, 2025
2fca5ff
use histograms in otel
dricross Oct 13, 2025
d691299
integrate with new conversion algorithm
dricross Oct 15, 2025
f358abe
Drop invalid histograms
dricross Oct 16, 2025
885a603
remove test data
dricross Oct 24, 2025
f5a42c2
Move histogram tests to contrib repo
dricross Oct 24, 2025
69f2d39
Moved code to contrib repo
dricross Oct 24, 2025
cc7ab39
make fmt
dricross Oct 24, 2025
2780b3d
Update pkg/aws contrib reference
dricross Oct 24, 2025
d75aab3
Revert go change
dricross Oct 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 35 additions & 27 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ replace (
)

replace (
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/aws => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/aws v0.0.0-20251024201210-4dff397492f8
// For clear resource attributes after copy functionality https://github.com/amazon-contributing/opentelemetry-collector-contrib/pull/148
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/resourcetotelemetry v0.0.0-20251014190537-ae1729ad22c4
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza => github.com/amazon-contributing/opentelemetry-collector-contrib/pkg/stanza v0.0.0-20251014190537-ae1729ad22c4
Expand Down Expand Up @@ -99,12 +100,17 @@ replace github.com/openshift/api v3.9.0+incompatible => github.com/openshift/api
// forces version bump to support log group classes
replace github.com/aws/aws-sdk-go => github.com/aws/aws-sdk-go v1.48.6

replace go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v1.35.0

replace go.opentelemetry.io/collector/pdata => go.opentelemetry.io/collector/pdata v1.30.0

require (
github.com/BurntSushi/toml v1.3.2
github.com/Jeffail/gabs v1.4.0
github.com/amazon-contributing/opentelemetry-collector-contrib/extension/awsmiddleware v0.124.0
github.com/aws/aws-sdk-go v1.55.7
github.com/aws/aws-sdk-go-v2 v1.36.3 // indirect
github.com/aws/aws-sdk-go-v2/config v1.29.13 // indirect
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.30.2
github.com/bigkevmcd/go-configparser v0.0.0-20200217161103-d137835d2579
github.com/deckarep/golang-set/v2 v2.3.1
Expand Down Expand Up @@ -138,6 +144,7 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.124.1
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.124.1
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.124.1
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/aws v0.0.0-00010101000000-000000000000
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry v0.124.1
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza v0.124.1
github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.124.1
Expand Down Expand Up @@ -174,7 +181,7 @@ require (
github.com/prometheus/prometheus v0.300.1
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/shirou/gopsutil/v3 v3.24.5
github.com/stretchr/testify v1.10.0
github.com/stretchr/testify v1.11.1
github.com/xeipuuv/gojsonschema v1.2.0
go.opentelemetry.io/collector/client v1.30.0
go.opentelemetry.io/collector/component v1.30.0
Expand Down Expand Up @@ -221,10 +228,10 @@ require (
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.27.0
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8
golang.org/x/net v0.39.0
golang.org/x/sync v0.13.0
golang.org/x/sys v0.32.0
golang.org/x/text v0.24.0
golang.org/x/net v0.46.0
golang.org/x/sync v0.17.0
golang.org/x/sys v0.37.0
golang.org/x/text v0.30.0
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/natefinch/lumberjack.v2 v2.0.0
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
Expand All @@ -236,10 +243,12 @@ require (
k8s.io/klog/v2 v2.130.1
)

require github.com/shirou/gopsutil/v4 v4.25.3

require (
cloud.google.com/go/auth v0.15.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.7 // indirect
cloud.google.com/go/compute/metadata v0.6.0 // indirect
cloud.google.com/go/compute/metadata v0.7.0 // indirect
collectd.org v0.4.0 // indirect
github.com/Azure/azure-sdk-for-go v67.1.0+incompatible // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.17.0 // indirect
Expand All @@ -251,7 +260,7 @@ require (
github.com/Azure/go-autorest/autorest/adal v0.9.23 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.3 // indirect
github.com/Code-Hex/go-generics-cache v1.5.1 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.27.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.29.0 // indirect
github.com/IBM/sarama v1.45.1 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/Microsoft/hcsshim v0.12.9 // indirect
Expand All @@ -270,7 +279,6 @@ require (
github.com/armon/go-metrics v0.4.1 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-msk-iam-sasl-signer-go v1.0.1 // indirect
github.com/aws/aws-sdk-go-v2/config v1.29.14 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.67 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34 // indirect
Expand All @@ -296,7 +304,7 @@ require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cloudwego/base64x v0.1.4 // indirect
github.com/cloudwego/iasm v0.2.0 // indirect
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42 // indirect
github.com/cncf/xds/go v0.0.0-20250501225837-2ac532fd4443 // indirect
github.com/containerd/cgroups/v3 v3.0.3 // indirect
github.com/containerd/containerd/api v1.8.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
Expand Down Expand Up @@ -334,7 +342,7 @@ require (
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-openapi/analysis v0.23.0 // indirect
Expand Down Expand Up @@ -436,7 +444,7 @@ require (
github.com/moby/sys/mountinfo v0.7.2 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
github.com/mostynb/go-grpc-compression v1.2.3 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
Expand Down Expand Up @@ -497,7 +505,6 @@ require (
github.com/rs/cors v1.11.1 // indirect
github.com/safchain/ethtool v0.0.0-20210803160452-9aa261dae9b1 // indirect
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 // indirect
github.com/shirou/gopsutil/v4 v4.25.3 // indirect
github.com/shurcooL/httpfs v0.0.0-20230704072500-f1e31cf0ba5c // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/sleepinggenius2/gosmi v0.4.4 // indirect
Expand Down Expand Up @@ -550,7 +557,7 @@ require (
go.opentelemetry.io/collector/extension/extensionauth v1.30.0 // indirect
go.opentelemetry.io/collector/extension/extensioncapabilities v0.124.0 // indirect
go.opentelemetry.io/collector/extension/xextension v0.124.0 // indirect
go.opentelemetry.io/collector/featuregate v1.30.0 // indirect
go.opentelemetry.io/collector/featuregate v1.43.0 // indirect
go.opentelemetry.io/collector/internal/fanoutconsumer v0.124.0 // indirect
go.opentelemetry.io/collector/internal/memorylimiter v0.124.0 // indirect
go.opentelemetry.io/collector/internal/sharedcomponent v0.124.0 // indirect
Expand All @@ -570,7 +577,7 @@ require (
go.opentelemetry.io/contrib/otelconf v0.15.0 // indirect
go.opentelemetry.io/contrib/propagators/b3 v1.35.0 // indirect
go.opentelemetry.io/contrib/zpages v0.60.0 // indirect
go.opentelemetry.io/otel v1.35.0 // indirect
go.opentelemetry.io/otel v1.37.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.11.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.11.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.35.0 // indirect
Expand All @@ -583,27 +590,28 @@ require (
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.35.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.35.0 // indirect
go.opentelemetry.io/otel/log v0.11.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.opentelemetry.io/otel/sdk v1.35.0 // indirect
go.opentelemetry.io/otel/metric v1.37.0 // indirect
go.opentelemetry.io/otel/sdk v1.37.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.11.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.35.0 // indirect
go.opentelemetry.io/otel/trace v1.35.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.37.0 // indirect
go.opentelemetry.io/otel/trace v1.37.0 // indirect
go.opentelemetry.io/proto/otlp v1.5.0 // indirect
go.uber.org/zap/exp v0.3.0 // indirect
golang.org/x/arch v0.8.0 // indirect
golang.org/x/crypto v0.37.0 // indirect
golang.org/x/mod v0.24.0 // indirect
golang.org/x/oauth2 v0.28.0 // indirect
golang.org/x/term v0.31.0 // indirect
golang.org/x/crypto v0.43.0 // indirect
golang.org/x/mod v0.28.0 // indirect
golang.org/x/oauth2 v0.30.0 // indirect
golang.org/x/term v0.36.0 // indirect
golang.org/x/time v0.11.0 // indirect
golang.org/x/tools v0.32.0 // indirect
golang.org/x/tools v0.37.0 // indirect
golang.org/x/tools/godoc v0.1.0-deprecated // indirect
gonum.org/v1/gonum v0.16.0 // indirect
google.golang.org/api v0.226.0 // indirect
google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250428153025-10db94c68c34 // indirect
google.golang.org/grpc v1.72.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250804133106-a7a43d27e69b // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251007200510-49b9836ed3ff // indirect
google.golang.org/grpc v1.76.0 // indirect
google.golang.org/protobuf v1.36.10 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
Expand Down
Loading
Loading