fix: reuse global TracerProvider to avoid memory leak

[mweibel]

/kind bug
What this PR does / why we need it:
Fixes a memory leak occurring since v1.18.0 by reusing the global tracer provider instead of re-instantiating it many times (for each request basicly). This also ensures tracing provider is set to a valid value only when enableTracing is set.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5410

Special notes for your reviewer:
Global TraceProvider is set in:

cluster-api-provider-azure/pkg/ot/traces.go

Lines 37 to 42 in b32f0c6

	func RegisterTracing(ctx context.Context, log logr.Logger) error {
	tp, err := OTLPTracerProvider(ctx)
	if err != nil {
	return err
	}
	otel.SetTracerProvider(tp)

Which is called in main:

cluster-api-provider-azure/main.go

Lines 354 to 359 in b32f0c6

	if enableTracing {
	if err := ot.RegisterTracing(ctx, setupLog); err != nil {
	setupLog.Error(err, "unable to initialize tracing")
	os.Exit(1)
	}
	}

It would be great if somebody could ensure tracing still works as intended - I did enable tracing to verify it works and it looks okay to me but I haven't verified it in great detail.

TODOs:

• squashed commits
• includes documentation
• adds unit tests
• cherry-pick candidate

Release note:

fixes potential OOM due to tracing (even if tracing not enabled)

[k8s-ci-robot]

Hi @mweibel. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mweibel]

I think @nawazkh did some work with tracing some time ago - pinging for visibility.

[codecov]

Codecov Report

Attention: Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.

Project coverage is 52.86%. Comparing base (b32f0c6) to head (f4efea5).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/ot/traces.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5540      +/-   ##
==========================================
- Coverage   52.93%   52.86%   -0.07%     
==========================================
  Files         272      272              
  Lines       29485    29474      -11     
==========================================
- Hits        15607    15582      -25     
- Misses      13061    13080      +19     
+ Partials      817      812       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jackfrancis]

Do these reflect changing things that were first introduced in 1.18? As far as I can tell these changes were introduced in the v1.17.0 release:

• https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/5144/files#diff-993cbd654628d8e1f18959c6e6ce555d7f3a96408639ddec70a62004666f68a5

cc @nojnhuh

[nojnhuh]

That PR only exists for 1.18 not backported AFAICT: abd8819

I don't really know if this is all totally "correct" or "the right way to do things"

Turns out it is neither. This looks much better! I can still see the traces.

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: ae3e77ebeba255197172487dd01e450f416073ce

[nojnhuh]

/cherry-pick release-1.18
/cherry-pick release-1.19

[k8s-infra-cherrypick-robot]

@nojnhuh: once the present PR merges, I will cherry-pick it on top of release-1.18, release-1.19 in new PRs and assign them to you.

In response to this:

/cherry-pick release-1.18
/cherry-pick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mboersma]

/lgtm

I tested with tilt and traces are still showing up.

[mboersma]

/ok-to-test

😄

[nojnhuh]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nojnhuh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nojnhuh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-infra-cherrypick-robot]

@nojnhuh: new pull request created: #5544

In response to this:

/cherry-pick release-1.18
/cherry-pick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-infra-cherrypick-robot]

@nojnhuh: new pull request created: #5545

In response to this:

/cherry-pick release-1.18
/cherry-pick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.