stats/opentelemetry: restore the changes from #8342 and fix the flaky test.

[Pranjali-2501]

Fixes #8700

This PR re-lands the changes from #8342 , which were reverted due to a flaky test. It cherry-picks the original commits and adds a fix for the underlying race condition that caused the test TestTraceSpan_WithRetriesAndNameResolutionDelay to flake.

Problem

As mentioned here, the test was flaky because of a race condition between the load balancing policy creating a ready picker and the RPC attempting to pick a connection. If the picker became ready before the first Pick attempt, the RPC would not be delayed, the "Delayed LB pick complete" event would not be emitted, and the test would fail.

Fix

To solve the race condition, the test now uses a custom stub balancer and returns a blocking picker that guarantees the RPC will wait for a connection. As soon as the RPC attempts to Pick and is confirmed to be in a waiting state, the balancer then provides a valid, "non-blocking" picker, allowing the RPC to succeed. This sequence reliably triggers the DelayedPickComplete event.

RELEASE NOTES:

• stats/opentelemetry: Retry attempts (grpc.previous-rpc-attempts) are now recorded as span attributes for non-transparent client retries.

[codecov]

Codecov Report

❌ Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.23%. Comparing base (b6f89f7) to head (0d0d007).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
stats/opentelemetry/client_tracing.go	78.57%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8923      +/-   ##
==========================================
+ Coverage   83.14%   83.23%   +0.08%     
==========================================
  Files         417      417              
  Lines       32937    32943       +6     
==========================================
+ Hits        27387    27420      +33     
+ Misses       4117     4098      -19     
+ Partials     1433     1425       -8     

Files with missing lines	Coverage Δ
stats/opentelemetry/opentelemetry.go	75.12% <ø> (ø)
stats/opentelemetry/trace.go	90.32% <ø> (-1.99%)	⬇️
stats/opentelemetry/client_tracing.go	86.36% <78.57%> (-2.73%)	⬇️

... and 22 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Pranjali-2501]

/gemini review

[gemini-code-assist]

Code Review

This pull request successfully re-lands the changes to record retry attempts as OpenTelemetry span attributes and fixes a flaky test that caused the original revert. The fix for the race condition in the test TestTraceSpan_WithRetriesAndNameResolutionDelay using a custom blocking balancer is well-implemented and should reliably prevent flakiness. The refactoring of the tracing logic to move attempt counting to callInfo and handle client-specific attributes in client_tracing.go is a good improvement. I've found one issue in the test validation logic that should be addressed.

[arjan-bal]

Instead of atomically swapping out the subchannel, a new picker object should be created and used while calling UpdateState.

[arjan-bal]

Instead of implementing a partial LB policy, we should wrap an existing LB policy like pickfirst and delegate calls to it. We can intercept call from pickfirst to UpdateState wrap the picker to detect calls to pick.

[arjan-bal]

The Pick method can be called multiple times concurrently. This implementation can result in a panic since it can result in closing a channel multiple times. Instead of this, you can use an Event.
https://github.com/grpc/grpc-go/blob/master/internal/grpcsync/event.go