telemetry(auth): move debugging telemetry upstream for aws_loadCredentials

[Hweinstock]

Problem

Follow up to: #6499

• The debounce efforts were not aggressive enough.
• The debugging efforts determined this is due to auth.listConnections being spammed, but unfortunately this function is called in many places, making it difficult to track. Therefore, more investigative work is required.

Solution

• Move debounce up to getCredentialProviderNames and convert it to throttle (described below). Note that we can't move it up to auth.listConnections because it is not a "pure" function and has expected side effects. Moving this up allows us to throttle both emit calls to this metric more easily.
• Put the function_call debugging telemetry inside the throttle to make it less noisy.
• Add more withTelemetryHelper decorators and manual wrapping.
• Increase aggressiveness in rate-limiting since aws_loadCredentials is still being emitted a lot.

Throttle Notes

• throttle is similar to debounce, but it returns the last cached result immediately on call. See the tests and docstring for more information.
• We want throttle over debounce because we require the function to resolve a result immediately. Delaying these results could degrade the UX when authenticating or using the toolkit as other pieces may be waiting on it.

Alternative Solution

• rather than throttle, continue using debounce, and make the delay more aggressive.
  • pro: keeps telemetry code as the only code affected. throttle does affect the behavior of the program.
  • con: will continue to spam function_call metrics. Which is now much noisier than aws_loadCredentials (but is temporary).

Future Work

• Investigate more ergonomic way to wrap functions in function_call metric. Non-class methods can't use decorator, but perhaps we can write something else.
• monitor telemetry following release to continue investigation.
• found some dead code in the auth folder that could potentially be removed.

---

• Treat all work as PUBLIC. Private feature/x branches will not be squash-merged at release time.
• Your code changes must meet the guidelines in CONTRIBUTING.md.
• License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jpinkney-aws]

Does removing the debounce here mean we think this piece is no longer the issue? the debounce is moved higher up the stack

[Hweinstock]

yep, and now behaves slightly differently.

[jpinkney-aws]

TBH i'm not too concerned about having to put debugging telemetry other places, I'm more scared of events like:

con: will continue to spam function_call metrics. Which is now much noisier than aws_loadCredentials (but is temporary).

considering how many times aws_loadCredentials was called

[Hweinstock]

Yeah, I agree. My thinking is that first priority is to stop spamming telemetry right now. Then, add utility for debugging the root issue. I don't think its worth the tradeoff to put more burden on telemetry to potentially debug this faster.

[nkomonen-amazon]

We'll also want to indicate that function calls will be throttled regardless of difference in input arguments

[nkomonen-amazon]

I think we should add a test that asserts the first function call is executed and resolved before the first interval even completes. I.e if we tick 1ms while the delay is 3ms, we will get a result

[justinmk3]

This is a lot of change, in a somewhat high-risk area,

• Move debounce up to getCredentialProviderName

That's the main change, right? And the other changes in this PR could be dropped.

[Hweinstock]

There is not enough evidence to suggest that auth.listConnections being spammed indicates a bug. Therefore, we can target the source of the issue by simply debouncing aggressively and removing debugging metrics to limit load on telemetry. #6548