[Issue #9021] Add retries if Login.gov fails when we ask for token details

[mdragon]

Summary

Fixes #9021

Changes proposed

Introduce a retry of up to 3 times for our call to Login.gov to fetch information about the token holder

Context for reviewers

We see occasional failures, once every few weeks, where upon a callback from Login.gov our request back to Login to get information about the holder of the token fails with a unspecified 500 error. This currently results in the user not successfully signing in. This change would give that user an opportunity to still get logged in.

Validation steps

• Unit tests pass
• Login still works in Staging after deploy following merge to main

[chouinar]

Rather than setting the retries by using the dict directly, what if we add a param to the add token function to do it?

Suggested change

	self.retries: dict[str, int] = {}
	def add_token_response(self, code: str, response: OauthTokenResponse) -> None:
	self.responses[code] = response
	self.retries: dict[str, int] = {}
	def add_token_response(self, code: str, response: OauthTokenResponse, retry_count: int = 0) -> None:
	self.responses[code] = response
	self.retries[code] = retry_count

Then that simplifies the logic below a bit as it'll always be set.

[chouinar]

I think you meant to do this - as written it would always call 3 times as the loop never breaks before 3 tries.

Suggested change

	else:
	logger.info(
	"Retrying call to Login.gov after receiving error",
	extra={"tries": tries, "limit": limit},
	)
	continue
	else:
	logger.info(
	"Retrying call to Login.gov after receiving error",
	extra={"tries": tries, "limit": limit},
	)
	break

Alternatively, if we wanted to use tenacity, seems they have a TryAgain exception we could use - although I don't think it's that big a deal.

[chouinar]

Probably should update this as with the retries it isn't quite right?