Condition indicating that API token is wrong gets removed when rate limit exceeded

[janiskemper]

/kind bug

What steps did you take and what happened:
There are certain cases where our main reconcile function of HCloudMachine doesn't return an error, even if the Hetzner API gave us an error, e.g. when we are hitting rate limits. The reason being that we don't want to end up in a loop where we hit the API again and again. However, we have currently a defer statement that checks for unauthorized errors. In case we don't return an error anymore, e.g. when rate limit exceeded, then in the defer statement we conclude that the unauthorized error is gone - which is not correct.

What did you expect to happen:
We have two options:
a) handle this specific case in some way
b) move the handling of the unauthorized error to the respective function.

I vote for using b). It is currently very hard to understand and the only advantage is that whenever we add API calls in the reconcile loop, we don't need to worry about the unauthorized error handling. It would be a lot cleaner if we just handled the unauthorized error once in the first place where we do an API call and that's it.

We can check all relevant controllers for similar cases.

[janiskemper]

@Dhairya-Arora01 this isn't solved yet somehow. Can you ensure before closing this that we have no other objects where something similar applies? E.g. the HetznerCluster?

[Dhairya-Arora01]

In HetznerCluster, we are still good with handling unauthorized error in defer statement, because the first API call is for findNetwork(), where we don't silence the rate limit error:

cluster-api-provider-hetzner/pkg/services/hcloud/network/network.go

Lines 163 to 164 in 7b9f7e4

	hcloudutil.HandleRateLimitExceeded(s.scope.HetznerCluster, err, "ListNetworks")
	return nil, fmt.Errorf("failed to list networks: %w", err)

Same during deletion, we silence the rate limit error:

cluster-api-provider-hetzner/pkg/services/hcloud/network/network.go

Lines 142 to 143 in 7b9f7e4

	if err := s.scope.HCloudClient.DeleteNetwork(ctx, &hcloud.Network{ID: id}); err != nil {
	hcloudutil.HandleRateLimitExceeded(s.scope.HetznerCluster, err, "DeleteNetwork")