func (client *k8sClient) ListApplications() in v1.NamespaceAll: kube-api CPU usage on big installations (~8K applications)

[amorozkin]

Hi!

1. Since v0.15.0 calling GetApplication() triggers in turn ListApplications() in v1.NamespaceAll.
2. GetApplication() is called multiple times in row.

These all caues very high resource (CPU mainly) usage on k8s masters as a result.
In our case (Muilt-cluster Argocd instance v2.14.10 + 8K argоcd apps) it looks like that on 6 CPU core masters (dedicated argocd k8s cluster, v1.29):

K8S API logs (for 1 minute):

We are not using "Application in any namespase" feature currently so it's ok for us to switch back to using v0.14.0.

As of 0.15.0+ versions, may be there is a need for optimization - say adding redis cache on application side (with 1 min TTL for ListApplications() for example) ?

[chengfang]

#1112 tries to fix this issue by introducing a new param AppNamespace so that image-updater will only searches apps in that namespace. Will that help mitigate the performance issue your'are seeing?

[amorozkin]

No, I think it wouldn't in the way it's going to be implemented.
Root cause of performance issue is here - GetApplication() now uses List() instead of Get(.., appName ) as version 0.14- did.
Getting full applications list every time when you have to get just a particlular one is a very expensive operation on "big" argocd installations.

#1112 still uses List().

If there is no way to roll back to using Get() app by its name - i see only two optimization options:

1. Adding cache for ListApplications() function as already been propoused
2. Refactor the code - add HahdleApplicationList() function that would be called periodically and would get apps list returned by ApplicationList() as its argument and then in the loop would process every app. In this way there would be no need to make List() applications every time you have to get just one of them.

[grahamalama]

@amorozkin @chengfang what if we did something like

// GetApplication retrieves an application by name, either in a specific namespace or all namespaces depending on client configuration.
func (client *k8sClient) GetApplication(ctx context.Context, appName string) (*v1alpha1.Application, error) {
	// List all applications across configured namespace or all namespaces (using empty labelSelector)
	if *client.appNamespace != v1.NamespaceAll {
		return client.kubeClient.ApplicationsClientset.ArgoprojV1alpha1().Applications(*client.appNamespace).Get(ctx, appName, v1.GetOptions{})
	}
	return client.getApplicationInAllNamespaces(ctx, appName)
}

where getApplicationInAllNamespaces is the current implementation of GetApplication.

This would at least solve for the use cases that aren't using "Application in any namespace" in the short term (both from the performance concerns in this issue and the RBAC stuff in #902), but would punt on some of the larger refactors or adding a cache into the mix.

[amorozkin]

@grahamalama Agree, it could be a workaround in the short term for "Applications in single namespace" cases.

[chengfang]

@grahamalama that looks a good option with low impact.