feat: Allow cluster-specific QPS and Burst limit configuration for the target clusters

[luwangVMW]

Summary

Currently, Argo CD uses a set of global environment variables (ARGOCD_K8S_CLIENT_QPS and ARGOCD_K8S_CLIENT_BURST, maps to controller.k8s.client.qps and controller.k8s.client.burst in argocd-cmd-params-cm) to configure the rate-limiting (QPS and Burst) of the Kubernetes REST clients for all managed destination clusters.
We propose introducing the ability to define cluster-specific QPS and Burst limits, allowing operators to fine-tune rate-limiting on a per-cluster basis.

Motivation

In large-scale multi-cluster environments managed by Argo CD, target clusters can vary drastically in their infrastructure capabilities, network topology, and workload density.
A "one-size-fits-all" global QPS/Burst configuration introduces several critical trade-offs:

1. Under-provisioned/Edge Clusters: Small or edge clusters can easily be overwhelmed by the default QPS of 50 and Burst of 100 during sync operations or intense cache discovery. This leads to API Server throttling, elevated latencies, or even API server crashes.
2. High-Performance/Scale Clusters: Large-scale enterprise clusters containing thousands of resources require high QPS/Burst limits to accelerate reconciliation, reduce Sync times, and handle rapid automatic rollouts (Auto-Sync).
3. The "Lowest Common Denominator" Problem: Operators are currently forced to either set a very low global QPS (slowing down synchronization for high-performance clusters) or a high global QPS (risking the stability of weak edge clusters).
   Allowing granular, per-cluster overrides would solve this bottleneck, bringing better stability to fragile clusters and higher performance to massive ones.

Proposal

I propose extending the Cluster Secret configuration to allow user to specify custom QPS and Burst values. If a Cluster Secret does not define these specific values, Argo CD will seamlessly fall back to the global defaults configured in the ConfigMap (argocd-cmd-params-cm).
To implement this on the Cluster Secret, we have identified two viable options:

Option 1: Leveraging Cluster Secret metadata.annotations (Non-intrusive metadata expansion)
This approach reads the custom limits directly from the Cluster Secret's annotations, which are already automatically deserialized into the Cluster.Annotations struct by Argo CD.

apiVersion: v1
kind: Secret
metadata:
  name: edge-cluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
  annotations:
    # Option 1: Define limits via annotations
    argocd.argoproj.io/k8s-client-qps: "10"
    argocd.argoproj.io/k8s-client-burst: "20"

Option 2: Extending Cluster Secret data.config JSON Schema (First-Class API Field)
This approach extends the ClusterConfig struct inside pkg/apis/application/v1alpha1/types.go by adding native typed fields. Since the Cluster Secret's data.config string is unmarshaled directly into ClusterConfig, this enables defining limits natively inside the Config JSON.

apiVersion: v1
kind: Secret
metadata:
  name: edge-cluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: edge-cluster
  server: https://edge-cluster.internal
  config: |
    {
      "bearerToken": "example-token",
      // Option 2: Define limits natively inside the config payload
      "qps": 10,
      "burst": 20
    }

Backward Compatibility & Fallback Mechanism (Crucial)
To guarantee 100% backward compatibility, both options will follow a strict fallback order during Kubernetes client initialization.
If neither Option 1 nor Option 2 is specified in the Cluster Secret, Argo CD will default to the global limits defined in argocd-cmd-params-cm (e.g., controller.k8s.client.qps and controller.k8s.client.burst).

I would love to get feedback from the maintainers on the preferred design option. I am also open to submitting a Pull Request for this feature once we align on the approach!

[github-actions]

🎯 Triage report

This is a well-motivated feature request to support per-cluster QPS/Burst overrides on Kubernetes REST clients, addressing the "lowest common denominator" problem in large multi-cluster Argo CD deployments. The author presents two concrete implementation options (annotations vs. ClusterConfig struct fields) and explicitly requests maintainer input before submitting a PR. The scope touches the cluster management and API layers.

📊 Assessment

Dimension	Value	Reasoning
Type	Feature	New capability, no existing equivalent
Labels	component:cluster-management, type:scalability, proposal:required	Affects cluster mgmt subsystem; directly a scalability concern; design choice between two options warrants prior proposal/discussion
Coding agent	Needs more info	Design decision (Option 1 vs. Option 2) must be resolved by maintainers first; once chosen, implementation is well-scoped

🔗 Similar issues

No exact duplicates found. Loosely related: #17257 (related, closed) — a regression where reconciliation dropped to near zero after an upgrade, demonstrating real-world impact of suboptimal global QPS settings in large deployments.

💡 Notes and suggestions

Design recommendation for maintainers:

• Option 2 (ClusterConfig struct extension) is the more idiomatic approach for Argo CD. It makes QPS/Burst a first-class typed API field, improves discoverability, and avoids relying on annotation string parsing. It also integrates cleanly with argocd cluster add --config.
• Option 1 (annotations) is non-intrusive but is less discoverable, harder to validate (stringly-typed), and mixes operational configuration with Kubernetes metadata semantics.

Key implementation points (once design is settled):

1. Extend ClusterConfig in pkg/apis/application/v1alpha1/types.go with QPS float32 and Burst int (omitempty).
2. In util/db/cluster.go, after unmarshaling the cluster secret config, apply cluster-level overrides to the REST client config before returning.
3. Run make codegen to regenerate manifests and API glue code.
4. Add unit tests for the fallback logic (cluster value → global default).
5. Update docs under docs/operator-manual/cluster-management.md.

Backward compatibility: The fallback to global argocd-cmd-params-cm values when cluster-level fields are absent (zero-value or omitted) preserves full backward compatibility.

Relevant code locations:

• pkg/apis/application/v1alpha1/types.go — ClusterConfig struct
• util/db/cluster.go — cluster secret deserialization
• controller/appcontroller.go — where k8s clients are initialized per cluster

Generated by Agentic Triage for #29001 · 33.1 AIC · ⌖ 5.95 AIC · ⊞ 5.7K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@e15e57b40918dbca11b350c55d02ab61934afa75