|
| 1 | +# Argo CD Agent Hub and Spoke Cluster Setup Documentation Using Operator |
| 2 | +This document outlines the process of setting up Argo CD Agent in a hub and spoke cluster architecture. This configuration allows a central Argo CD instance (the hub) to manage applications deployed across multiple remote Kubernetes clusters (the spokes) through the Argo CD-agent. |
| 3 | +## Prerequisites |
| 4 | +Before proceeding with the setup, ensure you have the following: |
| 5 | + |
| 6 | +- A running OpenShift/Kubernetes cluster designated as the hub, with Argo CD/OpenShift GitOps Operator installed. |
| 7 | +- One or more Kubernetes clusters designated as spokes, Argo CD/OpenShift GitOps Operator installed. |
| 8 | +- `oc` binary configured to access both hub and spoke clusters. |
| 9 | +- Argo CD instance must be [cluster scoped](https://argocd-operator.readthedocs.io/en/stable/usage/basics/#cluster-scoped-instance). |
| 10 | +- [Apps in any Namespace](https://argocd-operator.readthedocs.io/en/latest/usage/apps-in-any-namespace/) must be enabled for Hub Cluster. |
| 11 | +- `argocd-agentctl` binary (for non-production scenario) |
| 12 | + |
| 13 | + |
| 14 | +## Hub Cluster Setup |
| 15 | + |
| 16 | +Hub cluster will be installed in argocd namespace |
| 17 | + |
| 18 | +Before you install the principal's manifest, you will need to |
| 19 | +- Create a TLS secret containing the public CA certificate used by Argo CD-agent components |
| 20 | +- Create a TLS secret containing the certificate and private key used by the principal's gRPC service |
| 21 | +- Create a TLS secret containing the certificate and private key used by the principal's resource proxy |
| 22 | +- Create a secret containing the private RSA key used to sign JWT issued by the principal |
| 23 | + |
| 24 | +For non-production scenarios, you can use the argocd-agentctl CLI's PKI tooling. Please be advised that this is not a production-grade setup, and that important pieces of the PKI, such as the CA's private key, will be stored unprotected on your control plane cluster. |
| 25 | + |
| 26 | +Create the PKI on the principal: |
| 27 | + |
| 28 | +``` |
| 29 | +argocd-agentctl --principal-context <control plane context> pki init |
| 30 | +``` |
| 31 | + |
| 32 | +Issue the certificate for the principal's gRPC service: |
| 33 | +The gRPC service is the service the agents will connect to. This service usually needs to be exposed to the outside world, or at least from your other clusters where the agents are running on. Typically, this means the service is exposed by a load balancer or a node port. |
| 34 | + |
| 35 | +``` |
| 36 | +argocd-agentctl pki issue principal \ |
| 37 | + --principal-context <control plane context> \ |
| 38 | + --ip <ip addresses of principal> \ |
| 39 | + --dns <dns names of principal> |
| 40 | +``` |
| 41 | + |
| 42 | +For the --ip and --dns values, you want to specify all addresses and DNS names that the principal will be reachable at for the agents, for example --ip 5.5.5.5 --dns my-principal.example.com. If there is a load balancer in front of the principal, make sure it will pass through traffic to the principal - otherwise, client certificates might not be used for authentication. |
| 43 | + |
| 44 | +Issue the certificate for the principal's resource proxy: |
| 45 | +The principal's resource proxy needs to be reachable by your Argo CD API server (argocd-server), which usually runs in the same cluster as the principal. Typically, this service should not be reachable outside the cluster and is exposed using a Kubernetes Service. |
| 46 | + |
| 47 | +``` |
| 48 | +argocd-agentctl pki issue resource-proxy \ |
| 49 | + --principal-context <control plane context> \ |
| 50 | + --ip <ip addresses of resource proxy> \ |
| 51 | + --dns <dns names of principal> |
| 52 | +``` |
| 53 | + |
| 54 | +Deploy principal on hub cluster using argocd-operator/gitops-operator using Argo CD CR given below |
| 55 | + |
| 56 | +``` |
| 57 | +apiVersion: argoproj.io/v1beta1 |
| 58 | +kind: ArgoCD |
| 59 | +metadata: |
| 60 | + name: argocd |
| 61 | +spec: |
| 62 | + controller: |
| 63 | + enabled: false |
| 64 | + argoCDAgent: |
| 65 | + principal: |
| 66 | + enabled: true |
| 67 | + allowedNamespaces: |
| 68 | + - "*" |
| 69 | + jwtAllowGenerate: true |
| 70 | + auth: "mtls:CN=([^,]+)" |
| 71 | + logLevel: "trace" |
| 72 | + image: "ghcr.io/argoproj-labs/argocd-agent/argocd-agent:latest" |
| 73 | + sourceNamespaces: |
| 74 | + - "agent-managed" |
| 75 | + - "agent-autonomous" |
| 76 | +``` |
| 77 | + |
| 78 | +The above CR should create all the necessary resource for Argo CD as well as argocd-agent principal in argocd namespace. |
| 79 | + |
| 80 | +Create argocd-redis secret, because principal looks for it to fetch redis authentication details. |
| 81 | + |
| 82 | +``` |
| 83 | +oc create secret generic argocd-redis -n argocd --from-literal=auth="$(oc get secret argocd-redis-initial-password -n argocd -o jsonpath='{.data.admin\.password}' | base64 -d)" |
| 84 | +``` |
| 85 | + |
| 86 | +## Setting up agent workload cluster |
| 87 | + |
| 88 | +### Configure Argo CD for Agent |
| 89 | + |
| 90 | +Argo CD instance on Agent cluster |
| 91 | + |
| 92 | +Creating Argo CD instance for Workload/spoke cluster. |
| 93 | +``` |
| 94 | + apiVersion: argoproj.io/v1beta1 |
| 95 | + kind: ArgoCD |
| 96 | + metadata: |
| 97 | + name: argocd |
| 98 | + spec: |
| 99 | + server: |
| 100 | + enabled: false |
| 101 | +``` |
| 102 | + |
| 103 | +Create redis secret using below command for agent deployment |
| 104 | +``` |
| 105 | +kubectl create secret generic argocd-redis -n <workload namespace> --from-literal=auth="$(kubectl get secret argocd-redis-initial-password -n <argocd-namespace> -o jsonpath='{.data.admin\.password}' | base64 -d)" |
| 106 | +``` |
| 107 | + |
| 108 | +### Configure Agent in managed mode |
| 109 | + |
| 110 | +Before installing agent resources create |
| 111 | +- a TLS secret containing the issued certificate for agent |
| 112 | + |
| 113 | +Create the PKI on the agent: |
| 114 | +Run this command while connected to principal |
| 115 | +``` |
| 116 | +argocd-agentctl pki issue agent <agent-name> --principal-context <principal context> --agent-context <workload context> --agent-namespace <workload namespace> --upsert |
| 117 | +``` |
| 118 | + |
| 119 | +Apply the installation manifests for Argo CD-agent agent |
| 120 | +``` |
| 121 | +oc apply -n $(workload-namespace) -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/agent?ref=main' |
| 122 | +``` |
| 123 | +This should create all the required agent related resources. |
| 124 | + |
| 125 | +Note: If installation is done on other than `default` namespace, run the following command to update cluster-role with correct namespace |
| 126 | + |
| 127 | +``` |
| 128 | +kubectl patch clusterrolebinding argocd-agent-agent --type='json' -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "<workload-namespace>"}]' |
| 129 | +``` |
| 130 | + |
| 131 | + |
| 132 | +Update the configMap with name `argocd-agent-params` with parameters related to agent.mode,agent.creds, agent.namespace, agent.server.address. |
| 133 | +``` |
| 134 | + agent.keep-alive-ping-interval: 50s |
| 135 | + agent.mode: managed |
| 136 | + agent.creds: mtls:any |
| 137 | + agent.tls.client.insecure: "false" |
| 138 | + agent.tls.root-ca-path: "" |
| 139 | + agent.tls.client.cert-path: "" |
| 140 | + agent.tls.client.key-path: "" |
| 141 | + agent.log.level: info |
| 142 | + agent.namespace: <workload-namespace> |
| 143 | + agent.server.address: <argocd-principal-server> |
| 144 | + agent.server.port: 443 |
| 145 | + agent.metrics.port: 8181 |
| 146 | + agent.healthz.port: "8002" |
| 147 | + agent.tls.root-ca-secret-name: argocd-agent-ca |
| 148 | + agent.tls.secret-name: argocd-agent-client-tls |
| 149 | +``` |
| 150 | +Also Update RBAC, rolebinding/clusterrolebinding with `workload-namespace`, if pod is facing rbac issues. |
| 151 | + |
| 152 | + |
| 153 | + |
| 154 | +### Configure Agent in Autonomous mode |
| 155 | + |
| 156 | +Before installing agent resources create |
| 157 | +Create a TLS secret containing the issued certificate for agent |
| 158 | + |
| 159 | +Create the PKI on the agent: |
| 160 | +Run this command while connected to principal |
| 161 | +``` |
| 162 | +argocd-agentctl pki issue agent <agent-name> --principal-context <principal context> --agent-context <workload context> --agent-namespace argocd --upsert |
| 163 | +``` |
| 164 | + |
| 165 | +Apply the installation manifests for argocd agent |
| 166 | +``` |
| 167 | +oc apply -n argocd -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/agent?ref=main' |
| 168 | +``` |
| 169 | +This should create all the required agent related resources. |
| 170 | + |
| 171 | +Note: If installation is done on other than `default` namespace, run the following command to update cluster-role with correct namespace |
| 172 | + |
| 173 | +``` |
| 174 | +kubectl patch clusterrolebinding argocd-agent-agent --type='json' -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "<workload-namespace>"}]' |
| 175 | +``` |
| 176 | + |
| 177 | + |
| 178 | +Update the configMap with name `argocd-agent-params` with parameters related to agent.mode,agent.creds, agent.namespace, agent.server.address. |
| 179 | +``` |
| 180 | +data: |
| 181 | + agent.keep-alive-ping-interval: 50s |
| 182 | + agent.tls.client.insecure: 'false' |
| 183 | + agent.server.port: '443' |
| 184 | + agent.tls.root-ca-path: '' |
| 185 | + agent.tls.client.cert-path: '' |
| 186 | + agent.tls.root-ca-secret-name: argocd-agent-ca |
| 187 | + agent.tls.secret-name: argocd-agent-client-tls |
| 188 | + agent.mode: autonomous |
| 189 | + agent.log.level: info |
| 190 | + agent.namespace: argocd |
| 191 | + agent.server.address: <principal server address> |
| 192 | + agent.metrics.port: '8181' |
| 193 | + agent.tls.client.key-path: '' |
| 194 | + agent.healthz.port: '8002' |
| 195 | + agent.creds: 'mtls:any' |
| 196 | +``` |
| 197 | + |
| 198 | + |
| 199 | +#### Troubleshooting |
| 200 | +___ |
| 201 | + |
| 202 | +1. If pod fails to come up with error |
| 203 | +``` |
| 204 | +time="2025-07-30T14:58:33Z" level=warning msg="INSECURE: Not verifying remote TLS certificate" |
| 205 | +time="2025-07-30T14:58:33Z" level=info msg="Loading client TLS certificate from secret argocd/argocd-agent-client-tls" |
| 206 | +[FATAL]: Error creating remote: unable to read TLS client from secret: could not read TLS secret argocd/argocd-agent-client-tls: secrets "argocd-agent-client-tls" is forbidden: User "system:serviceaccount:argocd:argocd-agent-agent" cannot get resource "secrets" in API group "" in the namespace "argocd" |
| 207 | +``` |
| 208 | + |
| 209 | +update the ClusterRoleBinding to update the subject namespace to workload-namespace. |
| 210 | + |
| 211 | +``` |
| 212 | +kubectl patch clusterrolebinding argocd-agent-agent --type='json' -p='[{"op": "replace", "path": "/subjects/0/namespace", "value": "<workload-namespace>"}]' |
| 213 | +``` |
| 214 | + |
| 215 | +2. If facing error with appProject |
| 216 | + |
| 217 | +``` |
| 218 | +Unable to create application: app is not allowed in project "default", or the project does not exist |
| 219 | +``` |
| 220 | +refer to doc for [AppProject Synchronization](https://argocd-agent.readthedocs.io/latest/user-guide/appprojects/#managed-agent-mode). |
0 commit comments