Skip to content

Commit 2de8e01

Browse files
mikhail-klimkomikhail-klimko
authored
feat: inCluster onprem runtime (#378)
1 parent 7470f71 commit 2de8e01

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+2183
-876
lines changed

‎charts/cf-runtime/Chart.yaml‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
apiVersion: v2
22
description: A Helm chart for Codefresh Runner
33
name: cf-runtime
4-
version: 3.0.8
4+
version: 4.0.0
55
keywords:
66
- codefresh
77
- runner
@@ -14,11 +14,15 @@ maintainers:
1414
url: https://codefresh-io.github.io/
1515
annotations:
1616
artifacthub.io/changes: |
17-
- kind: fixed
18-
description: Fix security context for dind-lv-monitor (OpenShift support)
17+
- kind: added
18+
description: Add inCluster onprem runtime option (Codefresh On-Premises only)
1919
- kind: changed
20-
description: Use rootless cli image for runtime patch job
20+
description: Change templates to lib components
21+
- kind: changed
22+
description: Update cf-common dependency subchart
23+
- kind: added
24+
description: Post-delete hook to delete runtime from platform (Codefresh On-Premises only)
2125
dependencies:
2226
- name: cf-common
2327
repository: https://chartmuseum.codefresh.io/cf-common
24-
version: 0.11.1
28+
version: 0.11.2

‎charts/cf-runtime/README.md‎

Lines changed: 261 additions & 3 deletions
Large diffs are not rendered by default.

‎charts/cf-runtime/README.md.gotmpl‎

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ Helm chart for deploying [Codefresh Runner](https://codefresh.io/docs/docs/insta
1212
- [Upgrade Chart](#upgrade-chart)
1313
- [To 2.x](#to-2x)
1414
- [To 3.x](#to-3x)
15+
- [To 4.x](#to-4x)
1516
- [Architecture](#architecture)
1617
- [Configuration](#configuration)
1718
- [EBS backend volume configuration](#ebs-backend-volume-configuration)
@@ -20,6 +21,7 @@ Helm chart for deploying [Codefresh Runner](https://codefresh.io/docs/docs/insta
2021
- [Volume reuse policy](#volume-reuse-policy)
2122
- [Volume cleaners](#volume-cleaners)
2223
- [Openshift](#openshift)
24+
- [On-premise](#on-premise)
2325

2426
## Prerequisites
2527

@@ -96,6 +98,13 @@ Affected values:
9698
- `global.existingAgentToken` is replaced with `global.agentTokenSecretKeyRef`
9799
- `global.existingDindCertsSecret` is replaced with `global.dindCertsSecretRef`
98100

101+
### To 4.x
102+
103+
This major release adds **agentless inCluster** runtime mode (relevant only for [Codefresh On-Premises](#on-premise) users)
104+
105+
Affected values:
106+
- `runtime.agent` / `runtime.inCluster` / `runtime.accounts` / `runtime.description` are added
107+
99108
## Architecture
100109

101110
[Codefresh Runner architecture](https://codefresh.io/docs/docs/installation/codefresh-runner/#codefresh-runner-architecture)
@@ -408,6 +417,248 @@ oc adm policy add-scc-to-user privileged system:serviceaccount:codefresh:cf-runt
408417
oc adm policy add-scc-to-user privileged system:serviceaccount:codefresh:cf-runtime-volume-provisioner
409418
```
410419

420+
### On-premise
421+
422+
If you have [Codefresh On-Premises](https://artifacthub.io/packages/helm/codefresh-onprem/codefresh) deployed, you can install Codefresh Runner in **agentless** mode.
423+
424+
**What is agentless mode?**
425+
426+
Agent (aka venona) is Runner component which responsible for calling Codefresh API to run builds and create dind/engine pods and pvc objects. Agent can only be assigned to a single account, thus you can't share one runtime across multiple accounts. However, with **agentless** mode it's possible to register the runtime as **system**-type runtime so it's registered on the platform level and can be assigned/shared across multiple accounts.
427+
428+
**What are the prerequisites?**
429+
- You have a running [Codefresh On-Premises](https://artifacthub.io/packages/helm/codefresh-onprem/codefresh) control-plane environment
430+
- You have a Codefresh API token with platform **Admin** permissions scope
431+
432+
433+
### How to deploy agentless runtime when it's on the SAME k8s cluster as On-Premises control-plane environment?
434+
435+
- Enable cluster-level permissions for cf-api (On-Premises control-plane component)
436+
437+
> `values.yaml` for [Codefresh On-Premises](https://artifacthub.io/packages/helm/codefresh-onprem/codefresh) Helm chart
438+
```yaml
439+
cfapi:
440+
...
441+
# -- Enable ClusterRole/ClusterRoleBinding
442+
rbac:
443+
namespaced: false
444+
```
445+
446+
- Set the following values for Runner Helm chart
447+
448+
`.Values.global.codefreshHost=...` \
449+
`.Values.global.codefreshToken=...` \
450+
`.Values.global.runtimeName=system/...` \
451+
`.Values.runtime.agent=false` \
452+
`.Values.runtime.inCluster=true`
453+
454+
> `values.yaml` for [Codefresh Runner](https://artifacthub.io/packages/helm/codefresh-runner/cf-runtime) helm chart
455+
```yaml
456+
global:
457+
# -- URL of Codefresh On-Premises Platform
458+
codefreshHost: "https://myonprem.somedomain.com"
459+
# -- User token in plain text with Admin permission scope
460+
codefreshToken: ""
461+
# -- User token that references an existing secret containing API key.
462+
codefreshTokenSecretKeyRef: {}
463+
# E.g.
464+
# codefreshTokenSecretKeyRef:
465+
# name: my-codefresh-api-token
466+
# key: codefresh-api-token
467+
468+
# -- Distinguished runtime name
469+
# (for On-Premise only; mandatory!) Must be prefixed with "system/..."
470+
runtimeName: "system/prod-ue1-some-cluster-name"
471+
472+
# -- Set runtime parameters
473+
runtime:
474+
# -- (for On-Premise only; mandatory!) Disable agent
475+
agent: false
476+
# -- (for On-Premise only; optional) Set inCluster runtime (default: `true`)
477+
# `inCluster=true` flag is set when Runtime and On-Premises control-plane are run on the same cluster
478+
# `inCluster=false` flag is set when Runtime and On-Premises control-plane are on different clusters
479+
inCluster: true
480+
# -- (for On-Premise only; optional) Assign accounts to runtime (list of account ids; default is empty)
481+
# Accounts can be assigned to the runtime in Codefresh UI later so you can kepp it empty.
482+
accounts: []
483+
# -- Set parent runtime to inherit.
484+
runtimeExtends: []
485+
```
486+
487+
- Install the chart
488+
489+
```console
490+
helm upgrade --install cf-runtime oci://quay.io/codefresh/cf-runtime -f values.yaml --create-namespace --namespace cf-runtime
491+
```
492+
493+
- Verify the runtime and run test pipeline
494+
495+
Go to [https://<YOUR_ONPREM_DOMAIN_HERE>/admin/runtime-environments/system](https://<YOUR_ONPREM_DOMAIN_HERE>/admin/runtime-environments/system) to check the runtime. Assign it to the required account(s). Run test pipeline on it.
496+
497+
498+
### How to deploy agentless runtime when it's on the DIFFERENT k8s cluster than On-Premises control-plane environment?
499+
500+
In this case, it's required to mount runtime cluster's `KUBECONFIG` into On-Premises `cf-api` deployment
501+
502+
- Create the neccessary RBAC resources
503+
504+
> `values.yaml` for [Codefresh Runner](https://artifacthub.io/packages/helm/codefresh-runner/cf-runtime) helm chart
505+
```yaml
506+
extraResources:
507+
- apiVersion: rbac.authorization.k8s.io/v1
508+
kind: Role
509+
metadata:
510+
name: codefresh-role
511+
namespace: '{{ "{{ .Release.Namespace }}" }}'
512+
rules:
513+
- apiGroups: [""]
514+
resources: ["pods", "persistentvolumeclaims", "persistentvolumes"]
515+
verbs: ["list", "watch", "get", "create", "patch", "delete"]
516+
- apiGroups: ["snapshot.storage.k8s.io"]
517+
resources: ["volumesnapshots"]
518+
verbs: ["list", "watch", "get", "create", "patch", "delete"]
519+
- apiVersion: v1
520+
kind: ServiceAccount
521+
metadata:
522+
name: codefresh-runtime-user
523+
namespace: '{{ "{{ .Release.Namespace }}" }}'
524+
- apiVersion: rbac.authorization.k8s.io/v1
525+
kind: RoleBinding
526+
metadata:
527+
name: codefresh-runtime-user
528+
namespace: '{{ "{{ .Release.Namespace }}" }}'
529+
roleRef:
530+
apiGroup: rbac.authorization.k8s.io
531+
kind: Role
532+
name: codefresh-role
533+
subjects:
534+
- kind: ServiceAccount
535+
name: codefresh-runtime-user
536+
namespace: '{{ "{{ .Release.Namespace }}" }}'
537+
- apiVersion: v1
538+
kind: Secret
539+
metadata:
540+
name: codefresh-runtime-user-token
541+
namespace: '{{ "{{ .Release.Namespace }}" }}'
542+
annotations:
543+
kubernetes.io/service-account.name: codefresh-runtime-user
544+
type: kubernetes.io/service-account-token
545+
```
546+
547+
- Set up the following environment variables to create a `KUBECONFIG` file
548+
549+
```shell
550+
NAMESPACE=cf-runtime
551+
CLUSTER_NAME=prod-ue1-some-cluster-name
552+
CURRENT_CONTEXT=$(kubectl config current-context)
553+
554+
USER_TOKEN_VALUE=$(kubectl -n cf-runtime get secret/codefresh-runtime-user-token -o=go-template='{{ `{{.data.token}}` }}' | base64 --decode)
555+
CURRENT_CLUSTER=$(kubectl config view --raw -o=go-template='{{ `{{range .contexts}}{{if eq .name "'''${CURRENT_CONTEXT}'''"}}{{ index .context "cluster" }}{{end}}{{end}}` }}')
556+
CLUSTER_CA=$(kubectl config view --raw -o=go-template='{{ `{{range .clusters}}{{if eq .name "'''${CURRENT_CLUSTER}'''"}}"{{with index .cluster "certificate-authority-data" }}{{.}}{{end}}"{{ end }}{{ end }}` }}')
557+
CLUSTER_SERVER=$(kubectl config view --raw -o=go-template='{{ `{{range .clusters}}{{if eq .name "'''${CURRENT_CLUSTER}'''"}}{{ .cluster.server }}{{end}}{{ end }}` }}')
558+
559+
export -p USER_TOKEN_VALUE CURRENT_CONTEXT CURRENT_CLUSTER CLUSTER_CA CLUSTER_SERVER CLUSTER_NAME
560+
```
561+
562+
- Create a kubeconfig file
563+
564+
```console
565+
cat << EOF > $CLUSTER_NAME-kubeconfig
566+
apiVersion: v1
567+
kind: Config
568+
current-context: ${CLUSTER_NAME}
569+
contexts:
570+
- name: ${CLUSTER_NAME}
571+
context:
572+
cluster: ${CLUSTER_NAME}
573+
user: codefresh-runtime-user
574+
namespace: ${NAMESPACE}
575+
clusters:
576+
- name: ${CLUSTER_NAME}
577+
cluster:
578+
certificate-authority-data: ${CLUSTER_CA}
579+
server: ${CLUSTER_SERVER}
580+
users:
581+
- name: ${CLUSTER_NAME}
582+
user:
583+
token: ${USER_TOKEN_VALUE}
584+
EOF
585+
```
586+
587+
- **Switch context to On-Premises control-plane cluster**. Create k8s secret (via any tool like [ESO](https://external-secrets.io/v0.4.4/), `kubectl`, etc ) containing runtime cluster's `KUBECONFG` created in previous step.
588+
589+
```shell
590+
NAMESPACE=codefresh
591+
kubectl create secret generic dind-runtime-clusters --from-file=$CLUSTER_NAME=$CLUSTER_NAME-kubeconfig -n $NAMESPACE
592+
```
593+
594+
- Mount secret containing runtime cluster's `KUBECONFG` into cf-api in On-Premises control-plane cluster
595+
596+
> `values.yaml` for [Codefresh On-Premises](https://artifacthub.io/packages/helm/codefresh-onprem/codefresh) helm chart
597+
```yaml
598+
cf-api:
599+
...
600+
volumes:
601+
dind-clusters:
602+
enabled: true
603+
type: secret
604+
nameOverride: dind-runtime-clusters
605+
optional: true
606+
```
607+
> volumeMount `/etc/kubeconfig` is already configured in cf-api Helm chart template. No need to specify it.
608+
609+
- Set the following values for Runner helm chart
610+
611+
> `values.yaml` for [Codefresh Runner](https://artifacthub.io/packages/helm/codefresh-runner/cf-runtime) helm chart
612+
613+
`.Values.global.codefreshHost=...` \
614+
`.Values.global.codefreshToken=...` \
615+
`.Values.global.runtimeName=system/...` \
616+
`.Values.runtime.agent=false` \
617+
`.Values.runtime.inCluster=false`
618+
619+
**Important!**
620+
`.Values.global.name` ("system/" prefix is ignored!) should match the cluster name (key in `dind-runtime-clusters` secret created previously)
621+
```yaml
622+
global:
623+
# -- URL of Codefresh On-Premises Platform
624+
codefreshHost: "https://myonprem.somedomain.com"
625+
# -- User token in plain text with Admin permission scope
626+
codefreshToken: ""
627+
# -- User token that references an existing secret containing API key.
628+
codefreshTokenSecretKeyRef: {}
629+
# E.g.
630+
# codefreshTokenSecretKeyRef:
631+
# name: my-codefresh-api-token
632+
# key: codefresh-api-token
633+
634+
# -- Distinguished runtime name
635+
# (for On-Premise only; mandatory!) Must be prefixed with "system/..."
636+
name: "system/prod-ue1-some-cluster-name"
637+
638+
# -- Set runtime parameters
639+
runtime:
640+
# -- (for On-Premise only; mandatory!) Disable agent
641+
agent: false
642+
# -- (for On-Premise only; optional) Set inCluster runtime (default: `true`)
643+
# `inCluster=true` flag is set when Runtime and On-Premises control-plane are run on the same cluster
644+
# `inCluster=false` flag is set when Runtime and On-Premises control-plane are on different clusters
645+
inCluster: false
646+
# -- (for On-Premise only; optional) Assign accounts to runtime (list of account ids; default is empty)
647+
# Accounts can be assigned to the runtime in Codefresh UI later so you can kepp it empty.
648+
accounts: []
649+
# -- (optional) Set parent runtime to inherit.
650+
runtimeExtends: []
651+
```
652+
653+
- Install the chart
654+
655+
```console
656+
helm upgrade --install cf-runtime oci://quay.io/codefresh/cf-runtime -f values.yaml --create-namespace --namespace cf-runtime
657+
```
658+
659+
- Verify the runtime and run test pipeline
660+
661+
Go to [https://<YOUR_ONPREM_DOMAIN_HERE>/admin/runtime-environments/system](https://<YOUR_ONPREM_DOMAIN_HERE>/admin/runtime-environments/system) to see the runtime. Assign it to the required account(s).
411662

412663
{{ template "chart.requirementsSection" . }}
413664

0 commit comments

Comments
 (0)