Skip to content

Commit 63ff275

Browse files
gabemonterogabemontero
authored
Merge pull request #697 from adambkaplan/o11y-prometheus
Inner Loop Observability with Prometheus and Grafana
2 parents 1582f52 + 7e99fd5 commit 63ff275

File tree

10 files changed

+256
-1
lines changed

10 files changed

+256
-1
lines changed

‎developer/openshift/dev_setup.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -241,7 +241,7 @@ install_pipeline_service() {
241241

242242
# Patch the url/branch to target the expected repository/branch
243243
manifest_dir="$(find "$WORK_DIR/environment/compute" -mindepth 1 -maxdepth 1 -type d)"
244-
for app in "pipeline-service" "pipeline-service-storage"; do
244+
for app in "pipeline-service" "pipeline-service-storage" "pipeline-service-o11y"; do
245245
cat << EOF >"$manifest_dir/patch-$app.yaml"
246246
---
247247
apiVersion: argoproj.io/v1alpha1

‎developer/openshift/gitops/argocd/kustomization.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@ kind: Kustomization
55
resources:
66
- pipeline-service.yaml
77
- pipeline-service-storage.yaml
8+
- pipeline-service-o11y.yaml

‎developer/openshift/gitops/argocd/pipeline-service-o11y.yaml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
apiVersion: argoproj.io/v1alpha1
3+
kind: Application
4+
metadata:
5+
name: pipeline-service-o11y
6+
namespace: openshift-gitops
7+
spec:
8+
destination:
9+
namespace: openshift-gitops
10+
server: https://kubernetes.default.svc
11+
source:
12+
path: developer/openshift/gitops/argocd/pipeline-service-o11y
13+
repoURL: https://github.com/openshift-pipelines/pipeline-service.git
14+
targetRevision: main
15+
project: default
16+
syncPolicy:
17+
# Comment this out if you want to manually trigger deployments (using the
18+
# Argo CD Web UI or Argo CD CLI), rather than automatically deploying on
19+
# every new Git commit to your directory.
20+
automated:
21+
prune: true
22+
selfHeal: true
23+
syncOptions:
24+
- CreateNamespace=true
25+
retry:
26+
limit: -1 # number of failed sync attempt retries; unlimited number of attempts if less than 0
27+
backoff:
28+
duration: 10s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
29+
factor: 2 # a factor to multiply the base duration after each failed retry
30+
maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

‎developer/openshift/gitops/argocd/pipeline-service-o11y/README.md

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Pipeline Service Observability
2+
3+
This deploys observability (o11y) components used to configure Prometheus and
4+
Grafana. It utilizes the o11y stack from redhat-appstudio, with modifications
5+
to only deploy what is needed by the Pipeline Service for GitOps-based
6+
development.
7+
8+
## GitOps Inner Loop Development
9+
10+
The `dev_setup.sh` configures Prometheus and Grafana to deploy the Pipeline
11+
Service dashboards. Grafana's UI can be accessed at the
12+
`grafana-access-appstudio-grafana.apps...` route on your cluster.
13+
14+
To iterate on your dashboard (deploying with `--use-current-branch`):
15+
16+
1. Navigate to the Pipeline Service dashboard in Grafana.
17+
2. Add the panels and/or rows for your metrics as desired.
18+
3. Click the "Share" icon to save the dashboard to JSON.
19+
4. Copy the JSON into the pipeline-service-dashboard.json file, located in
20+
`operator/gitops/argocd/grafana/dashboards`.
21+
5. Commit the updated JSON, push to your branch, and verify the dashboard is
22+
updated once ArgoCD syncs your repository.
23+
24+
## Components
25+
26+
### appstudio-prometheus
27+
28+
This configures the Red Hat
29+
[Observability Operator](https://github.com/rhobs/observability-operator) using
30+
the same manifests used by App Studio. The operator consolidates the "cluster
31+
monitoring" and "user workload" monitoring stacks, allowing metrics to combined
32+
in a single data source view.
33+
34+
### appstudio-grafana
35+
36+
This configures the [Grafana Operator](https://github.com/grafana-operator/grafana-operator)
37+
using the same manifests as App Studio. The operator deploys and manages
38+
Grafana instances, and lets dashboards be configured with custom resources and
39+
JSON stored in ConfigMaps. The deployment includes the Pipeline Service
40+
dashboard, which is referenced in-tree.

‎developer/openshift/gitops/argocd/pipeline-service-o11y/appstudio-grafana/allow-argocd-to-manage.yaml

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
kind: ClusterRole
4+
metadata:
5+
name: openshift-gitops-apply-grafana
6+
rules:
7+
- apiGroups:
8+
- integreatly.org
9+
resources:
10+
- grafanas
11+
- grafanadashboards
12+
- grafanadatasources
13+
verbs:
14+
- get
15+
- list
16+
- create
17+
- update
18+
- patch
19+
- delete
20+
---
21+
apiVersion: rbac.authorization.k8s.io/v1
22+
kind: ClusterRoleBinding
23+
metadata:
24+
name: openshift-gitops-apply-grafana
25+
roleRef:
26+
apiGroup: rbac.authorization.k8s.io
27+
kind: ClusterRole
28+
name: openshift-gitops-apply-grafana
29+
subjects:
30+
- kind: ServiceAccount
31+
name: openshift-gitops-argocd-application-controller
32+
namespace: openshift-gitops
33+
---
34+
apiVersion: rbac.authorization.k8s.io/v1
35+
kind: Role
36+
metadata:
37+
name: openshift-gitops-manage-grafana
38+
namespace: appstudio-grafana
39+
rules:
40+
- apiGroups:
41+
- ""
42+
resources:
43+
- secrets
44+
- configmaps
45+
- serviceaccounts
46+
verbs:
47+
- get
48+
- list
49+
- create
50+
- update
51+
- patch
52+
- delete
53+
- apiGroups:
54+
- route.openshift.io
55+
resources:
56+
- routes
57+
verbs:
58+
- get
59+
- list
60+
- create
61+
- update
62+
- patch
63+
- delete
64+
---
65+
apiVersion: rbac.authorization.k8s.io/v1
66+
kind: RoleBinding
67+
metadata:
68+
name: openshift-gitops-manage-grafana
69+
namespace: appstudio-grafana
70+
roleRef:
71+
apiGroup: rbac.authorization.k8s.io
72+
kind: Role
73+
name: openshift-gitops-manage-grafana
74+
subjects:
75+
- kind: ServiceAccount
76+
name: openshift-gitops-argocd-application-controller
77+
namespace: openshift-gitops

‎developer/openshift/gitops/argocd/pipeline-service-o11y/appstudio-grafana/kustomization.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
resources:
5+
- allow-argocd-to-manage.yaml
6+
- namespace.yaml
7+
- https://raw.githubusercontent.com/redhat-appstudio/infra-deployments/main/components/monitoring/grafana/base/grafana-operator.yaml
8+
- https://raw.githubusercontent.com/redhat-appstudio/infra-deployments/main/components/monitoring/grafana/base/grafana-app.yaml
9+
- ../../../../../../operator/gitops/argocd/grafana
10+
commonAnnotations:
11+
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
12+
namespace: appstudio-grafana
13+
configurations:
14+
- https://raw.githubusercontent.com/redhat-appstudio/infra-deployments/main/components/monitoring/grafana/base/cm-dashboard.yaml

‎developer/openshift/gitops/argocd/pipeline-service-o11y/appstudio-grafana/namespace.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
apiVersion: v1
3+
kind: Namespace
4+
metadata:
5+
name: appstudio-grafana
6+
spec: {}

‎developer/openshift/gitops/argocd/pipeline-service-o11y/appstudio-prometheus/allow-argocd-to-manage.yaml

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
kind: ClusterRole
4+
metadata:
5+
name: openshift-gitops-apply-prometheus
6+
rules:
7+
- apiGroups:
8+
- monitoring.rhobs
9+
resources:
10+
- monitoringstacks
11+
- servicemonitors
12+
verbs:
13+
- get
14+
- list
15+
- patch
16+
- create
17+
- update
18+
- delete
19+
- apiGroups:
20+
- monitoring.coreos.com
21+
resources:
22+
- prometheusrules
23+
verbs:
24+
- get
25+
- list
26+
- patch
27+
- create
28+
- update
29+
- delete
30+
---
31+
apiVersion: rbac.authorization.k8s.io/v1
32+
kind: ClusterRoleBinding
33+
metadata:
34+
name: openshift-gitops-apply-prometheus
35+
roleRef:
36+
apiGroup: rbac.authorization.k8s.io
37+
kind: ClusterRole
38+
name: openshift-gitops-apply-prometheus
39+
subjects:
40+
- kind: ServiceAccount
41+
name: openshift-gitops-argocd-application-controller
42+
namespace: openshift-gitops
43+
---
44+
apiVersion: rbac.authorization.k8s.io/v1
45+
kind: Role
46+
metadata:
47+
name: openshift-gitops-manage-secrets
48+
namespace: o11y
49+
rules:
50+
- apiGroups:
51+
- ""
52+
resources:
53+
- secrets
54+
verbs:
55+
- get
56+
- list
57+
- create
58+
- update
59+
- patch
60+
- delete
61+
---
62+
apiVersion: rbac.authorization.k8s.io/v1
63+
kind: RoleBinding
64+
metadata:
65+
name: openshift-gitops-manage-secrets
66+
namespace: o11y
67+
roleRef:
68+
apiGroup: rbac.authorization.k8s.io
69+
kind: Role
70+
name: openshift-gitops-manage-secrets
71+
subjects:
72+
- kind: ServiceAccount
73+
name: openshift-gitops-argocd-application-controller
74+
namespace: openshift-gitops

‎developer/openshift/gitops/argocd/pipeline-service-o11y/appstudio-prometheus/kustomization.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
resources:
5+
- allow-argocd-to-manage.yaml
6+
- https://github.com/redhat-appstudio/infra-deployments/components/o11y/development?ref=main
7+
- https://github.com/redhat-appstudio/infra-deployments/components/monitoring/prometheus/development?ref=main

‎developer/openshift/gitops/argocd/pipeline-service-o11y/kustomization.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
resources:
5+
- appstudio-prometheus
6+
- appstudio-grafana

0 commit comments

Comments
 (0)