make smol-k8s-lab more async for federated apps and add new experimental apps (#363)

* make smol-k8s-lab more async for federated apps

* more work for asyncing everything

* make more things async

* get rid of duplicate refresh_bweso function in grafana_stack.py, update naming schema for appset secret

* change to use grafana_stack_oidc_credentials_bitwarden_id everywhere and always refresh the grafana oidc credentials

* try to create missing user credentials (#364)

* change thanos_hostame to mimir_hostname

* try to create missing user credentials

* fix name and force vaules to update

* sync with jesse

* avoid reloading the argocd secrets plugin deployment entirely

* resolve s3_user_id' is not defined. when refreshing bitwarden

* fix typo

* remove extra s3 user for monitoring

* update allowed repos for grafana_stack

* add seaweedfs to allowed repos for the grafana stack monitoring project

* add optionals tolerations and afffinity to grafana-stack

* fix restores for seaweedfs for grafana-stack

* add experimental opa gatekeeper app

* add link to opa doc and update example header

* fix crash when adding nodes with >1 label

* always create the loki valkey credentials

* add temp to be optionally installed

* update mkdocs

* update tempo allowed repos

* more grafana source repos

* fix tempo source repos again

* make tempo generate secrets

* more tempo stuff

* always include s3 endpoint for tempo

* add kube-state-metrics repo to grafana_stack

* update netmaker database hostname to be netmaker-postgresql

* update netmaker database hostname to be netmaker-postgresql.netmaker.svc.cluster.local

* start adding restore capabilities to netmaker

* Update docs/k8s_apps/prometheus.md

* update poetry lock file

---------

Co-authored-by: Max! <admin@cloudydev.net>

Author: jessebot

docs/k8s_apps/experimental/tempo.md

@@ -0,0 +1,75 @@
+[Tempo](https://grafana.com/docs/tempo/latest/) is an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo lets you search for traces, generate metrics from spans, and link your tracing data with logs and metrics. We're still experimenting with it here at small-hack, so feel free to grab anything you like, but understand that it is still in development.
+
+## Experimental
+
+```yaml
+apps:
+  tempo:
+    description: |
+      [magenta]⚠️ Experimental[/magenta]
+      [link=https://grafana.com.com/oss/tempo]Tempo[/link]
+    enabled: true
+    init:
+      # if init is enabled, we'll set up an app in Zitadel for using Oauth2 with Grafana
+      enabled: true
+      # restore:
+      #   enabled: false
+      #   restic_snapshot_ids:
+      #     seaweedfs_volume: latest
+      #     seaweedfs_filer: latest
+    # backups:
+    #   # cronjob syntax schedule to run forgejo pvc backups
+    #   pvc_schedule: 10 0 * * *
+    #   s3:
+    #     # these are for pushing remote backups of your local s3 storage, for speed and cost optimization
+    #     endpoint: ''
+    #     bucket: ''
+    #     region: ''
+    #     secret_access_key:
+    #       value_from:
+    #         env: TEMPO_S3_BACKUP_SECRET_KEY
+    #     access_key_id:
+    #       value_from:
+    #         env: TEMPO_S3_BACKUP_ACCESS_ID
+    #   restic_repo_password:
+    #     value_from:
+    #       env: TEMPO_RESTIC_REPO_PASSWORD
+
+    argo:
+      # secrets keys to make available to Argo CD ApplicationSets
+      # toleration affinity
+      # toleration_key: dedicated
+      # toleration_operator: Equal
+      # toleration_value: example
+      # toleration_effect: NoSchedule
+      # affinity_key: dedicated
+      # affinity_value: example
+      secret_keys:
+        hostname: tempo.example.com
+        # local s3 endpoint for tempo S3, backed up constantly
+        # make this the same as your grafana_stack if you're using that
+        s3_endpoint: ""
+      # git repo to install the Argo CD app from
+      repo: https://github.com/small-hack/argocd-apps
+      # path in the argo repo to point to
+      path: tempo/app_of_apps/
+      # either the branch or tag to point at in the argo repo above
+      revision: main
+      # kubernetes cluster to install the k8s app into, defaults to Argo CD default
+      cluster: https://kubernetes.default.svc
+      # namespace to install the k8s app in
+      namespace: monitoring
+      # recurse directories in the provided git repo
+      directory_recursion: false
+      # source repos for Argo CD App Project (in addition to argo.repo)
+      project:
+        name: monitoring
+        source_repos:
+        - registry-1.docker.io
+        - ghcr.io/grafana/helm-charts
+        - https://seaweedfs.github.io/seaweedfs/helm
+        - https://github.github.io/helm-charts
+        destination:
+          # automatically includes the app's namespace and argocd's namespace
+          namespaces: []
+```

docs/k8s_apps/grafana_monitoring_stack.md

@@ -80,15 +80,15 @@ apps:
     init:
       # if init is enabled, we'll set up an app in Zitadel for using Oauth2 with Grafana
       enabled: true
+      restore:
+        enabled: false
+        restic_snapshot_ids:
+          seaweedfs_volume: latest
+          seaweedfs_filer: latest
 
     backups:
-      # cronjob syntax schedule to run forgejo pvc backups
+      # cronjob syntax schedule to run grafana stack pvc backups
       pvc_schedule: 10 0 * * *
-      # cronjob syntax (with SECONDS field) for forgejo postgres backups
-      # must happen at least 10 minutes before pvc backups, to avoid corruption
-      # due to missing files. This is because the backup shows as completed before
-      # it actually is
-      postgres_schedule: 0 0 0 * * *
       s3:
         # these are for pushing remote backups of your local s3 storage, for speed and cost optimization
         endpoint: ""
@@ -107,6 +107,14 @@ apps:
     argo:
       # secrets keys to make available to Argo CD ApplicationSets
       secret_keys:
+        ## you can delete these if you're not using tolerations/affinity
+        # toleration_key: ""
+        # toleration_operator: ""
+        # toleration_value: ""
+        # toleration_effect: ""
+        ## these are for node affinity, delete if not in use
+        # affinity_key: ""
+        # affinity_value: ""
         # FQDN to use for Thanos web interface
         thanos_hostname: ""
         # FQDN to use for Grafana
@@ -140,11 +148,12 @@ apps:
       project:
         name: monitoring
         source_repos:
-          - "registry-1.docker.io"
-          - "https://grafana.github.io/helm-charts"
-          - "https://github.com/grafana/helm-charts.git"
-          - "https://github.com/prometheus-community/helm-charts.git"
-          - "https://prometheus-community.github.io/helm-charts"
+          - registry-1.docker.io
+          - https://grafana.github.io/helm-charts
+          - ghcr.io/grafana/helm-charts
+          - https://github.com/prometheus-community/helm-charts.git
+          - https://prometheus-community.github.io/helm-charts
+          - https://seaweedfs.github.io/seaweedfs/helm
         destination:
           # automatically includes the app's namespace and argocd's namespace
           namespaces:

docs/k8s_apps/opa.md

@@ -0,0 +1,43 @@
+# Open Policy Agent Gatekeeper
+
+An experimental app for writing security policies for Kubernetes.
+
+Learn more here: https://open-policy-agent.github.io/gatekeeper/website/
+
+## Example Config
+
+```yaml
+apps:
+  opa:
+    enabled: true
+    description: |
+      [magenta]⚠️ Experimental[/magenta]
+      [link=https://open-policy-agent.github.io/gatekeeper/website/]OPA (Open Policy Agent) Gatekeeper[/link] is a customizable cloud native policy controller that helps enforce policies and strengthen governance. Put simply, it lets you set up policies for security requirements in Kubernetes.
+
+    init:
+      enabled: true
+    argo:
+      # secrets keys to make available to Argo CD ApplicationSets
+      secret_keys: []
+      # git repo to install the Argo CD app from
+      repo: https://github.com/small-hack/argocd-apps
+      # path in the argo repo to point to
+      path: opa/app_of_apps/
+      # either the branch or tag to point at in the argo repo above
+      revision: main
+      # kubernetes cluster to install the k8s app into, defaults to Argo CD default
+      cluster: https://kubernetes.default.svc
+      # namespace to install the k8s app in
+      namespace: gatekeeper-system
+      # recurse directories in the provided git repo
+      # if set to false, we will not deploy the CSI driver
+      directory_recursion: false
+      # source repos for Argo CD App Project (in addition to argo.repo)
+      project:
+        name: opa
+        source_repos:
+          - https://open-policy-agent.github.io/gatekeeper/charts
+        destination:
+          # automatically includes the app's namespace and argocd's namespace
+          namespaces: []
+```

docs/k8s_apps/prometheus.md

@@ -6,6 +6,8 @@ You can see an overview of the whole Prometheus Stack Argo CD Application at [sm
 <img src="../../assets/images/screenshots/prometheus_screenshot.png" alt="screenshot of the Argo CD web interface showing the prometheus app of apps which includes the following children: loki, prometheus-crd, prometheus-appset, prometheus-pushgateway-appset">
 </a>
 
+*NOTE*: This stack is deprecated in favor of the [Grafana Monitoring Stack](k8s_apps/grafana_monitoring_stack.md)!
+
 ## Important note
 
 We haven't generated new screenshots, but we've updated how we now deploy the Prometheus CRDs. They are now deployed separately, so that anything that relies on them that gets deployed earlier on, such as your identity provider, which you may want to secure the prometheus related frontends.

mkdocs.yaml

@@ -95,11 +95,13 @@ nav:
       - Longhorn: k8s_apps/experimental/longhorn.md
       - Openbao: k8s_apps/experimental/openbao.md
       - MinIO: k8s_apps/experimental/minio.md
+      - Tempo: k8s_apps/experimental/tempo.md
       - WriteFreely: k8s_apps/experimental/writefreely.md
       - Zalando Postgress Operator: k8s_apps/experimental/postgres_operator.md
     - Generic App: k8s_apps/generic_app.md
     - Generic Device Plugin: k8s_apps/generic_device_plugin.md
     - GoToSocial: k8s_apps/gotosocial.md
+    - Grafana Monitoring Stack: k8s_apps/grafana_monitoring_stack.md
     - Home Assistant: k8s_apps/home_assistant.md
     - Ingress Nginx: k8s_apps/ingress_nginx.md
     - K8tz: k8s_apps/k8tz.md