DashVERSE

The dashboard prototype for the EVERSE project.

Warning

🚧 Work in Progress

This project is currently under active development and may not yet be fully stable. Features, functionality, and documentation are subject to change as progress continues.

We welcome feedback, contributions, and suggestions! If you encounter issues or have ideas for improvement, please feel free to open an issue or submit a pull request.

Requirements

• Python (3.12)
• Poetry (1.8.5)
• Podman (5.4.1)
• minikube (v1.34.0)
• helm (v3.16.4)

Links for the requirements

Python

https://www.python.org/downloads

Pyenv (optional)

Pyenv allows developers to install multiple versions of Python distribution and easy switching between the installed versions.

Website: https://github.com/pyenv/pyenv?tab=readme-ov-file#installation

Poetry (optional)

Poetry is used for dependency management of the Python packages.

https://python-poetry.org/docs/#installation

Podman

https://podman.io/docs/installation

Docker

https://docs.docker.com/engine/install

minikube

https://minikube.sigs.k8s.io/docs/start

helm

https://helm.sh/docs/intro/install

Deployment

If you would like to run the setup on a cloud (your own server)

1. Start a cluster using minikube and podman

   Set the default driver (podman or docker)

   minikube config set driver podman

   Start the cluster

   minikube start --cpus='4' --memory='4g' --driver=podman

   This will create a cluster using 4 cpus and 4GB of memory.

   The rest of the instructions need to be followed in kubernetes folder, so make sure you are in that folder before following the rest of the instructions:

   cd kubernetes

2. Generate Secrets for deployment

   bash generate-variables.sh

   The script can be customised via environment variables, e.g. EVERSE_DOMAIN_NAME=my.domain bash generate-variables.sh.

   It produces the following artefacts (all paths reside under kubernetes/):

   • deployments/XXXXXXXXXXXXXDD_MM_YYYYHH_MM/superset-deployment-secrets.yaml
   • deployments/XXXXXXXXXXXXXDD_MM_YYYYHH_MM/secrets.env
   • deployments/XXXXXXXXXXXXXDD_MM_YYYYHH_MM/generate_jwt.sh

   Warning: Do not share any of these generated files with anyone or push to a public repository!

   Now you can source the generated secrets.env file to set the environment variables which will be used for the deployment.

   source ./deployments/XXXXXXXXXXXXX**DD_MM_YYYY**HH_MM/secrets.env

   The generated environment variables are consumed by the Kubernetes manifests through envsubst. No plaintext database configuration files are created anymore.

   For a breakdown of the database schema and helper scripts, see docs/Database.md.

3. Build database initialization container

   Build the docker image

   docker build --no-cache -t ghcr.io/everse-researchsoftware/postgresql-setup-script:latest -t everse-db-scripts:latest ./DBModel

   Load the docker image to the cluster

   minikube image load everse-db-scripts:latest

   Check if the image is loaded

   minikube image ls

   Warning: Do not make this Docker image publicly available as it contains database password!

4. Create a namespace

   kubectl create namespace superset

5. Add the generated secrets to the cluster

   kubectl apply -f $DASHVERSE_SECRETS_FILE_NAME --namespace superset

6. Deploy db using deploy-db.yaml

   envsubst < deploy-db.yaml | kubectl apply --namespace superset -f -

   You can check the logs of initialization job using:

   DB_JOB_POD_NAME=$(kubectl get pods --namespace superset | grep "postgresql-init-job" | cut -d" " -f1)
   kubectl logs --namespace superset $DB_JOB_POD_NAME -c init-python-container

7. Deploy API using deploy-postgrest.yaml

   envsubst < deploy-postgrest.yaml | kubectl apply --namespace superset -f -

   You can check the logs using:

   POSTGREST_POD_NAME=$(kubectl get pods --namespace superset | grep "postgrest-" | cut -d" " -f1)
   kubectl logs --namespace superset $POSTGREST_POD_NAME --all-containers

   Test using:

   curl https://db.YOUR_DOMAIN/assessment

8. Deploy Apache Superset using dashverse-values.yaml

   Add superset repository to helm:

   helm repo add Superset https://apache.github.io/superset

   Set up Apache Superset:

   envsubst < dashverse-values.yaml > dashverse-values-with-secrets.yaml
   
   helm upgrade --install superset superset/superset --values dashverse-values-with-secrets.yaml --namespace superset --create-namespace --debug --cleanup-on-fail
   
   rm -f dashverse-values-with-secrets.yaml

   Below are the commands you can use for debugging the superset service.

   kubectl describe job --namespace superset superset-init-db

   kubectl logs --namespace superset  superset-init-db-7nccv # replace this with the actual name

   kubectl get pods --namespace superset  -l job-name=superset-init-db

   kubectl describe pod --namespace superset superset-init-db-hczfw

9. OPTIONAL - Deploy pgadmin

   envsubst < deploy-pgadmin.yaml | kubectl apply --namespace superset -f -

   To add the Postgresql database on pgadmin UI:

   Add a New server:
     General:
       Name --> postgres
     Connection:
       Host name --> superset-postgresql
       Port --> 5432
       Username --> $POSTGRES_USER in `XXXXXXXXXXXXX-superset-deployment-secrets` file
       Password --> see $POSTGRES_PASSWORD in `XXXXXXXXXXXXX-superset-deployment-secrets` file

10. Get the application URL by running these commands:

    kubectl get --namespace superset services

      export SUPERSET_NODE_PORT=$(kubectl get --namespace superset -o jsonpath="{.spec.ports[0].nodePort}" services superset)
      export POSTGREST_NODE_PORT=$(kubectl get --namespace superset -o jsonpath="{.spec.ports[0].nodePort}" services postgrest)
      export SWAGGER_NODE_PORT=$(kubectl get --namespace superset -o jsonpath="{.spec.ports[0].nodePort}" services swagger)
    
      export NODE_IP=$(kubectl get nodes --namespace superset -o jsonpath="{.items[0].status.addresses[0].address}")
      echo "superset: " http://$NODE_IP:$SUPERSET_NODE_PORT
      echo "postgrest: " http://$NODE_IP:$POSTGREST_NODE_PORT
      echo "swagger: " http://$NODE_IP:$SWAGGER_NODE_PORT

11. Set up the domain name for your server

12. Set DNS to point to your server ( you will need to setup a reverse proxy to be able to access the service)

Documentation

• docs/Kubernetes.md – operational commands for managing the Minikube deployment.
• docs/Database.md – details of the PostgreSQL schema, assessment mapping, and populate script usage.
• docs/API_examples.md – practical PostgREST calls, including the multi-step workflow for creating assessments.

Clean up

Delete services, deployments, volumes

List services

kubectl get --namespace superset services

Delete services

# database
kubectl delete service --namespace superset superset-postgresql

# postgrest API
kubectl delete service --namespace superset postgrest
kubectl delete service --namespace superset swagger

# pgadmin
kubectl delete service --namespace superset pgadmin

# superset
kubectl delete service --namespace superset superset
kubectl delete service --namespace superset superset-redis-headless
kubectl delete service --namespace superset superset-redis-master

List deployments

kubectl get deployments --namespace superset

Delete deployments

# database
kubectl delete deployment --namespace superset superset-postgresql
kubectl delete deployment --namespace superset postgrest

# pgadmin
kubectl delete deployment --namespace superset pgadmin

# superset
kubectl delete deployment --namespace superset superset
kubectl delete deployment --namespace superset superset-worker

List pvcs

kubectl get pvc --namespace superset

Delete pvcs

kubectl delete pvc --namespace superset superset-postgresql-data-pvc
kubectl delete pvc --namespace superset pgadmin-pvc

Get info

minikube --namespace superset service list
#minikube --namespace superset service --all
minikube --namespace superset service superset-postgresql --url
minikube --namespace superset service superset --url
minikube --namespace superset service postgrest --url
minikube --namespace superset service swagger --url
minikube --namespace superset service pgadmin --url

kubectl get --namespace superset services
kubectl get --namespace superset deployments
kubectl get --namespace superset deployments.apps

kubectl get pods --all-namespaces
kubectl get pods --namespace superset
kubectl describe pod --namespace superset superset-postgresql-774c87bbfc-vn5mk # show the details of a specific pod

kubectl describe pods --namespace superset

kubectl cluster-info

kubectl get pods -l app=superset-postgresql --namespace superset
kubectl get jobs -l job-name=superset-postgresql-init-job --namespace superset
kubectl logs -f <superset-postgresql-init-job-pod-name> --namespace superset

Connect to Postgresql server

kubectl get pods --namespace superset
kubectl exec -ti superset-postgresql-6c8dd65c-5k4d4 --namespace superset -- bash
psql -U postgres -d postgres -h localhost

Note: use your own username and password for the psql command

Dashboard

minikube dashboard --url

kubectl proxy --address='0.0.0.0' --disable-filter=true

http://SERVER_IP:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/workloads?namespace=superset

Delete the cluster and clean up

minikube stop
minikube delete --purge --all