Aces Metrics Handler

Will operate in the EMDC level to extract metrics and efficiently organize them. Licensed under MIT license

Key Components

• Storage Components: Object Storage (MinIO), Timeseries DB (TimescaleDB), Graph Store (Neo4j)
• Metrics Catalogue: REST API that exposes stored information and data in all storages
• Pull-Push Metrics Pipelines: Confluent Kafka, Prometheus, Metrics Scrapper
• Metrics Consumer: Kafka Consumer which receives extracted metrics

Installation Steps & Prerequisites

1. Install Workflow Components

cd config/external/k8s/workflow

1.1 Install Minio

cd minio/
kubectl apply -f pv.yaml
kustomize build infra | kubectl apply -f -
cd mc/
kubectl apply -f .

1.2 Install & Configure Prefect Server

cd prefect/
bash make_server.sh
cd set_prefect_scripts/
kubectl apply -f deployment.yaml
cd ../
bash make_agent.sh

1.3 Install Jupyter Notebook

cd jupyter/
kubectl apply -f .

1.4 Port forward Workflow Services

1.4.1 MinIO Console

 kubectl port-forward svc/console 9090:9090 -n minio-operator
 kubectl describe secrets/console-sa-secret -n minio-operator

1.4.2 Prefect Server UI

kubectl port-forward svc/prefect-server 4200:4200

1.4.3 Jupyter Notebook

kubectl port-forward svc/notebook 8888:8888

Deploy ETLs to Prefect Workflow Orchestrator

prefect deployment build flows/manage_metrics_flow.py:manage_metrics_flow -n 'manage_metrics_flow' -ib kubernetes-job/prod -sb 'remote-file-system/minio' --pool aces
prefect deployment apply manage_metrics_flow-deployment.yaml 

2. Storage Components

cd config/k8s/external/storage-components

2.1 Install Neo4j

cd neo4j/standalone
bash setup.sh

2.2 Install TimescaleDB

cd timescaledb
kubectl apply -f pvc.yaml
kubectl apply -f deployment.yaml

2.3 Port Forward Storage Components

2.3.1 Neo4j

kubectl port-forward svc/neo4j 7474:7474

2.3.2 Timescaledb

kubectl port-forward svc/timescaledb 5432:5432
cd storage/timescaledb
python init_table.py

note: you need to create a virtual environment with psycopg2 and activate the virtual environment before running the script

3. Metrics Catalogue

0. How to build Metrics catalogue dockerfile see documentation here
1. cd config/k8s/aces/metrics_catalogue
2. kubectl apply -f .
3. kubectl port-forward svc/metrics-catalogue 8000:8000
4. Init the Metrics Management System using the following CURL API

curl -X 'GET' \
  'http://localhost:8000/init' \
  -H 'accept: application/json'

4. Pull-push Metrics Pipeline

cd config/k8s/external/pull-push-pipeline

4.1 Deploy Prometheus

1. cd prometheus
2. For local deployment run bash setup.sh. For HIRO AWS testbed run kubectl apply -f prom-chart.yaml.

4.2 Install NATS

cd nats/
kubectl apply -f.

4.3 Install prometheus-NATS adapter

cd nats-adapter/
kubectl apply -f.

4.4 Port Forward NATS

Port Forward NATS

kubectl port-forward svc/nats-server 4222:4222

5. Metrics Consumer

0. How to build Metrics consumer dockerfile see documentation here
1. cd config/k8s/aces/metrics_consumer
2. kubectl apply -f .