chore: refactor and productionize

Author: DaMandal0rian

k8s/README.md

@@ -1,90 +1,193 @@
 # Service Quality Oracle - Kubernetes Deployment
 
-This directory contains Kubernetes manifests for deploying the Service Quality Oracle with persistent state management.
+This directory contains Kubernetes manifests for deploying the Service Quality Oracle in different environments using Kustomize with persistent state management.
+
+## Structure
+
+```
+k8s/
+├── README.md                    # This file
+├── auth.sh                     # Global auth script (configure for your cluster)
+├── base/                       # Common base resources
+│   ├── kustomization.yaml
+│   ├── namespace.yaml
+│   ├── deployment.yaml
+│   ├── service.yaml
+│   ├── servicemonitor.yaml
+│   ├── serviceaccount.yaml
+│   └── podmonitor.yaml
+└── environments/
+    ├── mainnet/                # Production environment
+    │   ├── kustomization.yaml
+    │   ├── config.yaml         # Mainnet configuration
+    │   ├── config.secret.yaml  # Mainnet secrets (configure before use)
+    │   ├── persistent-volume-claim.yaml
+    │   ├── auth.sh
+    │   ├── apply.sh
+    │   ├── diff.sh
+    │   └── restart-deployments.sh
+    └── testnet/                # Staging environment
+        ├── kustomization.yaml
+        ├── config.yaml         # Testnet configuration
+        ├── config.secret.yaml  # Testnet secrets (configure before use)
+        ├── persistent-volume-claim.yaml
+        ├── auth.sh
+        ├── apply.sh
+        ├── diff.sh
+        └── restart-deployments.sh
+```
 
 ## Prerequisites
 
 - Kubernetes cluster (version 1.19+)
 - `kubectl` configured to access your cluster
+- Kustomize (built into kubectl v1.14+)
 - Docker image published to `ghcr.io/graphprotocol/service-quality-oracle`
 - **Storage class configured** (see Storage Configuration below)
 
 ## Quick Start
 
-### 1. Create Secrets (Required)
+### 1. Configure Cluster Access
+
+Update `auth.sh` with your GKE cluster details:
+
+```bash
+# Edit auth.sh
+vim auth.sh
+
+# Connect to your cluster
+./auth.sh
+```
+
+### 2. Deploy to Testnet
+
+```bash
+cd environments/testnet
+
+# Configure secrets (replace placeholder values)
+vim config.secret.yaml
+
+# Preview changes
+./diff.sh
+
+# Deploy
+./apply.sh
+
+# Monitor
+kubectl logs -f deployment/service-quality-oracle -n service-quality-oracle
+```
+
+### 3. Deploy to Mainnet
 
 ```bash
-# Copy the example secrets file
-cp k8s/secrets.yaml.example k8s/secrets.yaml
+cd environments/mainnet
 
-# Edit with your actual credentials
-# IMPORTANT: Never commit secrets.yaml to version control
-nano k8s/secrets.yaml
+# Configure secrets (replace placeholder values with production keys)
+vim config.secret.yaml
+
+# Configure mainnet contract address
+vim config.yaml
+# Update BLOCKCHAIN_CONTRACT_ADDRESS with actual mainnet contract
+
+# Preview changes
+./diff.sh
+
+# Deploy (includes safety checks)
+./apply.sh
+
+# Monitor
+kubectl logs -f deployment/service-quality-oracle -n service-quality-oracle
 ```
 
-**Required secrets:**
+## Environment Configuration
+
+### Environment Differences
+
+| Setting | Testnet | Mainnet |
+|---------|---------|---------|
+| Chain | Arbitrum Sepolia | Arbitrum One |
+| Contract | 0x6d5...91f6 | Configure in config.yaml |
+| Image Tag | testnet-latest | mainnet-latest |
+| Labels | environment: testnet, variant: staging | environment: mainnet, variant: production |
+
+### Secret Configuration
+
+Before deploying, you must configure the following secrets in each environment's `config.secret.yaml`:
+
 - **`google-credentials`**: Service account JSON for BigQuery access
-- **`blockchain-private-key`**: Private key for Arbitrum Sepolia transactions  
+- **`blockchain-private-key`**: Private key for blockchain transactions (64 chars, no 0x)
+- **`etherscan-api-key`**: Etherscan API key
 - **`arbitrum-api-key`**: API key for Arbiscan contract verification
-- **`slack-webhook-url`**: Webhook URL for operational notifications
+- **`studio-api-key`**: The Graph Studio API key
+- **`studio-deploy-key`**: The Graph Studio deploy key
+- **`slack-webhook-url`**: Slack webhook for notifications
 
-### 2. Configure Storage (Required)
+## Storage Configuration
 
 ```bash
 # Check available storage classes
 kubectl get storageclass
 
-# If you see a default storage class (marked with *), skip to step 3
-# Otherwise, edit persistent-volume-claim.yaml and uncomment the appropriate storageClassName
+# The manifests use 'ssd-retain' storage class by default
+# Edit environments/{mainnet,testnet}/persistent-volume-claim.yaml if needed
 ```
 
 **Common storage classes by platform:**
 - **AWS EKS**: `gp2`, `gp3`, `ebs-csi`
-- **Google GKE**: `standard`, `ssd`  
+- **Google GKE**: `standard`, `ssd`
 - **Azure AKS**: `managed-premium`, `managed`
 - **Local/Development**: `hostpath`, `local-path`
 
-### 3. Deploy to Kubernetes
+## Operations
+
+### Restart Deployments
 
 ```bash
-# Apply all manifests
-kubectl apply -f k8s/
+./restart-deployments.sh
+```
+
+### View Logs
 
-# Verify deployment
-kubectl get pods -l app=service-quality-oracle
-kubectl get pvc -l app=service-quality-oracle
+```bash
+kubectl logs -f deployment/service-quality-oracle -n service-quality-oracle
 ```
 
-### 4. Monitor Deployment
+### Check Status
 
 ```bash
-# Check pod status
-kubectl describe pod -l app=service-quality-oracle
+kubectl get all -n service-quality-oracle
+```
 
-# View logs
-kubectl logs -l app=service-quality-oracle -f
+### Delete Environment
 
-# Check persistent volumes
-kubectl get pv
+```bash
+kubectl delete -k .
 ```
 
+## Monitoring
+
+- Prometheus scraping enabled via annotations
+- ServiceMonitor and PodMonitor configured for metrics collection
+- Metrics exposed on port 8000 at `/metrics` endpoint
+- Labels applied for environment-specific alerting
+
 ## Architecture
 
 ### Persistent Storage
 
 The service uses **two persistent volumes** to maintain state across pod restarts:
 
-- **`service-quality-oracle-data` (5GB)**: Circuit breaker state, last run tracking, BigQuery cache, CSV outputs
-- **`service-quality-oracle-logs` (2GB)**: Application logs
+- **`service-quality-oracle-data` (10GB)**: Circuit breaker state, last run tracking, BigQuery cache, CSV outputs
+- **`service-quality-oracle-logs` (5GB)**: Application logs
 
 **Mount points:**
 - `/app/data` → Critical state files (circuit breaker, cache, outputs)
 - `/app/logs` → Application logs
 
 ### Configuration Management
 
-**Non-sensitive configuration** → `ConfigMap` (`configmap.yaml`)
-**Sensitive credentials** → `Secret` (`secrets.yaml`)
+**Non-sensitive configuration** → `ConfigMap` (generated from `config.yaml`)
+**Sensitive credentials** → `Secret` (generated from `config.secret.yaml`)
 
 This separation provides:
 - ✅ Easy configuration updates without rebuilding images
@@ -98,7 +201,7 @@ This separation provides:
 - Memory: 512M
 
 **Limits (maximum):**
-- CPU: 1000m (1.0 core)  
+- CPU: 1000m (1.0 core)
 - Memory: 1G
 
 ## State Persistence Benefits
@@ -127,7 +230,7 @@ kubectl describe pod -l app=service-quality-oracle
 
 # Common issues:
 # - Missing secrets
-# - PVC provisioning failures
+# - PVC provisioning failures  
 # - Image pull errors
 ```
 
@@ -151,13 +254,16 @@ kubectl exec -it deployment/service-quality-oracle -- env | grep -E "(BIGQUERY|B
 kubectl exec -it deployment/service-quality-oracle -- ls -la /etc/secrets
 ```
 
-## Security Best Practices
+## Security
 
-✅ **Secrets never committed** to version control  
+✅ **Never commit actual secrets** - `config.secret.yaml` files contain placeholders only  
+✅ **Mainnet deployment safety checks** for production secrets  
+✅ **Non-root containers** with dropped capabilities  
 ✅ **Service account** with minimal BigQuery permissions  
 ✅ **Private key** stored in Kubernetes secrets (base64 encoded)  
 ✅ **Resource limits** prevent resource exhaustion  
-✅ **Read-only filesystem** where possible  
+✅ **Workload Identity** configured for secure GCP access  
+✅ **SSD storage with retention** for data persistence
 
 ## Production Considerations
 
@@ -171,7 +277,7 @@ kubectl exec -it deployment/service-quality-oracle -- ls -la /etc/secrets
 ## Next Steps
 
 1. **Test deployment** in staging environment
-2. **Verify state persistence** across pod restarts  
+2. **Verify state persistence** across pod restarts
 3. **Set up monitoring** and alerting
 4. **Configure backup** for persistent volumes
-5. **Enable quality checking** after successful validation
+5. **Enable quality checking** after successful validation

k8s/auth.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# Get cluster credentials for service-quality-oracle deployment
+# Update the following values based on your GKE cluster configuration:
+# - PROJECT: Your GCP project ID
+# - CLUSTER: Your GKE cluster name
+# - ZONE: Your GKE cluster zone/region
+
+PROJECT="graph-mainnet"
+CLUSTER="network"
+ZONE="us-central1-a"
+
+gcloud container clusters get-credentials $CLUSTER --project $PROJECT --zone $ZONE

k8s/deployment.yaml renamed to k8s/base/deployment.yaml

@@ -13,10 +13,20 @@ spec:
     metadata:
       labels:
         app: service-quality-oracle
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/path: "/metrics"
+        prometheus.io/port: "8000"
     spec:
+      imagePullSecrets:
+        - name: docker-registry
       containers:
       - name: service-quality-oracle
         image: ghcr.io/graphprotocol/service-quality-oracle:latest
+        imagePullPolicy: IfNotPresent
+        ports:
+          - containerPort: 8000
+            name: metrics
         envFrom:
         # Load all non-sensitive configuration from ConfigMap
         - configMapRef:
@@ -89,11 +99,21 @@ spec:
             - "import os; assert os.path.exists('/app/healthcheck'), 'Healthcheck file missing'"
           initialDelaySeconds: 10
           periodSeconds: 30
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 1000
+          readOnlyRootFilesystem: false
+          capabilities:
+            drop:
+            - ALL
       volumes:
       - name: data-volume
         persistentVolumeClaim:
           claimName: service-quality-oracle-data
       - name: logs-volume
         persistentVolumeClaim:
           claimName: service-quality-oracle-logs
-      restartPolicy: Always
+      serviceAccountName: service-quality-oracle
+      restartPolicy: Always

k8s/base/kustomization.yaml

@@ -0,0 +1,10 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - deployment.yaml
+  - service.yaml
+  - servicemonitor.yaml
+  - serviceaccount.yaml
+  - podmonitor.yaml

k8s/base/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: service-quality-oracle
+  labels:
+    name: service-quality-oracle

k8s/base/podmonitor.yaml

@@ -0,0 +1,15 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: service-quality-oracle
+  labels:
+    app: service-quality-oracle
+spec:
+  selector:
+    matchLabels:
+      app: service-quality-oracle
+  podMetricsEndpoints:
+  - port: metrics
+    path: /metrics
+    interval: 30s
+    scrapeTimeout: 10s

k8s/base/service.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: service-quality-oracle
+  labels:
+    app: service-quality-oracle
+spec:
+  selector:
+    app: service-quality-oracle
+  ports:
+    - name: metrics
+      port: 8000
+      targetPort: 8000
+      protocol: TCP
+  type: ClusterIP

k8s/base/serviceaccount.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: service-quality-oracle
+  labels:
+    app: service-quality-oracle
+  annotations:
+    iam.gke.io/gcp-service-account: [email protected]
+automountServiceAccountToken: false