Skip to content

README.md

Latest commit

 

History

History
70 lines (58 loc) · 1.9 KB

README.md

File metadata and controls

70 lines (58 loc) · 1.9 KB

Local Infrastructure Deployment

This guide explains how to stand up the local environment for the BigData Republic Open Data Platform using Colima (container runtime) and k3d (Kubernetes-in-Docker). It provisions a lightweight object storage service (MinIO) and prepares configuration for the data platform.

Overview

You will:

  1. Install required tooling
  2. Start a local container runtime
  3. Create a k3d Kubernetes cluster with required port mappings
  4. Deploy local object storage via Terraform (MinIO Helm release)
  5. Prepare Trino configuration values and ENV variables

Prerequisites

macOS with Homebrew. Ensure you have sufficient local resources (recommended: 4 CPUs, 8 GB memory for the VM plus headroom).

1. Install Dependencies

brew install terraform
brew install colima
brew install k3d
brew install helm
helm repo add minio https://charts.min.io/
helm repo update

2. Start Container Runtime

# Start Colima with explicit resource limits (adjust as needed)
colima start --cpu 4 --memory 8

3. Create Local Kubernetes Cluster

Expose the Superset UI NodePort (30088) on host port 8088 for convenience.

K3D_FIX_DNS=0 k3d cluster create data \
  --agents-memory 8G \
  --port "8088:30088@server:0"

Deploy Object Storage

4. Deploy Object Storage (MinIO)

From repo root:

cd infra/local
terraform init
terraform apply

5. Prepare Trino Configuration and ENV Variables

Generate a local values file for Trino.

# From repo root
cd configs/trino

S3_ENDPOINT=http://storage:9000 \
S3_REGION=us-east-1 \
S3_AWS_ACCESS_KEY=admin \
S3_AWS_SECRET_KEY=password \
envsubst < values-template.yaml > values-local.yaml

Prepare ENV Variables used for the platform deployment:

export KUBE_CONFIG_PATH=$(realpath ~/.kube/config)
export KUBECONFIG=$KUBE_CONFIG_PATH

Next Steps

Continue with the platform deployment