Is it Observable

Episode : How to secure your secrets?

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode presenting the solution managing your secrets

this tutorial will also utilize the OpenTelemetry Operator with:

• OpenTelemetry Demo Application
• The openTelemetry Collector to collect logs, traces and metrics
• The Dynatrace Operator to report the health on our k8s objects
• external secret operator
• the sops operator
• the vault injector
• and last a cluster having the rest encryption enabled All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm
• Vault installed on a seperate machine
• Vault cli installed
• Age installed
• sops

3. Clone Github repo

git clone  https://github.com/isItObservable/K8s-secrets
cd K8s-secrets

Getting started

Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

Vault

0. Login to your Vault

export VAULT_ADDR='http://127.0.0.1:8200'

# Login with root token (save this if you don't have it)
vault login

# Check status
vault status

Command line tooling

sops

Go to SOPS and install the sops CLI compatible with your operating system

age

Go to age and install the age cli compatible with your operating system

Kubernetes clusters

For each solution we will utilize different clusters :

• one for the External secret operator
• one for the SOPS operator
• one using the vault injector
• last one from the k8S encryption.

For each solution except the k8S encryption your could you use the Cloud provider of your choice, in my case i will use the Cluster API on my Proxmox cluster.

Here is the command to spin up a cluster in GCP:

GCP

1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

2.Create a GKE cluster

ZONE=europe-west3-a
NAME=isitobservable-secrets
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2 --monitoring=NONE --logging=NONE

Cluster API

Otherwise you will find in each solution a cluster api manifest file for proxmox. Each of those manifest would need to be customized by modifying :

• the local ip adresses
• the proxmox nodes
• the reference to your local docker registry

External Secret Operator

Deploy the external Secret operator and the application

The application will deploy the entire environment: Let's connect to harshicorp using the root token:

vault login
vault kv put secret/k8s/dynatrace/dynakube \
  apiToken=$API_TOKEN \
  dataIngestToken=$DATA_INGEST_TOKEN


vault kv put secret/k8s/default/dataingest \
  dynatrace_oltp_url=$DT_TENANT_URL \
  dt_api_token=$DATA_INGEST_TOKEN 

chmod 777 secret-operator/deploy.sh
NAME=observable-eso
./secret-operator/deploy.sh  --clustername "${NAME}"  -dturl "${DT_TENANT_URL}" --vaulturl "${VAULT_ADDR}" "

SOPS Operator

First let's generate the age key if you don't have it in your current workstation:

age-keygen -o ~/.config/sops/age/keys.txt

Deploy the SOPS operator and the application

Create a new cluster and deploy the environemnt with:

chmod 777 sops/deploy.sh
NAME=observable-sops
./sops/deploy.sh --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" 

Vault Injector

we will use the same vault secret created for the external secret operator

Deploy the vault injector and the application

Create a new cluster and deploy the environemnt with:

chmod 777 vault/deploy.sh
NAME=observable-vault
./vault/deploy.sh --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --vaulturl "${VAULT_ADDR}" --dtoperatortoken "${API_TOKEN}" 

k8S with rest encryption enabled ( needs a local cluster with the cluster api)

Adjust the cluster api definition file for you cluster located in : rest-encryption/capi_observable_encryption.yaml

once your cluster is running, you can deploy the solution using the following script

chmod 777 rest-encryption/deploy.sh
NAME=observable-enc
./rest-encryption/deploy.sh --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}"