Skip to content

benswinney/kube-singlenode-homelab

BranchesTags

Repository files navigation

Home Lab Setup

Steps to configure homelab kubernetes cluster - specific to my set up, but could be easily replicated within other environments

Operating System

Bare Metal install of Ubuntu Server 18.04 LTS on both Master and Worker nodes

Pre-Reqs

sudo apt install libnss3-tools
wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.0/mkcert-v1.4.0-linux-amd64
mv mkcert-v1.4.0-linux-amd64 mkcert
chmod +x mkcert
sudo mv mkcert /usr/local/bin/mkcert

Create x509 Certificates

# install local CA
mkcert -install
mkcert '*.home.swinney.io'
mkdir -p cert && mv *-key.pem cert/key.pem && mv *.pem cert/cert.pem

Install GoLANG

wget https://dl.google.com/go/go1.13.linux-amd64.tar.gz
tar -xvf go1.13.linux-amd64.tar.gz
sudo mv go /usr/local/bin
rm -f go1.13.linux-amd64.tar.gz
mkdir ~/go-projects

Set GOROOT and GOPATH in .profile

# set PATH for GoLang
if [ -d "$HOME/go-projects" ] ; then
        GOROOT="/usr/local/go"
        GOPATH="$HOME/go-projects"
        PATH="$GOPATH/bin:$GOROOT/bin:$PATH"
fi

Install GCC

sudo apt update
sudo apt install -y gcc

Install CFSSL

sudo apt install golang-cfssl

Install Istioctl



Kubernetes Configuration


Multi node deployment, with Pods deployable on Master node, suits my homelab environment.
Deep Learning Worker node is marked for ML/DL deployments only. No other Pods will run on it.


Master Node


Configure Kubernetes & Docker Repositories


sudo apt update && sudo apt install -y apt-transport-https curl ca-certificates software-properties-common nfs-common


curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -


sudo add-apt-repository \
  "deb https://apt.kubernetes.io/ kubernetes-xenial main"


curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -


sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"


sudo apt update


Disable Swap


sudo swapoff -a

sudo sed -i '/ swap / s/^/#/' /etc/fstab


The /etc/fstab file should now be commentted out for the swap mount point


Install Docker packages


sudo apt-get install docker-ce=18.06.2~ce~3-0~ubuntu


Hold Docker Version


sudo apt-mark hold docker-ce=18.06.2~ce~3-0~ubuntu


Modify Docker to use systemd driver and overlay2 storage driver


cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF


Restart docker


sudo systemctl daemon-reload && sudo systemctl restart docker


Install Kubernetes Packages


sudo apt install -y kubelet kubeadm kubectl


Hold Kuberneted Packages


sudo apt-mark hold kubelet kubeadm kubectl


Initialize Kubernetes node on the master node


sudo kubeadm init --config kubeinit.conf


Allow non-root user to run kubectl and configure Kubernetes


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


Apply a network model (Flannel or Secure Weave)


kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml


kubectl create secret -n kube-system generic weave-passwd --from-literal=weave-passwd=$(hexdump -n 16 -e '4/4 "%08x" 1 "\n"' /dev/random)
kubectl apply -n kube-system -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&password-secret=weave-passwd"


Configure as a single node cluster


kubectl taint nodes --all node-role.kubernetes.io/master-


Set up helm


kubectl create -f helm/helm-rbac.yaml
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
sudo snap install helm --classic
helm init --upgrade
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'


Install metalLB for Bare-Metal LB


helm install --name=metallb --namespace=metallb-system -f metallb/metallb-values.yaml stable/metallb


Install NFS Storage Provisioner & set as default storage class


kubectl create -f nfs-client/deploy/rbac.yaml
kubectl apply -f nfs-client/deploy/deployment.yaml
kubectl apply -f nfs-client/deploy/class.yaml
kubectl patch deployment nfs-client-provisioner -p '{"spec":{"template":{"spec":{"serviceAccount":"nfs-client-provisioner"}}}}'
kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'


Install Metric Server


kubectl create -f 1.8-metricserver


Install Kubernetes Dashboard


kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta1/aio/deploy/recommended.yaml


Edit the Kubernetes Dashboard service in order to expose it via MetalLB.


Change spec.type to LoadBalancer:


kubectl --namespace kubernetes-dashboard edit service kubernetes-dashboard
kubectl apply -f dashboard/dashboard-admin.yaml


Get Token for Kubernetes Dashboard


kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')


Example


eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXNtbTQyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3YjlkMzk4Ni1kYTQyLTQwMTUtOWI4ZC1mYjgzNzgxM2I1YTciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.jjGAVDeJJBIXe7jzSbmC_azlT5MAnH3yemX81m9Bv9W_I5u2Nm9aezTPZyRnO46UN7Eb2piWH5fUeNCiVZylPQt-FI4L4BGLEl5RWJInckollrSRw2bhEBkdtmEdHWjqsKXNQLV2qbuTin6ZE4lpuMa0PbkCkX-wtdpf0ejnq_PIIEdkOAvrYOKzIO6LHAEkCtK4nFObwEGPUH1yDoIbGCbdlg_xbEx-6Uv7Xz8YfbZ3DBDljcL_tyk8LwmaUWmNryTNclWBXNPOKnqrfkx1DEdj6RXTrG9TIbaIJ8YW324PmYPkPt_MDGQNxDDwpWAgH7BsogOcb7XWRGuix16_pQ


Get Kubernetes Dashboard ExternalIP


kubectl --namespace kubernetes-dashboard get service kubernetes-dashboard


Connect via https://


Install Heapster for cluster metrics and health info


Replaced by Metric Server Deployment, so no longer needed


helm install stable/heapster --name heapster --set rbac.create=true


Install SealedSecrets (bitnami-labs)


wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.8.3/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
rm -f kubeseal

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.8.3/controller.yaml

kubeseal --fetch-cert > certs/kubecert.pem


Example Usage:


echo -n <SECRET> | kubectl create secret generic <SECRET-NAME> --dry-run --from-file=<VALUE>=/dev/stdin -o yaml > <SECRET-FILENAME>.yaml
kubeseal --cert certs/kubecert.pem --format yaml < <SECRET-FILENAME>.yaml > <SEALEDSECRET-FILENAME>.yaml


Install Traefik for LoadBalancing/Ingress - We'll use Node-labels to prevent pods being deployed on Deep Learning Worker


kubectl apply -f traefik/traefik-service-acc.yaml
kubectl apply -f traefik/traefik-cr.yaml
kubectl apply -f traefik/traefik-crb.yaml
kubectl apply -f traefik/traefik-deployment.yaml
kubectl apply -f traefik/traefik-svc.yaml
kubectl apply -f traefik/traefik-webui-svc.yaml
kubectl apply -f traefik/traefik-webui-ingress.yaml


Create Additional NameSpaces


kubectl apply -f namespaces


Install Prometheus


kubectl apply -f monitoring


Add DeepLearning Worker Node


A lot of this could be done during the initial Master/Worker build


Install Nvidia GPU Drivers


TBC


REBOOT!


sudo apt update && sudo apt install -y apt-transport-https curl ca-certificates software-properties-common nfs-common


curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -


sudo add-apt-repository \
  "deb https://apt.kubernetes.io/ kubernetes-xenial main"  


curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo pt-key add -


sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"


Add nvidia-docker2 repository for passing GPU's into docker containers


curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list


sudo apt update


Disable Swap


sudo swapoff -a


Check /etc/fstab file and comment out the swap mounting point


Install Docker packages


sudo apt-get install docker-ce=18.06.2~ce~3-0~ubuntu 


Hold Docker Version


sudo apt-mark hold docker-ce=18.06.2~ce~3-0~ubuntu


Install nvidia-docker2


sudo apt-get install nvidia-docker2


Modify Docker to use systemd driver, overlay2 storage driver and default nvidia-docker2 runtime


cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
          "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "default-runtime": "nvidia",
  "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
  }
}
EOF


sudo mkdir -p /etc/systemd/system/docker.service.d


Restart docker


sudo systemctl daemon-reload && sudo systemctl restart docker


Install Kubernetes Packages


sudo apt install -y kubelet kubeadm kubectl


RBEOOT!


Add Deep Learning Worker node to Cluster


Create token for joining cluster and list it out


kubeadm token create && kubeadm token list 


List token-ca-cert-hash


openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'


On Worker Node


sudo kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>


Example:


sudo kubeadm join --token zo9ju3.ezaz85c7oha3x9jp kube:6443 --discovery-token-ca-cert-hash sha256:6877abd8a6f646680ad1fd8ef0373d128890715decfbb1724d75431dd8bbdd80


Confirm on master node that the worker node has joined


kubectl get nodes -o wide


Add NodeSelector Label to Deep Learning Worker Node


kubectl label nodes deeplab.home.swinney.io workload=mldl


Confirm label has been applied


kubectl get nodes --show-labels


Enable GPU Support within Kubernetes


On Master Node, install Nvidia DevicePlugin DaemonSet


kubectl create -f deviceplugins/nvidia-device-plugin.yml


Install homelab utilities


PiHole Ad Blocker for Kube


kubectl apply -f deployments/pihole


UniFi


Transmission


Media Server


Plex Media Server


KubeFlow for ML/DL


Install kfctl


opsys=linux

curl -s https://api.github.com/repos/kubeflow/kubeflow/releases/latest |\
    grep browser_download |\
    grep $opsys |\
    cut -d '"' -f 4 |\
    xargs curl -O -L && \
    tar -zvxf kfctl_*_${opsys}.tar.gz


Install KubeFlow


# Add kfctl to PATH, to make the kfctl binary easier to use.
export PATH=$PATH:"<path to kfctl>"
export KFAPP="<your choice of application directory name>"

# Installs Istio by default. Comment out Istio components in the config file to skip Istio installation. See https://github.com/kubeflow/kubeflow/pull/3663
export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_k8s_istio.0.6.2.yaml"

kfctl init ${KFAPP} --config=${CONFIG} -V
cd ${KFAPP}
kfctl generate all -V
kfctl apply all -V






  

    

      

  
About


      

        Home Lab Setup
      



    
Resources

    

      
        
        Readme
    


  

    
Code of conduct

    

      
        
        Code of conduct
      
    



    
Contributing

    

      
        
        Contributing
      
    



  
  

  

    

      
        Activity    



    
Stars

    

      
        2
        stars    


    
Watchers

    

      
        0
        watching    


    
Forks

    

      
        0
        forks    



    

      
          Report repository
    




    

  


  
      

        

          

  Releases


    
No releases published


        

      


  
  
  
      

        

          

  
  

  Packages
      



      

        

        
 

      

      

        

        
 

      

      

        

        
 

      




  

        

      


  
      

  
    

      

        
  
  

    Contributors  


  
    
      
  • 
        
    
      
    • 
      
  • 
        
    
      
    • 
      
  • 
        
    
      
    • 
  


  

      

    


  
      

        

          
Languages