Document local development setup

[afritzler]

Summary

We should extend the current local setup guide with a working end to end flow.

I would suggest to use two base requirements regarding the Kubernetes cluster:

1. Using Minikube (as we need local storage to get rook + Ceph running)
2. Real dev cluster and the requirements needed to get e.g. the dynamic based provisioning of Ceph running

Additionally we should add/or reference a conclusive set of sample manifests from the onmetal-api project for VolumePool, VolumeClass and Volume which can be used to validated that the cephlet manager is behaving as expected.

/ref
https://rook.io/docs/rook/v1.9/Getting-Started/quickstart/

/cc @EvgeniaOrfeeva

[EvgeniaOrfeeva]

1. I would start with something like this as an example:

Prepare environment + List of Requirements

Following tools are required to work on that project.

If you have an access to the docker registry and k8s installation that you can use for development purposes, you may skip corresponding steps.

Otherwise, create a local instance of docker registry and k8s.

# start minikube
minikube start
# enable registry
minikube addons enable registry
# run proxy to registry
docker run --rm -d --name registry-bridge --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000" 

@afritzler Here goes the question. Can we use something different then Minikube like Kind/ Docker Desktop or whatever else?

2. It has been mentioned 'kustomize'

Should we include information like:

Deployment

Operator can be deployed to the environment with kustomize

# build and apply
kustomize build config/default | kubectl apply -f -
# build and remove
kustomize build config/default | kubectl delete -f -

3. Real dev cluster and the requirements needed to get4.

@afritzler Do you mean that? --->

go >= 1.19
git, make and kubectl
[Kustomize](https://kustomize.io/)
[Minikube](https://minikube.sigs.k8s.io/docs/) or a real cluster

and

Rook cluster
Raw devices (no partitions or formatted filesystems)
This requires lvm2 to be installed on the host. To avoid this dependency, you can create a single full-disk partition on the disk (see below)
Raw partitions (no formatted filesystem)
Persistent Volumes available from a storage class in block mode

with the following steps on how to run a simple rook cluster ect. ?