Test Run using Minikube on Macbook Pro #155

Hsankesara · 2021-11-23T12:17:18Z

Hsankesara
Nov 23, 2021
Maintainer

@K1Hyve, I am trying to run the Radar-Kubernetes on my Macbook using Minikube. Looks like I am getting some sort of timeout error.

I am currently using:

minikube version: v1.24.0
Kubernetes Version: 1.21.1
Helm Version: 3.7.0

The error log is:

install.go:437: [debug] Install failed and atomic is set, uninstalling release
  uninstall.go:95: [debug] uninstall: Deleting cp-zookeeper
  client.go:299: [debug] Starting delete for "cp-zookeeper" Service
  client.go:299: [debug] Starting delete for "cp-zookeeper-headless" Service
  client.go:299: [debug] Starting delete for "cp-zookeeper" StatefulSet
  client.go:299: [debug] Starting delete for "cp-zookeeper-jmx-configmap" ConfigMap
  client.go:299: [debug] Starting delete for "cp-zookeeper-pdb" PodDisruptionBudget
  W1123 15:14:39.317288   29404 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
  client.go:299: [debug] Starting delete for "cp-zookeeper" ServiceMonitor
  uninstall.go:144: [debug] purge requested for cp-zookeeper
  Error: release cp-zookeeper failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
  helm.go:88: [debug] timed out waiting for the condition
  release cp-zookeeper failed, and has been uninstalled due to atomic being set
  helm.sh/helm/v3/pkg/action.(*Install).failRelease
        helm.sh/helm/v3/pkg/action/install.go:445
  helm.sh/helm/v3/pkg/action.(*Install).reportToRun
        helm.sh/helm/v3/pkg/action/install.go:429
  helm.sh/helm/v3/pkg/action.(*Install).performInstall
        helm.sh/helm/v3/pkg/action/install.go:387
  runtime.goexit
        runtime/asm_amd64.s:1581
err 1: failed processing release cp-kafka: command "/usr/local/bin/helm" exited with non-zero status:

PATH:
  /usr/local/bin/helm

ARGS:
  0: helm (4 bytes)
  1: --kube-context (14 bytes)
  2: minikube (8 bytes)
  3: upgrade (7 bytes)
  4: --install (9 bytes)
  5: --reset-values (14 bytes)
  6: cp-kafka (8 bytes)
  7: ../cp-helm-charts/charts/cp-kafka (33 bytes)
  8: --wait (6 bytes)
  9: --timeout (9 bytes)
  10: 600s (4 bytes)
  11: --atomic (8 bytes)
  12: --create-namespace (18 bytes)
  13: --kube-context (14 bytes)
  14: minikube (8 bytes)
  15: --values (8 bytes)
  16: /var/folders/g0/gxjgvhyx3h3_qt0p7105cjk40000gn/T/helmfile888243866/cp-kafka-values-756c46c764 (93 bytes)
  17: --values (8 bytes)
  18: /var/folders/g0/gxjgvhyx3h3_qt0p7105cjk40000gn/T/helmfile1307905795/cp-kafka-values-5bbc6b7c78 (94 bytes)
  19: --set (5 bytes)
  20: brokers=3 (9 bytes)
  21: --history-max (13 bytes)
  22: 10 (2 bytes)
  23: --debug (7 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  history.go:56: [debug] getting history for release cp-kafka
  upgrade.go:139: [debug] preparing upgrade for cp-kafka
  Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
  helm.go:88: [debug] another operation (install/upgrade/rollback) is in progress
  helm.sh/helm/v3/pkg/action.init
        helm.sh/helm/v3/pkg/action/action.go:62
  runtime.doInit
        runtime/proc.go:6498
  runtime.doInit
        runtime/proc.go:6475
  runtime.main
        runtime/proc.go:238
  runtime.goexit
        runtime/asm_amd64.s:1581
  UPGRADE FAILED
  main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:200
  github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
  github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
  github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
  main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
  runtime.main
        runtime/proc.go:255
  runtime.goexit
        runtime/asm_amd64.s:1581

COMBINED OUTPUT:
  history.go:56: [debug] getting history for release cp-kafka
  upgrade.go:139: [debug] preparing upgrade for cp-kafka
  Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
  helm.go:88: [debug] another operation (install/upgrade/rollback) is in progress
  helm.sh/helm/v3/pkg/action.init
        helm.sh/helm/v3/pkg/action/action.go:62
  runtime.doInit
        runtime/proc.go:6498
  runtime.doInit
        runtime/proc.go:6475
  runtime.main
        runtime/proc.go:238
  runtime.goexit
        runtime/asm_amd64.s:1581
  UPGRADE FAILED
  main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:200
  github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
  github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
  github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
  main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
  runtime.main
        runtime/proc.go:255
  runtime.goexit
        runtime/asm_amd64.s:1581
err 2: failed processing release cp-schema-registry: command "/usr/local/bin/helm" exited with non-zero status:

PATH:
  /usr/local/bin/helm

ARGS:
  0: helm (4 bytes)
  1: --kube-context (14 bytes)
  2: minikube (8 bytes)
  3: upgrade (7 bytes)
  4: --install (9 bytes)
  5: --reset-values (14 bytes)
  6: cp-schema-registry (18 bytes)
  7: ../cp-helm-charts/charts/cp-schema-registry (43 bytes)
  8: --wait (6 bytes)
  9: --timeout (9 bytes)
  10: 600s (4 bytes)
  11: --atomic (8 bytes)
  12: --create-namespace (18 bytes)
  13: --kube-context (14 bytes)
  14: minikube (8 bytes)
  15: --values (8 bytes)
  16: /var/folders/g0/gxjgvhyx3h3_qt0p7105cjk40000gn/T/helmfile1808763976/cp-schema-registry-values-54b444dc96 (104 bytes)
  17: --values (8 bytes)
  18: /var/folders/g0/gxjgvhyx3h3_qt0p7105cjk40000gn/T/helmfile3875012166/cp-schema-registry-values-c4c564fd4 (103 bytes)
  19: --set (5 bytes)
  20: ingress.hosts[0].host=example.com (33 bytes)
  21: --set (5 bytes)
  22: ingress.hosts[0].paths={/schema/?(.*)} (38 bytes)
  23: --set (5 bytes)
  24: ingress.tls[0].secretName=radar-base-tls (40 bytes)
  25: --set (5 bytes)
  26: ingress.tls[0].hosts={example.com} (34 bytes)
  27: --history-max (13 bytes)
  28: 10 (2 bytes)
  29: --debug (7 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  history.go:56: [debug] getting history for release cp-schema-registry
  install.go:178: [debug] Original chart version: ""
  install.go:199: [debug] CHART PATH: /Users/heetsankesara/work/RADAR/temp/RADAR-Kubernetes/cp-helm-charts/charts/cp-schema-registry
  W1123 15:14:42.275395   29702 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
  client.go:128: [debug] creating 1 resource(s)
  client.go:128: [debug] creating 5 resource(s)
  W1123 15:14:42.600209   29702 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
  wait.go:48: [debug] beginning wait for 5 resources with timeout of 10m0s
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  install.go:437: [debug] Install failed and atomic is set, uninstalling release
  uninstall.go:95: [debug] uninstall: Deleting cp-schema-registry
  client.go:299: [debug] Starting delete for "cp-schema-registry" Ingress
  W1123 15:24:44.901036   29702 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
  client.go:299: [debug] Starting delete for "cp-schema-registry" Service
  client.go:299: [debug] Starting delete for "cp-schema-registry" Deployment
  client.go:299: [debug] Starting delete for "cp-schema-registry-jmx-configmap" ConfigMap
  client.go:299: [debug] Starting delete for "cp-schema-registry" ServiceMonitor
  uninstall.go:144: [debug] purge requested for cp-schema-registry
  Error: release cp-schema-registry failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
  helm.go:88: [debug] timed out waiting for the condition
  release cp-schema-registry failed, and has been uninstalled due to atomic being set
  helm.sh/helm/v3/pkg/action.(*Install).failRelease
        helm.sh/helm/v3/pkg/action/install.go:445
  helm.sh/helm/v3/pkg/action.(*Install).reportToRun
        helm.sh/helm/v3/pkg/action/install.go:429
  helm.sh/helm/v3/pkg/action.(*Install).performInstall
        helm.sh/helm/v3/pkg/action/install.go:387
  runtime.goexit
        runtime/asm_amd64.s:1581

COMBINED OUTPUT:
  history.go:56: [debug] getting history for release cp-schema-registry
  Release "cp-schema-registry" does not exist. Installing it now.
  install.go:178: [debug] Original chart version: ""
  install.go:199: [debug] CHART PATH: /Users/heetsankesara/work/RADAR/temp/RADAR-Kubernetes/cp-helm-charts/charts/cp-schema-registry
  W1123 15:14:42.275395   29702 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
  client.go:128: [debug] creating 1 resource(s)
  client.go:128: [debug] creating 5 resource(s)
  W1123 15:14:42.600209   29702 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
  wait.go:48: [debug] beginning wait for 5 resources with timeout of 10m0s
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  ready.go:277: [debug] Deployment is not ready: default/cp-schema-registry. 0 out of 1 expected pods are ready
  install.go:437: [debug] Install failed and atomic is set, uninstalling release
  uninstall.go:95: [debug] uninstall: Deleting cp-schema-registry
  client.go:299: [debug] Starting delete for "cp-schema-registry" Ingress
  W1123 15:24:44.901036   29702 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
  client.go:299: [debug] Starting delete for "cp-schema-registry" Service
  client.go:299: [debug] Starting delete for "cp-schema-registry" Deployment
  client.go:299: [debug] Starting delete for "cp-schema-registry-jmx-configmap" ConfigMap
  client.go:299: [debug] Starting delete for "cp-schema-registry" ServiceMonitor
  uninstall.go:144: [debug] purge requested for cp-schema-registry
  Error: release cp-schema-registry failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
  helm.go:88: [debug] timed out waiting for the condition
  release cp-schema-registry failed, and has been uninstalled due to atomic being set
  helm.sh/helm/v3/pkg/action.(*Install).failRelease
        helm.sh/helm/v3/pkg/action/install.go:445
  helm.sh/helm/v3/pkg/action.(*Install).reportToRun
        helm.sh/helm/v3/pkg/action/install.go:429
  helm.sh/helm/v3/pkg/action.(*Install).performInstall
        helm.sh/helm/v3/pkg/action/install.go:387
  runtime.goexit
        runtime/asm_amd64.s:1581

Do you have any comments on what's causing this and how can I fix this?

afolarin · 2021-11-23T19:48:39Z

afolarin
Nov 23, 2021
Maintainer

sounds like an issue releasing zookeeper - could you get the logs from that before it is cleaned up? Subsequent confluent components are dependent on ZK.

2 replies

mpgxvii Nov 23, 2021
Maintainer

Yeah I had a similar issue before and from the logs it looked like it wasn't able to resolve the kafka brokers which is why it was timing out.

Hsankesara Nov 24, 2021
Maintainer Author

I rerun the Radar-kubernetes and got the same error but this time on nginx-ingress and cert-manager

FAILED RELEASES:
NAME
cert-manager
kube-prometheus-stack
nginx-ingress
in helmfile.d/00-init.yaml: 3 errors:
err 0: failed processing release cert-manager: command "/usr/local/bin/helm" exited with non-zero status:

PATH:
  /usr/local/bin/helm

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Error: timed out waiting for the condition

COMBINED OUTPUT:
  Release "cert-manager" does not exist. Installing it now.
  Error: timed out waiting for the condition
err 1: failed processing release kube-prometheus-stack: command "/usr/local/bin/helm" exited with non-zero status:

STDERR:
  Error: failed pre-install: timed out waiting for the condition

COMBINED OUTPUT:
  Release "nginx-ingress" does not exist. Installing it now.
  Error: failed pre-install: timed out waiting for the condition

Ingress logs return
Error from server (BadRequest): container "create" in pod "nginx-ingress-ingress-nginx-admission-create-glnc6" is waiting to start: ContainerCreating

kubectl get -A pods

NAMESPACE              NAME                                                       READY   STATUS              RESTARTS   AGE
cert-manager           cert-manager-756bb56c5-f7lcz                               0/1     ContainerCreating   0          26m
cert-manager           cert-manager-cainjector-86bc6dc648-zs2qc                   0/1     ContainerCreating   0          26m
cert-manager           cert-manager-webhook-78b6f5dfcc-x5jsw                      0/1     ContainerCreating   0          26m
default                nginx-ingress-ingress-nginx-admission-create-glnc6         0/1     ContainerCreating   0          16m
graylog                elasticsearch-master-0                                     0/1     Init:0/1            0          26m
graylog                elasticsearch-master-1                                     0/1     Init:0/1            0          26m
graylog                elasticsearch-master-2                                     0/1     Init:0/1            0          26m
graylog                fluent-bit-9gnbx                                           0/1     ContainerCreating   0          26m
graylog                graylog-0                                                  0/1     Init:0/1            0          26m
graylog                mongodb-0                                                  0/2     ContainerCreating   0          26m
graylog                mongodb-arbiter-0                                          1/1     Running             4          26m
kube-system            coredns-558bd4d5db-89szq                                   1/1     Running             0          37m
kube-system            etcd-minikube                                              1/1     Running             0          37m
kube-system            kube-apiserver-minikube                                    1/1     Running             0          37m
kube-system            kube-controller-manager-minikube                           1/1     Running             0          37m
kube-system            kube-proxy-2xxn6                                           1/1     Running             0          37m
kube-system            kube-scheduler-minikube                                    1/1     Running             0          37m
kube-system            storage-provisioner                                        1/1     Running             0          37m
kubernetes-dashboard   dashboard-metrics-scraper-5594458c94-zt26b                 1/1     Running             0          36m
kubernetes-dashboard   kubernetes-dashboard-654cf69797-t9pk9                      1/1     Running             0          36m
monitoring             kube-prometheus-stack-grafana-5584d9bbf4-lc8nx             0/2     Init:0/1            0          21m
monitoring             kube-prometheus-stack-kube-state-metrics-54df5cbb6-8j48v   0/1     ContainerCreating   0          21m
monitoring             kube-prometheus-stack-operator-69c95bd5f-p8z6l             0/1     ContainerCreating   0          21m
monitoring             kube-prometheus-stack-prometheus-node-exporter-59jnt       0/1     ContainerCreating   0          21m

Hsankesara · 2021-11-24T12:15:23Z

Hsankesara
Nov 24, 2021
Maintainer Author

Yeah I had a similar issue before and from the logs it looked like it wasn't able to resolve the kafka brokers which is why it was timing out.

Yes, how did you resolve it? I am currently trying to diagnose the real issue but cannot pin it down.

5 replies

Hsankesara Nov 24, 2021
Maintainer Author

In debug mode, I am getting this error

UPGRADE FAILED
  main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:200
  github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
  github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
  github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
  main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
  runtime.main
        runtime/proc.go:255
  runtime.goexit
        runtime/asm_amd64.s:1581

COMBINED OUTPUT:
  history.go:56: [debug] getting history for release kube-prometheus-stack
  upgrade.go:139: [debug] preparing upgrade for kube-prometheus-stack
  upgrade.go:488: [debug] resetting values to the chart's original version
  upgrade.go:147: [debug] performing update for kube-prometheus-stack
  upgrade.go:319: [debug] creating upgraded release for kube-prometheus-stack
  client.go:218: [debug] checking 127 resources for changes

afolarin Nov 24, 2021
Maintainer

I suspect the issue is one of resources see @mpgxvii #156 seems on memory limiting environments random things fail. What is the machine you are running this on?

Hsankesara Nov 25, 2021
Maintainer Author

Possibly, I am currently using hyperkit hypervisor with 8GM RAM, 60 GB disk space and 6 CPUs. What is the recommended amount of RAM and CPUs @K1Hyve ?

afolarin Nov 25, 2021
Maintainer

I think the CPUs probably fine (but I think something like 16GB RAM needed (just on an idling stack), but @K1Hyve suggesting 32GB).

Hsankesara Nov 25, 2021
Maintainer Author

Oh, My MacBook only has a total RAM of 16 GB. I can create a VM at Rosalind and see if I can run it there.

Test Run using Minikube on Macbook Pro #155

Uh oh!

Hsankesara Nov 23, 2021 Maintainer

Replies: 2 comments · 7 replies

Uh oh!

Uh oh!

afolarin Nov 23, 2021 Maintainer

Uh oh!

mpgxvii Nov 23, 2021 Maintainer

Uh oh!

Uh oh!

Hsankesara Nov 24, 2021 Maintainer Author

Uh oh!

Hsankesara Nov 24, 2021 Maintainer Author

Uh oh!

Hsankesara Nov 24, 2021 Maintainer Author

Uh oh!

afolarin Nov 24, 2021 Maintainer

Uh oh!

Hsankesara Nov 25, 2021 Maintainer Author

Uh oh!

afolarin Nov 25, 2021 Maintainer

Uh oh!

Hsankesara Nov 25, 2021 Maintainer Author

Hsankesara
Nov 23, 2021
Maintainer

Replies: 2 comments 7 replies

afolarin
Nov 23, 2021
Maintainer

mpgxvii Nov 23, 2021
Maintainer

Hsankesara Nov 24, 2021
Maintainer Author

Hsankesara
Nov 24, 2021
Maintainer Author

Hsankesara Nov 24, 2021
Maintainer Author

afolarin Nov 24, 2021
Maintainer

Hsankesara Nov 25, 2021
Maintainer Author

afolarin Nov 25, 2021
Maintainer

Hsankesara Nov 25, 2021
Maintainer Author