Failed to connect to ordermgr endpoint

[ericzji]

Hi,
I encountered the following error, after the services have been successfully deployed and I started try out the services:
zji@~/opentracing-prometheus-example/simple> curl http://192.168.99.100:30772/buy
curl: (7) Failed to connect to 192.168.99.100 port 30772: Connection refused

I deployed the 2 services on minikube:

zji@~/opentracing-prometheus-example/simple> kubectl get pod
NAME                                 READY     STATUS    RESTARTS   AGE
accountmgr-59fd8dc6c8-frsq4          1/1       Running   2          18m
accountmgr-59fd8dc6c8-h7rdw          1/1       Running   1          18m
accountmgr-59fd8dc6c8-zkqnp          1/1       Running   1          18m
jaeger-deployment-84c69bc88d-l9s8h   1/1       Running   0          20m
ordermgr-7998c8794c-78csd            1/1       Running   1          18m
ordermgr-7998c8794c-f2gql            1/1       Running   1          18m
ordermgr-7998c8794c-zvdbw            1/1       Running   1          18m
prometheus-865564b8f5-5c6b4          1/1       Running   0          21m
zji@~/opentracing-prometheus-example/simple> kubectl get svc
NAME               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
accountmgr         LoadBalancer   10.111.221.254   <pending>     8080:30515/TCP                 18m
jaeger-agent       ClusterIP      None             <none>        5775/UDP,6831/UDP,6832/UDP     21m
jaeger-collector   ClusterIP      10.104.49.71     <none>        14267/TCP,14268/TCP,9411/TCP   21m
jaeger-query       LoadBalancer   10.108.114.168   <pending>     80:32428/TCP                   21m
kubernetes         ClusterIP      10.96.0.1        <none>        443/TCP                        25m
ordermgr           LoadBalancer   10.96.39.56      <pending>     8080:30772/TCP                 18m
prometheus         LoadBalancer   10.105.126.213   <pending>     9090:30900/TCP                 22m
zipkin             ClusterIP      None             <none>        9411/TCP                       21m
zji@~/opentracing-prometheus-example/simple> minikube service ordermgr --url
http://192.168.99.100:30772

Also noticed Jaeger is going up and down:

zji@~/opentracing-prometheus-example/simple> kubectl get pods
NAME                                 READY     STATUS    RESTARTS   AGE
accountmgr-59fd8dc6c8-frsq4          1/1       Running   3          40m
accountmgr-59fd8dc6c8-h7rdw          1/1       Running   3          40m
accountmgr-59fd8dc6c8-zkqnp          1/1       Running   3          40m
jaeger-deployment-84c69bc88d-l9s8h   0/1       Running   0          42m
ordermgr-7998c8794c-78csd            1/1       Running   1          40m
ordermgr-7998c8794c-f2gql            1/1       Running   2          40m
ordermgr-7998c8794c-zvdbw            1/1       Running   4          40m
prometheus-865564b8f5-5c6b4          1/1       Running   0          43m
zji@~/opentracing-prometheus-example/simple> kubectl describe pod jaeger-deployment-84c69bc88d-l9s8h
Name:           jaeger-deployment-84c69bc88d-l9s8h
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Tue, 06 Feb 2018 10:33:28 -0800
Labels:         app=jaeger
                jaeger-infra=jaeger-pod
                pod-template-hash=4072567448
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"jaeger-deployment-84c69bc88d","uid":"3953422a-0b6c-11e8-8238-080...
Status:         Running
IP:             172.17.0.5
Controlled By:  ReplicaSet/jaeger-deployment-84c69bc88d
Containers:
  jaeger:
    Container ID:   docker://195994efcf29127298a85b60f4e5375a912a1fdb92b8991f74f0f7018c7e8e18
    Image:          jaegertracing/all-in-one
    Image ID:       docker-pullable://jaegertracing/all-in-one@sha256:38a827ec75750e31af32d129686676e065b496a90264c953b423c60ebec26a4e
    Ports:          5775/UDP, 6831/UDP, 6832/UDP, 16686/TCP, 9411/TCP
    State:          Running
      Started:      Tue, 06 Feb 2018 10:33:32 -0800
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:16686/ delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      COLLECTOR_ZIPKIN_HTTP_PORT:  9411
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rv2pb (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-rv2pb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rv2pb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                 Age                From               Message
  ----     ------                 ----               ----               -------
  Normal   Scheduled              38m                default-scheduler  Successfully assigned jaeger-deployment-84c69bc88d-l9s8h to minikube
  Normal   SuccessfulMountVolume  38m                kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"
  Normal   Pulling                38m                kubelet, minikube  pulling image "jaegertracing/all-in-one"
  Normal   Pulled                 38m                kubelet, minikube  Successfully pulled image "jaegertracing/all-in-one"
  Normal   Created                38m                kubelet, minikube  Created container
  Normal   Started                38m                kubelet, minikube  Started container
  Warning  Unhealthy              32m (x9 over 35m)  kubelet, minikube  Readiness probe failed: Get http://172.17.0.5:16686/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Normal   SuccessfulMountVolume  31m                kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"
  Normal   SuccessfulMountVolume  27m                kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"
  Normal   SuccessfulMountVolume  21m                kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"
  Normal   SuccessfulMountVolume  13m                kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"
  Normal   SuccessfulMountVolume  9m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"
  Normal   SuccessfulMountVolume  4m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rv2pb"

Any help is appreciated.
thanks
-Eric

[objectiser]

Hi Eric
The example services don't have readiness probes, so you may need to wait before the endpoint is actually available. If you could try the service after say 10 or 20 seconds and let me know if you still fail to connect.
Haven't seen the restarts - what versions of minikube and kubenetes client/server are you using?

[ericzji]

Thanks @objectiser

The problem is not about waiting, and the Jaeger pod just went up and down continuously.

attached below please find the version information:

> minikube  version
minikube version: v0.24.1
> kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T19:59:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

[objectiser]

For the jaeger problem, could you create an issue on https://github.com/jaegertracing/jaeger-kubernetes providing any details you have.