Failed to see trace in Jaeger UI

[hongbo-miao]

I am using Linkerd. I am trying to show trace in Jaeger UI.

I have read these related issues, but none of them has accepted answers:
#5472
#5095
#4777

Following the Linkerd tracing tutorial which uses emojivoto app as demo, I am trying to repeat it in my own app.

1. emojivoto App

For the demo app emojivoto, after running

linkerd inject https://run.linkerd.io/emojivoto.yml | kubectl apply -f -

I can see linkerd-proxy got added to Jaeger UI service list first.

Then after set env by

kubectl -n emojivoto set env --all deploy OC_AGENT_HOST=collector.linkerd-jaeger:55678

I can see all server services vote-bot, voting, web, plus earlier linkerd-proxy got added to Jaeger UI service list.

2. My app

Based on emojivoto tutorial. I am trying to add trace in my project by basically coping emojivoto's codes.

My project has this structure:

• web is a frontend app.
• api-go is backend for frontend / a Go (Gin) server.
• api-go-grpc is a GRPC server.
• opal-server can be ignored because it is not related with this question.

I am using

annotations:
  linkerd.io/inject: enabled

for each my deployment yaml file such as this api-go deployment yaml file to use Automatic Proxy Injection.

After I start my app, all deployments successfully got meshed, and you can see my project traffic in the screenshot:

Based on emojivoto's commit that adding trace, below are my codes (basically just copy and paste from emojivoto):

I have this block of codes in my GPRC server api-go-grpc:

oce, err := ocagent.NewExporter(
	ocagent.WithInsecure(),
	ocagent.WithReconnectionPeriod(5*time.Second),
	ocagent.WithAddress(config.OpenCensusAgentHost+":"+config.OpenCensusAgentPort),  // this is "collector.linkerd-jaeger:55678" which also prints in my log
	ocagent.WithServiceName("api-go-grpc"))
if err != nil {
	log.Error().Err(err).Msg("ocagent.NewExporter")
}
trace.RegisterExporter(oce)

lis, err := net.Listen("tcp", ":"+config.GRPCPort)
if err != nil {
	log.Error().Err(err).Msg("net.Listen")
}

s := grpc.NewServer(grpc.StatsHandler(&ocgrpc.ServerHandler{}))

Also, I have this block of codes in my API Go (Gin) server api-go:

oce, err := ocagent.NewExporter(
	ocagent.WithInsecure(),
	ocagent.WithReconnectionPeriod(5*time.Second),
	ocagent.WithAddress(config.OpenCensusAgentHost+":"+config.OpenCensusAgentPort),  // this is "collector.linkerd-jaeger:55678" which also prints in my log
	ocagent.WithServiceName("api-go"))
if err != nil {
	log.Error().Err(err).Msg("ocagent.NewExporter")
}
trace.RegisterExporter(oce)

r := routes.SetupRouter()
_ = http.ListenAndServe(
	":"+config.Port,
	&ochttp.Handler{
		Handler: r,
	},
)

Both OpenCensus Agent addresses are collector.linkerd-jaeger:55678 which can be confirmed by my logs.

Also, the Linkerd screenshot below can double confirm my servers are pointing to collector.linkerd-jaeger:55678 correctly:

However, if I only deploy my app (not together with emojivoto app), I am not be able to see my app's linkerd-proxy in the Jaeger UI service list, neither of my other servers. So no trace at all.

Any guide to help further debug will be appreciated. Thanks!

[cpretzer]

@hongbo-miao do you have an ingress configured? The Linkerd distributed tracing mechanism expects that a trace has been started, and the proxy will add the span ID to the existing trace ID.

If there is no trace initiated, then you won't see any spans connected together. That being said, I'd expect that the Jaeger UI would show several traces without span IDs.

[cpretzer]

I injected by

kubectl get deployment ingress-nginx-controller --namespace=ingress-nginx --output=yaml | linkerd inject --ingress - | kubectl apply --filename=-

Then verified by

kubectl describe pods/ingress-nginx-controller-7c56cf7f5-tctm4 --namespace=ingress-nginx | grep "linkerd.io/inject: ingress"

It looks like the nginx ingress controller was properly injected with the Linkerd proxy. Did you follow the instructions to enable opentracing in the ingress?

I don't see the annotation in the ingress definition in the repo that you linked.

[hongbo-miao]

Thanks for the enable opentracing doc link! I wasn't aware of that before.

After ingress controller was properly injected with the Linkerd proxy. I patched my ingress-nginx-controller by

kubectl patch configmap ingress-nginx-controller --namespace=ingress-nginx --patch "$(cat ingress-nginx-controller-configmap-patch.yaml)"

ingress-nginx-controller-configmap-patch.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-opentracing: "true"
  jaeger-collector-host: collector.linkerd-jaeger.svc.cluster.local
  # if I use "jaeger-collector-host: jaeger-agent.default.svc.cluster.local"
  # it will fail later when apply my Ingress yaml file

The doc is using jaeger-collector-host: jaeger-agent.default.svc.cluster.local, if I use this I will get error when I apply my ingress file later by kubectl apply --filename=hm-ingress.yaml:

Error from server (BadRequest): error when creating "kubernetes/hm-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2021/07/07 15:24:01 [error] 3721#3721: Failed to construct tracer: Error resolving address: Name does not resolve
nginx: [error] Failed to construct tracer: Error resolving address: Name does not resolve
nginx: configuration file /tmp/nginx-cfg347946464 test failed

Here is my deployments:

So I updated to jaeger-collector-host: collector.linkerd-jaeger.svc.cluster.local. This time I can apply to my ingress yaml file hm-ingress.yaml, however, I still don't get any traces in Jaeger UI.

The doc says

NOTE: While the option is called jaeger-collector-host, you will need to point this to a jaeger-agent, and not the jaeger-collector component. Alternatively, you can set jaeger-endpoint and specify the full endpoint for uploading traces. This will use TCP and should be used for a collector rather than an agent.

Currently I am using jaeger-collector-host: collector.linkerd-jaeger.svc.cluster.local which is not an agent.
linkerd-jaeger currently only has a collector, no agent. Does that mean I need provide my own Jaeger agent? Thanks

(I will ask some of my coworkers who has done Jaeger and Ingress work without service mesh for help too see if they can spot some errors)

[cpretzer]

@hongbo-miao, the Linkerd distributed tracing docs use the zipkin-collector-host setting, so I'd recommend using that, unless you're using an external Jaeger instance, that is to say if you decide to use something other than linkerd-jaeger in the future.

That being said, the linkerd-jaeger chart deploys the jaeger-all-in-one image which deploys an agent listening on ports 5775, 6831, and6832. I think the port that you want to use is 6832 if you are going to use the jaeger agent and you will have to update the Service resource to expose that port.

I've never tried that myself, and since you're working through this PoC, I suggest sticking with the zipkin collector as outlined in the Linkerd docs.

[hongbo-miao]

Ah, it is using jaeger-all-in-one inside! Thanks for the info.
After changing to zipkin-collector-host: collector.linkerd-jaeger, now it works!
Not very intuitive though, as I thought it would be jaeger-collector-host: collector.linkerd-jaeger.

In future, I will change to external Jaeger instance and OpenTelemetry for further experiment. Will share back for how I make it. 😀

Thanks @cpretzer for support all time!

[cpretzer]

@hongbo-miao great to hear that you've got it working now.

The change to use OpenTelemetry is already in the edge releases thanks to this PR.

Regarding the jaeger-collector-host vs the zipkin-collector-host, please feel free to file a docs issue and PR to help us clarify the docs. 😄