Config for spans and opentracing_propagate_context

[s1rc]

Does this package support spans of traces as they pass through the services?

i.e.

1. nginx-ingress
2. nginx
3. PHP app

Currently I'm seeing 1 & 2 as a span of the same trace, but all subsequent requests in the actual app are showing as individual traces.

[svensp]

Let's say it is supposed to do that.

I could not find a standard for HTTP headers at the time of writing the package so it propagates(and reads) context through the X-TRACE header.

There seems to be some movement towards a standard header here: https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md but I would appreciate an example trace header from your setup to see where nginx places the trace context

[s1rc]

I'm still relatively new to tracing and jaeger, is there a way to find this out somewhere in the Jaeger UI?

I had followed this doc by nginx, https://www.nginx.com/blog/opentracing-with-nginx-ingress-controller-for-kubernetes/ only difference is that my nginx-ingress controller wasn't self compiled, it supports opentracing/jaeger.

Edit: Should note that I did do a curl -I of the url and there was no X-TRACE header present.

[svensp]

No you would check this in your php application.
Simply dump the information from \Request::header() and post the result here. Remove anything that might be confidential of course. The tracing context should be somewhere in there.

As for no X-TRACE header being present:

• You're looking at the response to the browser. There's no reason to include the trace there as far as I know. Tracing is done in server-to-server requests
• This package only reads X-TRACE. ipunkt/laravel-jaeger-guzzle adds a middleware to guzzle to read the current span and send the request with matching X-TRACE. It is not widely used thought as most of our traced server-to-server traffic currently going through rabbitmq.

[s1rc]

I've attached a sanitized set of headers as json. Doesn't look to be any X-TRACE header present. Perhaps my nginx.conf for Laravel doesn't forward that on to php-fpm/the app.

Headers.txt

[svensp]

Indeed there is nothing there. Did you add the location-snippet as mentioned in the tutorial?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: opentracing-ingress
  annotations:
    nginx.org/location-snippets: |
      opentracing_propagate_context;
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: app1-svc
          servicePort: 80

According to the tutorial that line should pass the tracing context to the application(well, it's nginx but it should just pass through that)

[s1rc]

Yeah that is there in my ingress yml.

I found this reference to OpenTracing directives: https://github.com/opentracing-contrib/nginx-opentracing/blob/master/doc/Reference.md

I see that there is a opentracing_fastcgi_propagate_context and tried replacing opentracing_propagate_context in my location location ~ \.php$ {} nginx.conf block. Now there is a request header with key uber-trace-id, I've attached another json txt of the sanitized output.

Headers-with-opentracing_fastcgi_propagate_context.txt

location ~ \.php$ {
      # Opentracing
      opentracing_operation_name $uri;
      opentracing_tag app app-nginx-php-fpm;
      opentracing_fastcgi_propagate_context;
      try_files $uri /index.php =404;
      fastcgi_split_path_info ^(.+\.php)(/.+)$;
      #fastcgi_pass unix:/run/php/php7.2-fpm.sock;
      fastcgi_pass 127.0.0.1:9000;
      fastcgi_buffers 16 16k;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      include fastcgi_params;
}

[svensp]

That should be what we're looking for.

I'll have to do some digging to find out how to pass only that Id to my jaeger lib - the context there requires some other parameters to be set.

[s1rc]

If you need any more info or to test I'm happy to help where I can.

[svensp]

I have created a branch uber_trace_id where I'm trying to get this to work but as of right now it is completly untested, pretty much just 'this might work that way'

I'll try to set up a minikube test env sometimes tomorrow

[s1rc]

Testing with that branch I'm getting exception:
7#7: *24 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught Error: Call to a member function addLog() on null in /var/www/vendor/ipunkt/laravel-jaeger/src/Context/SpanContext.php:215

[svensp]

I'm trying to build a test setup in minikube that works similar to yours. So far I have the standalone working and sending traces from laravel but not the part about sending traces from the nginx ingress.
Still, the error about addLog on null should be fixed now.

[s1rc]

Thank the addLog error is fixed.

Jaeger is still showing the nginx/ingress and Laravel traces as different spans.

[svensp]

I finally got it working on my end.

It looks to me like the tutorial you posted above is using a somewhat out of date version of the nginx-ingress.

The current config file directives to start opentracing with jager are here:

• https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-opentracing
• https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#jaeger-collector-host

The current annotation to add a snippet to the location is here:

• https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#configuration-snippet

For reference here is the ConfigMap which enabled jaeger traces from the nginx-ingress for me:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-load-balancer-conf
  namespace: kube-system
data:
  hsts: "false"
  map-hash-bucket-size: "128"
  enable-opentracing: "True"
  jaeger-collector-host: "agent.spacebattle.svc.cluster.local"
  jaeger-service-name: "nginx-ingress"
  opentracing: "True"

And the ingress:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: spacebattle
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      opentracing_propagate_context;
spec:
  rules:
  - host: spacebattle.cluster.test
    http:
      paths:
      - path: /
        backend:
          serviceName: spacebattle-asterkeks-de-http
          servicePort: web

The UBER-TRACE-ID sent by the ingress looks like this: 950798db9aa1e246:20ea53ef7306ae4a:950798db9aa1e246:1 - that is the full information required by the jaeger lib I use in the background.

I see the nginx-ingress and laravel spans as part of a single trace now.

[svensp]

Looking at the location entry you posted earlier I think you configured your laravel application nginx to post the trace instead of the ingress controller - I'm seeing 2 spans from the ingress controller alone as well - and for some reason, probably a version difference, your application nginx sends a shortened version of the uber-trace-id which my backend lib can't deal with

[s1rc]

I've used your annotations for the actual ingress and now it seems to work. Before I was enabling opentracing on just the ingress instead of globally, so now with it globally enabled I need to figure out how to disable tracing on certain ingresses (i.e. I don't need to trace prometheus).

Now I'm able to trace through the call, however when spanning the trace from nginx-ingress > nginx > Laravel, it lists only nginx-igress: / <Initial Span ID> so it's really hard to find which trace span to diagnose (they are all the same at the root). Before opentracing_propagate_context tracing the Laravel only trace would show App name and full URL. I'm not sure if that's an nginx-ingress/nginx config or not.

Also as a suggestion, would it be possible to override the service name used? It looks to use the app name, but when we have multiple apps (dev/staging/prod) with the same env app name, it would be good to be able to differentiate in Jaeger UI.