Services not linking in Elastic APM service Map

[akashdeep-sharma]

Service 1 generates random uuid and sends to service 2 via HTTP
Service 2 says ok and then further sends it to service 3 via AWS SQS
Service 3 says ok and then just echoes it maybe.
Service 2 inserts the message in the SQS along with the context of active span
Service 3 reads from the queue
So that we can see how to correlate when the request is passing through one service to another.

If I disable automatic instrumentation in service 3 the service map for service 3 disappears

Question 1- How to get a service on the service map
Question 2- Is it possible that we get the context from the sqs in service 3 and change the context of the active span in the service 3
CODE SNIPPET for service 2

@Get('/sendMessage')
  async sendMessage(@Headers() requestHeaders: any) {
    console.log(`Request Headers : ${JSON.stringify(requestHeaders)}`);
    const contextWeGot = JSON.parse(requestHeaders.ctx);
    const activeSpanContext = trace.getSpanContext(context.active());
    const newCTX = trace.setSpanContext(context.active(), contextWeGot);
    track.startActiveSpan('Controller span', {}, newCTX, async (span) => {
      console.log('Something logged');
      span.setAttribute('custom.method', 'PAtch');
      span.end();
    });
    return await this.appService.InsertMessage(activeSpanContext);
  }

CODE SNIPPET for service 3

while (true) {
    const recievedMessages = await appService.recieveMessage();
    let activeSpan = trace.getSpan(context.active());
    if (recievedMessages) {
      for (let message of recievedMessages) {
        let contextWeGot = JSON.parse(message['Body'])['ctx'];
        const newCTX = trace.setSpanContext(context.active(), contextWeGot);
        track.startActiveSpan('HTTP GET', {}, newCTX, async (span) => {
          console.log(`Processing Message : ${message['Body']}`);
          span.end();
        });
      }
    }
    await sleep(10000);
  }

[dyladan]

Question 1 sounds like a question for Elastic APM and not an opentelemetry issue.

Question 2:

Unless I'm missing something the line let activeSpan = trace.getSpan(context.active()); is doing nothing.

The code for setting the new context active looks correct to me, but I'm not an expert in Elastic APM and there might be a missing piece such as timeouts or something like that that I'm not aware of. Can you confirm that the contextWeGot variable contains the context you expect? If you do the same thing and use another exporter like jaeger, does the trace look like you expect?

[akashdeep-sharma]

contextWegot contains the context that we get from the sqs message that the recieveMessage() returns. But when im trying to set the context contextWeGot for the outer span that is wrapping the while loop and the automatic span that is getting generated.
We are able to set the context for the HTTPGet span in startActiveSpan() function but not able to set the context for the activeSpan

[cyrille-leclerc]

Hello @Akash19980 , can you please create a ticket reporting this problem on https://github.com/elastic/apm-server so we can help you on this problem that may relate to the support of OpenTelemetry in Elastic Observability. Cc @axw

[akashdeep-sharma]

Yes I will but also need help in context propagation in opentelemetry
Posted on Elastic
link to thread- https://discuss.elastic.co/t/figure-out-how-to-link-services-using-manual-instrumentation/283027

[cyrille-leclerc]

Cross posting with https://discuss.elastic.co/t/figure-out-how-to-link-services-using-manual-instrumentation/283027

For proper visualization of your messaging trace in the service map of Elastic Observability, you must comply with the OpenTelemetry Semantic Conventions for messaging systems and particularly:

• Specify the following attributes messaging.system: AmazonSQS , messaging.destination: MyQueue, messaging.destination_kind: queue (spec details here)
• Specify the right span kind PRODUCERand CONSUMER (spec details here)
• Specify the right messaging.operation attribute value send, receive, and process (spec details here)