Discuss use of OpenTelemetry in Mojaloop.

[bushjames]

Raised by @vijayg10 and @kalinkrustev.

See https://opentelemetry.io/

Current Mojaloop codebase uses a system based on w3c recommendations for distributed tracing (https://www.w3.org/TR/trace-context-1/). See: https://github.com/mojaloop/mojaloop-specification/blob/c65d78961d204e19a7afa3aa84d646ac8ba54e93/fspiop-api/documents/Tracing%20v1.0.md

(OpenTelemetry is a set of tools (SDK etc...) for implementing distributed tracing based on this standard also.)

Current solution does not propagate trace headers across kafka topics.

Proposed solution is to use OpenTelemetry SDK to handle this.

Also proposed it the possibility to use OpenTelemetry to improve the detail provided in existing traces by utilising OT zero-code option enabled by OT k8s operator and pod annotations. OT operator adds init container which mounts OT SDK in main container and sets node env var to load this at startup. (this may require changes to master HEML charts and/or values files to enable). Some discussion about how to best expose this functionality to adopters/users. Currently all deps are deployed separately. Adding OT to backing services chart(s) is acceptable.

Note: Current solution was based on indexing traces in elastic search. ES has turned out to not be a cheap or easy to use at production scale. We may want to consider changing what we deploy to a newer, more appropriate solution for storing and analysing trace data.

No objections raised from anyone present on DA call.

Conversation to continue at next available DA session:

1. OT SDK did not perform "well", fast.
2. OT may not be suitable for use in production.

[bushjames]

OpenTelemetry operator being trialled by Infitx. More feedback to come.

[bushjames]

@vijayg10 reports that OpenTelemetry is being used by Infitx in test environments only as it introduces some mutations of K8S resources. Infitx do not believe it is safe for production due to security concerns around how node.js telemetry is injected into containers. Also, a huge volume of data is generated which is proving impractical. It is possible to address these issues by not using the auto-inject mechanism but this requires further work and investigation.

[bushjames]

Reviewed at DA call today. Still requires work to enable this for production in a secure way.

[bushjames]

VK reports no progress made yet towards adapting this for production deployments. PoC is in progress to align logging with open telemetry to enable correlation between logs and ot events. We are expecting PB to present on this topic in an upcoming DA call.