Tracing support in eCAL core

[hannemn]

Requirement:
Currently there is no possibility to have introspections into the data / program flow of internal ecal states. This shall allow ecal users and developers to retrieve information about latency, transport layers, stalls.

What do we want to trace?
We would like to trace communication through the ecal stack

• publish/subscribe: trace from point of sending a message through the different layers to receiving that message on different processes / hosts and user execution callback

Architecture Idea

• Tracing data is collected by each node in an efficient application-specific manner
• Collector application retrieves data from nodes (e.g. by subscribing to a specific topic)
• Data is transformed to high-level format by collector to be displayed in a frontend (e.g. Jaeger)

Pub-Sub

Requirements

• Messages must be observed from End to End across multiple machines
  • --> Trace ID must be shared between Sender and multiple Receivers
  • Trace ID can be computed from Publisher Entity ID + Message counter, as that information is available on both Sender and Reciever for each message
• Spans of Receiver must state the correct Layer-Span of the remote Sender Node
  • --> Span ID of Sender-Node Layer Spans must be shared between Sender and multiple Receivers
  • Sender Layer only occurs 1x per span => can be set deterministically with an Enum. Receiver knows the layer it received message from and can set the parent span ID to the same deterministic enum values
• User Callback must be monitored separately
• Tracing must hurt performance as little as possible (--> doesn't need to be realtime accessible)
• Trace must provide a possibility to obtain information where it originated from

  • Hostname + eCAL Node Name of Sender (-> Trace meta information)

  • Topic Name (-> Trace meta information)

  • Hostname + eCAL Node Name of each Receiver (-> Span meta information)

    • Metainformation ID is set to publisher / subscriber entity ID (i.e. it's different on sender and each receiver, so it can be used to identify sender and reciever)
    • monitoring information needs to be stored (or at least a subset of it). Monitoring Information contains a entity id -> Hostname / Node Name / Topic Name mapping

Prerequisites

• Time Synchronization between Sender Node and Receiver Node is required

Information for each Span

• Trace ID (est 16 Bytes)

• Span ID (est 8 Bytes)

• Parent Span ID (est 8 Bytes)

• Start Timestamp (est 8 Bytes)

• End Timestamp (est 8 Bytes)

• Metainformation ID (est 8 Bytes)

• Span Type (est 8 Bytes)_

  --> Estimated 57 Bytes per Span

Spans Publish Subscribe:

1 x Sender Node

• Send Operation (User Send Call -> Message has left the node)
  • Serialization
  • UDP Layer Send
  • TCP Layer Send
  • SHM Layer Send

n x Receiver Node (shall state the correct layer-span of sender node as parent)

• Receive Operation (eCAL got serialized Message -> User callback has finished)
  • UDP Layer Receive
  • TCP Layer Receive
  • SHM Layer Receive
  • User Callback

Server/Client

Spans:

• Service Call (Client CallService -> Callback executed)
  • Server Service Execution
  • Callback execution

[KerstinKeller]

Currently open points

• tracing format is jsonl, which is a lot less compact than a binar format like json
• process ID is redundandly recorded in every span data
• process name is not recorded (should be recorded for better readability)
• trace and span id's should be stable so we don't export duplicate data into the databases