Unified model for Open Telemetry tracing in JasperFx

[jeremydmiller]

See the HighWaterAgent line 102 where it has code commented out for MartenTracing. We need to do something about that. Some way maybe to inject the activity source.

[jeremydmiller]

Notes

• Inject ActivitySource into HighWaterAgent
• Make it possible to disable HighWaterAgent tracing
• New counter metric for the high water mark???
• Be nice to show the available metrics in command line output

All activity today

Marten Projections

• Span on checking for high water mark. Noisy. Could add a straight up counter instead
• execution, grouping, fetching for each projection and/or async subscription. Name today is marten.name.shardkey.execution / group / fetch. Move to name.shardkey. Tag as "marten"?
• "gap" & "processed" for each

Wolverine

• Span names are "send", "receive" and "process". Tags by destination & message type
• Always tags by tenant id, message type, destination
• Maybe use message name?
• "wolverine_node_assignments" -- might be noisy

[jeremydmiller]

Proposed Changes for Marten

• Do not ever put the database name into span names, but make the tenant.id always be a tag. Same for metrics
• So, Marten projection/subscription span names become "marten.[projection name].[shard key].execution/grouping/loading
• Only use "marten." prefix on all Marten span names. Do not embed database names in ever
• Ability to turn off verbose spans for the high water agent.
• Ability to turn off async daemon spans?

Proposed Wolverine Otel changes

• Today every activity for message processing is just process because that's an Otel thing. We always tag activity with tenant id, message type, and destination (the Uri the message is being executed at). Do we want to rename the span names to the message type? Prefix with "wolverine.****"? Some feedback is that folks want to be able to more easily pick out Wolverine behavior.
• Leave the existing metrics alone: https://wolverinefx.net/guide/logging.html#metrics

[richardszalay]

For Wolverine, I vote for the spans having the name of the message type, no prefix (a send/recv prefix, but not a wolverine prefix)

[PascalSenn]

i second @richardszalay here. Having at least the option to configure the handler name would be good.

[boeschenstein]

a less chatty "marten.daemon.highwatermark" would be very welcome