feat: ingestion metrics for system monitoring

[Ziinc]

System monitoring for users:

• provisions system-managed sources for each User
• Each system source is specific to a type of event
• otel-format metrics are collected and then exported into a :metrics source pre-aggregated.
• a centralized OtelMetricExporter is used to avoid running :keep callback multiple times for each each SourceSup
• export_callback/2 overrides default library behaviour
• we rely on string metadata keys as a declarative way of deciding what metadata keys to keep and set as otel metric attribute.

for user-specific logs, these are routed to the source based on whether specific identifying metadata keys are present. If the log is pertaining to a specific source/backend belonging to the user, then it will get ingested into that source.

4 specific user metrics are implemented

• egress on http adaptors
• total ingested events
• total ingested bytes
• total bytes processed for queries

Refactors tests to avoid testing implementation.

• docs
• test
• log drains egress metric
• endpoints query bytes metric

Closes ANL-1202

[josevalim]

Hi @Ziinc! 👋

Comments inline!

[josevalim]

For completeness, you could use custom Ecto.Types to store those aleady as lists in the database: https://hexdocs.pm/ecto/Ecto.Type.html - although I am not expecting this code to be performance sensitive enough to matter.

[josevalim]

This is a separate conversation but I wonder if something could be done to remove the number of "sources" here.

Also, it is worth noting that you are caching whole structs, but you may only need the user_id field. For example, Cache.get_endpoint_query seems to be used exclusively here and you don 't care about the other fields, so there is no need to store or read them. You may benefit from adding a get_user_id_by_endpoint_query. This may also apply to other operations, but please double check!

[Ziinc]

yes, i intend to refactor that off as it isn't as performant, and we should be able to derive the user_id from wherever the telemetry is getting emitted and set it on the metadata, thereby bypassing the whole need to fetch from Cachex

removed rejected events docs