Document usage of TSDB and LogsDB

[felixbarny]

One of the benefits of the OTel native data streams are that we store logs and traces in LogsDB and metrics in TSDB (without having to manually define dimensions). This leads to significant storage reductions compared to the classic APM data streams.

This only applies when using the otel mapping mode.

We currently don't have docs about this but it seems like a recurring question and something we should advertise as one of the benefits of the OTel native data streams.

[felixbarny]

@alexandra5000 asked some good questions about this. I'll answer my view on it on this issue for posterity and so that others can chime in more easily:

1. You mentioned 'significant storage reductions compared to classic APM data streams.' Can we quantify this? Do we have internal benchmarks or estimates?
   For example:
   X% less storage for traces with LogsDB
   Y% less storage for metrics with TSDB vs ECS-based mappings
   Are there example workloads or ingest rates (e.g., 1000 spans/sec) I could use to illustrate this?

I think we should link to the ES documentation rather than re-stating it in the OTel documentation which might get stale. Fortunately, the docs for LogsDB and TSDB contain some numbers.

2. Are there any limitations or tradeoffs users should know about when using mapping_mode: otel, LogsDB, or TSDB? Are there scenarios where users should still use mapping_mode: ecs instead of otel?

I think we'll want to steer users towards the otel mode and don't confuse them too much with the other modes. They can get the glory details in the collector-contrib readme of the ES exporter but the Elastic OTel docs are our way to bring an opinionated and easy to digest view on it. We mainly use the other mapping modes internally and for migration/compatibility use cases.

3. TSDB vs. TSDS - what's the difference in Elastic's context? Internally, we refer to TSDB, but our public docs mainly mention TSDS (Time Series Data Stream). Are these terms interchangeable when documenting EDOT metric storage? From what I understood, it works like this:
   TSDB = Elasticsearch index mode (index.mode: time_series)
   TSDS = The full feature + usage pattern built on top of TSDB + data streams + ILM
   Should I align the doc's terminology to match what's already published - meaning use TSDS rather than TSDB?

I often use TSDB and TSDS interchangeably but you're right that it makes sense to be consistent and I think you're right in your understanding when which term is used. I guess that means we should use TSDS in the OTel docs.

[alexandra5000]

Thank you Felix!