Thanos-receiver How to handle timestamps increasing per series but can be x days old

[pmcarreiro]

Hello all,

In my project, I'm unable to control the client that ingests information in my Thanos deployment.
The information always comes with a timestamp increasing, but it's only sent when my clients have a connection.
This means that I can receive information from the past 3 days, or older.

From what I'm able to understand, the out-of-order considers the current time to reject information.

If i increase the out-of-order, this means that i will need more memory in each thanos-receiver, correct?

A good solution for me was if we were able to configure that the out-of-order should use the latest series timestamp and not the current time.

What i should do in order to have a good balance between server resources and the data to be ingested?

Many thanks and best regards,
Paulo Carreiro

[dosubot[bot]]

To handle timestamps that increase per series but may be several days old, you can enable support for out-of-order samples in Thanos by setting the tsdb.out-of-order.time-window flag on the Receiver. This allows the system to ingest samples that arrive out of order. However, enabling this feature will increase the TSDB's memory usage proportional to the number of out-of-order samples ^[1].

Additionally, ensure that the --compact.enable-vertical-compaction flag is enabled on the compactor to handle overlapping blocks that result from out-of-order samples ^[1].

Unfortunately, there isn't a built-in feature to configure the out-of-order handling to use the latest series timestamp instead of the current time. You might need to adjust the tsdb.out-of-order.time-window to accommodate the maximum expected delay in data arrival while monitoring resource usage to find a balance that works for your setup.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[pmcarreiro]

Hello,

Thanks for the clarification, but this means there's no plan to change that behaviour, like we have in mimir?

Many thanks,
Paulo