Implicitly use last_over_time for time-series aggregations

[dnhatn]

Currently, time-series aggregations require two explicit aggregations: an over-time aggregation and an outer aggregation. This means a query like TS metrics | STATS AVG(memory_usage) calculates the average of all memory_usage values, functioning in a document-centric way like the FROM command. This is not the desired metrics-centric behavior for the TS command.

This change proposes to implicitly use last_over_time for time-series aggregations that do not specify an over-time aggregation.

For example:

TS metrics | STATS AVG(memory_usage)

becomes:

FROM metrics
| WHERE memory_usage IS NOT NULL
| STATS mem = last_over_time(memory) BY _tsid
| STATS AVG(mem) as `AVG(memory_usage)`

---

TS metrics | WHERE TRANGE(5m) | STATS AVG(memory_usage)

becomes:

FROM metrics
| WHERE memory_usage IS NOT NULL AND @timestamp < now() AND @timestamp >= now() - 5m
| STATS mem = last_over_time(memory) BY _tsid
| STATS AVG(mem) as `AVG(memory_usage)`

---

And:

TS metrics | WHERE TRANGE(5m) | STATS AVG(memory_usage), SUM(rate(requests)) BY cluster, TBUCKET(1m)

becomes:

FROM metrics
| WHERE (memory_usage IS NOT NULL OR requests IS NOT NULL) 
                AND @timestamp < now() AND @timestamp >= now() - 5m
| STATS m = last_over_time(memory), r=rate(requests), cluster=VALUES(cluster) BY _tsid, TBUCKET(1m)
| STATS AVG(m) as `AVG(memory_usage)`, SUM(r) as `SUM(rate(request))` BY cluster

---

This change does not require a default lookback window. Users can specify the lookback window in the WHERE clause with TRANGE, as shown in the last example.

[dnhatn]

@kkrik-es @martijnvg One question here is whether we should support this behavior with a warning that recommends users move to explicit OVER_TIME aggregations. While this makes queries more concise, we should consider if users will be okay with this default implicit behavior.

[dnhatn]

This is the main change.

[kkrik-es]

@kkrik-es @martijnvg One question here is whether we should support this behavior with a warning that recommends users move to explicit OVER_TIME aggregations. While this makes queries more concise, we should consider if users will be okay with this default implicit behavior.

I think we should document expectations and best practices but have the query return without warnings. This way, we can have may existing queries to work by switching to TS, with the assumptions that also apply to PromQL.

[kkrik-es]

Looks good!

[kkrik-es]

Thanks Nhat, let's proceed with this. I'm still thinking about this example:

TS metrics | STATS AVG(memory_usage)

This will scan all data so it'll likely timeout or run out of resources. I wonder if we should be including the default lookback window, even if it's just for this case. Then again, there are many ways that this can happen in ES|QL, so we can delay that for PromQL proper.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[dnhatn]

This will scan all data so it'll likely timeout or run out of resources. I wonder if we should be including the default lookback window, even if it's just for this case. Then again, there are many ways that this can happen in ES|QL, so we can delay that for PromQL proper

Sure @kkrik-es. I will do this in a follow-up. Thanks for the review.