Change query fuzz test for bottomk, topk

[SungJin1212]

This PR improves the query fuzz test to skip non-deterministic queries.
It can prevent test fail due to the non-deterministic queries like bottomk by (job) (pi(), days_in_month(-{__name__="test_series_a"} offset 1m17s)).

Change log

• We decide to change comparer for bottomk and topk.

Which issue(s) this PR fixes:
Fixes #6323

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

If you run the same bottomk or topk query against Prometheus or Cortex within a short period time it is highly likely to have the same result I think. I have tested this against Prometheus.

What we can do instead is to use a different result comparer to only check result series length rather than content if we see bottomk or topk in the query. Or we have dedicated test cases to test topk, bottomk and ignore them in regular test cases

[SungJin1212]

Is it sufficient to compare the length of the series?

Why do bottom or topk query results differ? I was thinking the reason is non-determinism. But, I feel like my thinking might be wrong.

[yeya24]

As you said, it is non deterministic but the non deterministic is not that high I think. If you run 100 queries probably the issue occurs once or twice. I think running the same query with c2 is also highly likely to produce the same result so this can be still flaky.

Is it sufficient to compare the length of the series?

We can add other verification if find doable but also reduce flakiness. At least comparing series length is better than not testing those functions at all.

[SungJin1212]

Ok, yes.. the same query with c2 is also flaky.
What about comparing # of samples? The series length is also flaky.

[yeya24]

I don't think number of samples is guaranteed the same. We randomized the timestamp of the first sample from each series so depending on your query you might be able to query some series but miss other series. When the result is non-deterministic depending on the returned series you will have different number of samples

[alanprot]

Can we change promsmith to not use bottomk/topk with some "time based" functions (day_of_week, time_of_day, etc) ? i think this is the bulk of the errors?

Maybe tbh we can even remove those functions from the fuzzy test, and and create a new fuzzy test just to test those functions?

[SungJin1212]

@alanprot
I haven't found the root cause yet.
I'm suspicious of all the same values. I tested the query bottomk by (job) (pi(), day_of_year(-{__name__="test_series_a"} offset 1m17s)) where I changed the values to return a random value (1~366) of day_of_year.
Then, there is a high chance of success.

Btw, it sounds good to create the fuzz test testing those functions.

FYI. I changed https://github.com/cortexproject/cortex/blob/master/integration/e2e/util.go#L290 to ref, err = app.Append(ref, series[i], start, float64(time.Hour*4114*time.Duration(i+j+1)))

[alanprot]

Maybe we can change this a bit to basically still compare all the results and just ignore the series - as we know if the results are the same, the series returned can be random?

[SungJin1212]

@alanprot
I run it, but the value could be different if we ignore the series.

instant query: topk without (__name__) (pi(), -{__name__="test_series_b",series="2"} offset -1m20s)
        res1: {} => -28 @[1733790497.075]
        {job="test", series="2", status_code="200"} => -50 @[1733790497.075]
        {job="test", series="2", status_code="404"} => -9 @[1733790497.075]
        res2: {} => -50 @[1733790497.075]
        {job="test", series="2", status_code="404"} => -9 @[1733790497.075]
        {job="test", series="2", status_code="500"} => -28 @[1733790497.075]

instant query: topk by (status_code, series, __name__) (
          time(),
          --{__name__="test_series_a",series="0"} offset -53s
        )
        res1: {} => 43 @[1733790497.075]
        {job="test", series="0", status_code="200"} => 24 @[1733790497.075]
        {job="test", series="0", status_code="400"} => 66 @[1733790497.075]
        {job="test", series="0", status_code="502"} => 53 @[1733790497.075]
        res2: {} => 24 @[1733790497.075]
        {job="test", series="0", status_code="400"} => 66 @[1733790497.075]
        {job="test", series="0", status_code="500"} => 43 @[1733790497.075]
        {job="test", series="0", status_code="502"} => 53 @[1733790497.075]

[alanprot]

Hum.. Interesting... Im trying to make sense of:

instant query: topk by (status_code, series, __name__) (
          time(),
          --{__name__="test_series_a",series="0"} offset -53s
        )
        res1: {} => 43 @[1733790497.075]
        {job="test", series="0", status_code="200"} => 24 @[1733790497.075]
        {job="test", series="0", status_code="400"} => 66 @[1733790497.075]
        {job="test", series="0", status_code="502"} => 53 @[1733790497.075]
        res2: {} => 24 @[1733790497.075]
        {job="test", series="0", status_code="400"} => 66 @[1733790497.075]
        {job="test", series="0", status_code="500"} => 43 @[1733790497.075]
        {job="test", series="0", status_code="502"} => 53 @[1733790497.075]

Do you know why one of the results does not have labels?

But seems that the problem is that the order of the results are not guaranteed? (as probably the result is ordered by series labels??)

[SungJin1212]

Yeah, the sample is the same if two series are the same.
But, the series set of the query results could be different.
The empty series seems to be weird..

But seems that the problem is that the order of the results are not guaranteed? (as probably the result is ordered by series labels??)

Yes, there seems to be non-determinism in query results.

[alanprot]

I think im ok or merging as is, and we can maybe improve after... let me try to rebase this.