ESQL: Compute engine support for stateful grouping functions #112757

nik9000 · 2024-09-11T17:27:35Z

This adds support to the compute engine for "stateful grouping functions". Think of these like ExpressionEvaluators but they can:

Encode extra state to be passed to the coordinating node as part of the agg
Use that extra state to transform the group keys on the coordinating node
Apply a transformation using the output after the aggregation is complete
Use a different intermediate representation as final representation (think, "I group on an integer, but when finished I transform into a string")

nik9000 · 2024-09-11T17:28:41Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+import java.util.ArrayList;
+import java.util.List;
+
+public record GroupingKey(AggregatorMode mode, Thing thing) implements EvalOperator.ExpressionEvaluator {


I've somewhat mirrored the way we do grouping aggs with this and it seems to have worked out fairly well. It's not perfect, but it's a lot less confusing than I thought it would be.

nik9000 · 2024-09-11T17:29:32Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

                }

-                blockHash.add(wrapPage(page), add);
+                blockHash.add(new Page(keys), add);


I'd like to modify BlockHash to take a Block[] with the blocks in the right position. But that seems like something for another time.

nik9000 · 2024-09-11T17:29:55Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java


-                blockHash.add(wrapPage(page), add);
+                blockHash.add(new Page(keys), add);
                hashNanos += System.nanoTime() - add.hashStart;


It's probably worth timing the evaluation here.

nik9000 · 2024-09-11T17:31:01Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

+            int[] aggBlockCounts = new int[aggregators.size()];
+            for (int a = 0; a < aggregators.size(); a++) {
+                aggBlockCounts[a] = aggregators.get(a).evaluateBlockCount();
+                blockCount += aggBlockCounts[a];


I found it a lot easier to read if I encoded the resultOffsets into the GroupKeys. It'd be even easier to read if the offsets were encoded into the aggregators too. Or if we returns Block[].

nik9000 · 2024-09-17T21:53:54Z

@jan-elastic have a look at this one. It's closer, I think.

Once we can figure out how this is supposed to work in the unit test I think we can iterate some more on the language side to figure out how to make it build that.

...in/ml/src/main/java/org/elasticsearch/xpack/ml/aggs/categorization/TokenListCategorizer.java

Categorize

nik9000 · 2024-09-19T14:32:37Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+
+        @Override
+        public void replaceIntermediateKeys(BlockFactory blockFactory, Block[] blocks) {
+            // NOCOMMIT this offset can't be the same in the result array and intermediate array


I bumped into this a few days ago and I think I need to dig some more - in this brave new world there's two "shapes" of data coming out of these grouping functions - the intermediate shape and the final shape. This is pretty similar to how aggs work - which is something I never fully understood to be honest. Anyway, I'm using resultOffset here - but that's the result offset of the intermediate data. not the final offset. So it can't be right.

dnhatn

Thanks @nik9000. I might be missing some context, but it seems we're trying to include SerializableTokenListCategory alongside the aggregated results for each driver. Could we resolve this by adding infrastructure to support SerializableTokenListCategory (or a variant) as the new block hash key?

dnhatn · 2024-09-22T05:24:59Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+                    CategorizationPartOfSpeechDictionary.getInstance(),
+                    0.70f
+                );
+                evaluator = new CategorizeEvaluator(


It seems the CategorizeEvaluator will be executed twice: here and in toEvaluator?

I was wondering the same.

However, when running the CategorizeOperatorTests it seems that Categorize::toEvaluator is never executed.

ivancea

Partial review with some questions

ivancea · 2024-10-01T13:41:14Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

+            int offset = 0;
+            for (int g = 0; g < groups.size(); g++) {
+                blocks[offset] = keys[g];
+                groups.get(g).finish(blocks, selected, driverContext);


No offset passed to the finish() here? How does it know where to place the blocks?

ivancea · 2024-10-01T13:42:47Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+        return mode.isInputPartial() ? thing.evalIntermediateInput(blockFactory, page) : thing.evalRawInput(page);
+    }
+
+    public int finishBlockCount() {


Nit: We're calling this "finish", while in the aggregator it's "evaluate". Some reason to keep those names separated? From what I understand, the operation is nearly the same (?)

ivancea · 2024-10-01T13:44:54Z

.../esql/compute/src/main/java/org/elasticsearch/compute/operator/OrdinalsGroupingOperator.java

-                    maxPageSize,
-                    false
+                List.of(
+                    // NOCOMMIT double check the mode


Commenting just in case this was forgotten

ivancea · 2024-10-01T14:38:49Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+import java.util.List;
+
+public record GroupingKey(AggregatorMode mode, Thing thing, BlockFactory blockFactory) implements EvalOperator.ExpressionEvaluator {
+    public interface Thing extends Releasable {


Is "Thing" the final name here?

A smidgen of javadoc would also greatly help understand what the "Thing" is supposed to do.

nik9000 · 2024-10-04T13:02:12Z

Could we resolve this by adding infrastructure to support SerializableTokenListCategory (or a variant) as the new block hash key?

I missed this comment two weeks ago and it's pretty important.

So I didn't make it a special BlockHash implementation because I thought "we need this to be available at any positions and to be able to combine with other functions". But that's not actually true. We're fine, at least right now, with not doing that. With only supporting GROUP BY CATEGORIZE(message). No other groupings.

In fact! When we got to support stuff like GROUP BY DATE_TRUNC(timestamp), CATEGORIZE(message), we want a very specific behavior for CATEGORIZE.

I suppose my objection is "this is a function in my mind". But I think your right. It doesn't have to be.

I'm going to see what it'd look like to build this as a specialize BlockHash implementation.

nik9000 · 2024-10-04T13:07:20Z

Thanks @iverase for pointing out that comment. I'd like to blame missing it on being sick. But I dunno. It's really important!

nik9000 · 2024-10-04T16:12:26Z

OK! I tried it and I think it will work but my head's spinning a bit so I'm going to work on something easier for a little while and get back to it.

alex-spies

Had a first look from the query planning/optimization perspective and mostly left a bunch of questions :)

alex-spies · 2024-10-07T08:47:22Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

    }
+
+    public GroupingKey.Supplier groupingKey(Function<Expression, ExpressionEvaluator.Factory> toEvaluator) {
+        return mode -> new GroupingKeyFactory(source(), toEvaluator.apply(field), mode);


Question: Is it correct that we plan to have expressions like STATS ... BY CATEGORIZE(SUBSTRING(field, 10)) be fully evaluated as part of running the aggregation? I'm asking because we're passing an expression evaluator here, which makes this feel close to a normal function that could be run in an EVAL (except that it needs another counterpart on the coordinator node).

Currently, we move evaluations out of the agg into a separate EVAL step before, so that STATS ... BY 2*field becomes EVAL $$temp = 2*field | STATS ... BY 2*field and I wonder how this will work with CATEGORIZE.

I'd say ideally that works. Even though the usecase is pretty narrow, I think it's a bit ugly if it doesn't work.

What problem occurs with categorize? I think the following queries are equivalent:

... | STATS ... BY CATEGORIZE(SUBSTRING(field, 10)) ... | EVAL s = SUBSTRING(field, 10) | STATS ... BY CATEGORIZE(s)

More complicated is the following:

... | STATS ... BY SOME_FUNCTION(CATEGORIZE(field))

Currently there is no function that works on a category. However, if we let CATEGORIZE output the category regexes (as a string), then you could do:

... | STATS ... BY SUBSTRING(CATEGORIZE(field), 10)

which I guess is a pain to evaluate.

It's somewhat equivalent to

... | STATS ... BY c = CATEGORIZE(field) | EVAL s = SUBSTRING(c, 10) | STATS BY ... c

But first you need to finalize the CATEGORIZE on the coordinating node, next do the SUBSTRING, and finally aggregate the first STATS one more time. Unless I'm overlooking something...

... | STATS ... BY SOME_FUNCTION(CATEGORIZE(field))

is very interesting. For now, I think we should just consider this invalid, but there's options to make this work:

CATEGORIZE actually is a pipeline breaker, i.e. it needs to be finalized on the coordinator. We could declare it its own, standalone physical execution plan node, and the execution plans could be something like categorize (initial) -> exchange (send to coord.) -> categorize (final) -> eval (some_function) -> stats (on the coordinator). In cases without a function on top of the categorize, the optimizer could fuse the stats and the categorize into one combined node.

We could use a 2-phase approach, where we determine the full categorizer in the first phase, so that categorize can be executed like a regular function in the second phase.

... | STATS ... BY CATEGORIZE(SUBSTRING(field, 10)) ... | EVAL s = SUBSTRING(field, 10) | STATS ... BY CATEGORIZE(s)

For this equivalence, our optimizer already has a mechanism to turn the former into the latter; we just need to teach it about CATEGORIZE.

alex-spies · 2024-10-07T08:59:50Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+                    new CategorizationBytesRefHash(new BytesRefHash(2048, context.bigArrays())),
+                    CategorizationPartOfSpeechDictionary.getInstance(),
+                    0.70f


nit: let's extract these constants into final statics with good names, or let's have a helper method for creating the token list categorizer since there's some repetition in this file.

alex-spies · 2024-10-07T09:32:23Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+        }
+    }
+
+    private static class GroupingKeyThing implements GroupingKey.Thing {


Let's give this a better name + javadoc

alex-spies · 2024-10-07T09:43:23Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+
+        @Override
+        public void fetchIntermediateState(BlockFactory blockFactory, Block[] blocks, int positionCount) {
+            blocks[resultOffset + 1] = buildIntermediateBlock(blockFactory, positionCount);


What meaning does resultOffset + 1 have vs. resultOffset? Is this the offset of the block holding the final category?

Can we abstract this into a private member variable, so this becomes clearer, like blocks[finalCategoryBlockOffset] or so?

alex-spies · 2024-10-07T09:52:59Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+    }
+
+    private static class GroupingKeyThing implements GroupingKey.Thing {
+        private final int resultOffset;


Maybe this could be more meaningful?

Suggested change

private final int resultOffset;

private final int initialCategoryBlockOffset;

Provided my suggested name is even correct :D

alex-spies · 2024-10-07T10:06:48Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+
+        @Override
+        public void close() {
+            evaluator.close();


It looks like we also have to close the categorizer?

alex-spies · 2024-10-07T10:33:50Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+                for (int oldCategoryId = 0; oldCategoryId < count; oldCategoryId++) {
+                    SerializableTokenListCategory category = new SerializableTokenListCategory(in);
+                    int newCategoryId = categorizer.mergeWireCategory(category).getId();
+                    System.err.println("category id map: " + oldCategoryId + " -> " + newCategoryId + " (" + category.getRegex() + ")");


alex-spies · 2024-10-07T10:38:22Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+    record GroupingKeyFactory(Source source, ExpressionEvaluator.Factory field, AggregatorMode mode) implements GroupingKey.Factory {
+        @Override
+        public GroupingKey apply(DriverContext context, int resultOffset) {
+            ExpressionEvaluator field = this.field.get(context);


Why do we need an expression evaluator here? So we can compute expressions like STATS ... BY CATEGORIZE(SUBSTRING(field, 10))?

If CATEGORIZE is only run as part of a STATS, this may not be the best way to generalize it to nested expressions; for aggs, our current way is to extract nested expressions into an EVAL - like EVAL $$temp = SUBSTRING(field, 10) | STATS ... BY CATEGORIZE($$temp).

Related and important: Currently our optimizer expects the BY ... groups to contain only field and reference attributes after the initial substitution step; I understand that CATEGORIZE breaks this assumption, so we'll have to go and update some optimizer rules to expect references, fields and CATEGORIZE in the groupings.

Bleh, never mind. I realized we require running the initial categorization as an Eval and the second one as part of the final Aggregate, like so

Coordinator: AggregateExec[[categorize(x){r}#57],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#57],FINAL,[categorize(x){r}#57, count{r }#62, seen{r}#63],58] \_ExchangeSourceExec[[categorize(x){r}#57, count{r}#62, seen{r}#63],true] Local: ExchangeSinkExec[[categorize(x){r}#72, count{r}#77, seen{r}#78],true] \_AggregateExec[[categorize(x){r}#72],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#72],INITIAL,[categorize(x){r}#72, count {r}#81, seen{r}#82],58] \_EvalExec[[CATEGORIZE(x{f}#75) AS categorize(x)]] \_FieldExtractExec[x{f}#75]<[]> \_EsQueryExec[test], indexMode[standard], query[][_doc{f}#83], limit[], sort[] estimatedRowSize[104]

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

alex-spies

Hey, I think I'm starting to wrap my head around this.

I have a couple concerns. The biggest one is that this looks like we're changing how aggs work quite a lot in a way that seems very specific to CATEGORIZE; I wonder if this is needed.

More specifically, I can see that the physical plans for a query like FROM test | STATS count(*) BY categorize(x) look like:

Coordinator:
AggregateExec[[categorize(x){r}#57],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#57],FINAL,[categorize(x){r}#57, count{r
}#62, seen{r}#63],58] // final categorization, aggregation, and replacing category ID by category STRING
\_ExchangeSourceExec[[categorize(x){r}#57, count{r}#62, seen{r}#63],true]

Local:
ExchangeSinkExec[[categorize(x){r}#72, count{r}#77, seen{r}#78],true]
\_AggregateExec[[categorize(x){r}#72],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#72],INITIAL,[categorize(x){r}#72, count
{r}#81, seen{r}#82],58]
  \_EvalExec[[CATEGORIZE(x{f}#75) AS categorize(x)]] // initial categorization
...

So we

first run an EVAL to get the initial categorization, then
we perform the initial aggregation based on the computed, local category id, and finally
on the coordinator, we compute the global category id, do the final aggregation, and transform the output, all in the final AggregateExec node.

I'm afraid that may lead to a lot of special casing, and it prevents us from easily computing stuff like STATS ... BY SUBSTRING(CATEGORIZE(field), 10), because that requires to compute the final category id before performing the aggregation.

So, my main point is: Do the aggregation and the categorization aggregation need to be so intimately intertwined? Couldn't we make this work with fewer changes by pulling this apart into an execution plan like

EvalExec[[transform_categorize(x)]] // replace by category ID by category STRING
\_AggregateExec[[categorize(x){r}#57],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#57],FINAL,[categorize(x){r}#57, count{r
}#62, seen{r}#63],58] // ONLY aggregate based on category ID
  \_EvalExec[[final_categorize(x)]] // compute final category ids
    \_ExchangeSourceExec[[categorize(x){r}#57, count{r}#62, seen{r}#63],true]

Local:
ExchangeSinkExec[[categorize(x){r}#72, count{r}#77, seen{r}#78],true]
\_AggregateExec[[categorize(x){r}#72],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#72],INITIAL,[categorize(x){r}#72, count
{r}#81, seen{r}#82],58]
  \_EvalExec[[CATEGORIZE(x{f}#75) AS categorize(x)]] // initial categorization
...

Of course, this requires that the aggregation passes through the mapping from category id to category string rather than consuming both immediately.

alex-spies · 2024-10-07T12:34:44Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+
+        Block evalIntermediateInput(BlockFactory blockFactory, Page page);
+
+        void fetchIntermediateState(BlockFactory blockFactory, Block[] blocks, int positionCount);


naming nit: I've seen in the implementation that this actually builds the intermediate state, rather than just fetching it.

alex-spies · 2024-10-07T12:54:33Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+        GroupingAggregator.Factory valuesAggregatorForGroupingsInTimeSeries(int timeBucketChannel);
+    }
+
+    public static GroupingKey.Supplier forStatelessGrouping(int channel, ElementType elementType) {


nit: a short javadoc would help understand what this does and what "stateless" means in this context.

alex-spies · 2024-10-07T12:59:48Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+
+        ElementType intermediateElementType();
+
+        GroupingAggregator.Factory valuesAggregatorForGroupingsInTimeSeries(int timeBucketChannel);


I'd love some javadoc to understand what this does.

alex-spies · 2024-10-07T13:03:01Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingKey.java

+import java.util.List;
+
+public record GroupingKey(AggregatorMode mode, Thing thing, BlockFactory blockFactory) implements EvalOperator.ExpressionEvaluator {
+    public interface Thing extends Releasable {


A smidgen of javadoc would also greatly help understand what the "Thing" is supposed to do.

alex-spies · 2024-10-07T14:10:23Z

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Categorize.java

+    record GroupingKeyFactory(Source source, ExpressionEvaluator.Factory field, AggregatorMode mode) implements GroupingKey.Factory {
+        @Override
+        public GroupingKey apply(DriverContext context, int resultOffset) {
+            ExpressionEvaluator field = this.field.get(context);


Bleh, never mind. I realized we require running the initial categorization as an Eval and the second one as part of the final Aggregate, like so

Coordinator: AggregateExec[[categorize(x){r}#57],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#57],FINAL,[categorize(x){r}#57, count{r }#62, seen{r}#63],58] \_ExchangeSourceExec[[categorize(x){r}#57, count{r}#62, seen{r}#63],true] Local: ExchangeSinkExec[[categorize(x){r}#72, count{r}#77, seen{r}#78],true] \_AggregateExec[[categorize(x){r}#72],[COUNT([2a][KEYWORD]) AS count(*), categorize(x){r}#72],INITIAL,[categorize(x){r}#72, count {r}#81, seen{r}#82],58] \_EvalExec[[CATEGORIZE(x{f}#75) AS categorize(x)]] \_FieldExtractExec[x{f}#75]<[]> \_EsQueryExec[test], indexMode[standard], query[][_doc{f}#83], limit[], sort[] estimatedRowSize[104]

alex-spies · 2024-10-07T14:43:45Z

...l/src/main/java/org/elasticsearch/xpack/esql/planner/AbstractPhysicalOperationProviders.java

+                List<GroupingKey.Factory> groupings = new ArrayList<>(groupSpecs.size());
+                for (GroupSpec group : groupSpecs) {
+                    groupings.add(group.toGroupingKey().get(aggregatorMode));


This doesn't do anything yet, does it? Currently, the groups contain only references to a CATEGORIZE expressions, but the reference attributes have no knowledge of what to do in the FINAL aggregator mode, no?

alex-spies

A few other things I noticed:

In follow ups, we'll need to test lots of edge cases with different queries to ensure interaction with other commands and with optimization rules is correct; as always with new language features :)
Currently, the implementation seems incomplete, right? At least queries like FROM test | STATS count(*) by categorize(x) and ROW x = \"asdf fdsa\" | STATS count(*) by categorize(x) result in a NPE.
In terms of the logical and physical plans, I think we need to model CATEGORIZE's execution a bit cleaner:
- The intermediate attributes in AggregateExec don't mention that CATEGORIZE outputs 2 columns, and that the final AggregateExec consumes both of those columns.
- The EVAL that does the initial categorization doesn't say that it outputs 2 columns.
- The physical plan on the coordinator node doesn't seem to actually represent that it does something special with the columns from CATEGORIZE, it looks like a regular aggregation.

nik9000 · 2025-01-06T13:12:18Z

This has been replaced by #114317. CATEGORIZE is mostly done now. This was a lovely prototype to learn more. but Nhat pointed us to a better way and we abandoned this code. Much nicer.

nik9000 added 3 commits September 11, 2024 11:26

Start

bc389e4

Next

712b672

Like so?

37415d2

nik9000 added >non-issue :Analytics/ES|QL AKA ESQL v8.16.0 labels Sep 11, 2024

nik9000 requested a review from dnhatn September 11, 2024 17:27

elasticsearchmachine added the v9.0.0 label Sep 11, 2024

nik9000 commented Sep 11, 2024

View reviewed changes

Fix one test

1fb09b1

jan-elastic mentioned this pull request Sep 17, 2024

ES|QL categorize v1 #112860

Merged

nik9000 added 2 commits September 17, 2024 14:58

Merge branch 'main' into stateful_grouping

5a088b6

foooooooooo

522c0bc

jan-elastic reviewed Sep 18, 2024

View reviewed changes

...in/ml/src/main/java/org/elasticsearch/xpack/ml/aggs/categorization/TokenListCategorizer.java Outdated Show resolved Hide resolved

jan-elastic and others added 6 commits September 18, 2024 16:41

Output correct regexes

96e6505

Remap intermediate category IDs

f6ef350

Fix mem leak

a385441

spotless

5a46823

Test categorize operator on multiple nodes

7685a0c

Merge pull request #10 from jan-elastic/categorize

f35fccb

Categorize

nik9000 commented Sep 19, 2024

View reviewed changes

dnhatn reviewed Sep 22, 2024

View reviewed changes

ivancea self-requested a review September 30, 2024 15:27

alex-spies self-requested a review September 30, 2024 15:28

costin self-requested a review September 30, 2024 15:28

iverase self-requested a review September 30, 2024 15:29

craigtaverner self-requested a review September 30, 2024 15:52

ivancea reviewed Oct 2, 2024

View reviewed changes

alex-spies reviewed Oct 7, 2024

View reviewed changes

brianseeders added v8.17.0 and removed v8.16.0 labels Oct 16, 2024

elasticsearchmachine added v8.18.0 and removed v8.17.0 labels Nov 20, 2024

nik9000 closed this Jan 6, 2025

	private final int resultOffset;
	private final int initialCategoryBlockOffset;


		Block evalIntermediateInput(BlockFactory blockFactory, Page page);

		void fetchIntermediateState(BlockFactory blockFactory, Block[] blocks, int positionCount);


		ElementType intermediateElementType();

		GroupingAggregator.Factory valuesAggregatorForGroupingsInTimeSeries(int timeBucketChannel);

ESQL: Compute engine support for stateful grouping functions #112757

ESQL: Compute engine support for stateful grouping functions #112757

Uh oh!

Conversation

nik9000 commented Sep 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Sep 17, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivancea left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Oct 4, 2024

Uh oh!

nik9000 commented Oct 4, 2024

Uh oh!

nik9000 commented Oct 4, 2024

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!