Try the the collection counts have expired step before interacting with the collections sidebar/the fandom counts instead of disabling caching like this

The test still fails. It doesn't on my end which makes troubleshooting difficult. The fact it used to work and broke on an unrelated change confuses me.

This does not include works in the subcollections, but the counts on the collection page do include them, so this is a mismatch - I can click on "new fandom (1)" and get to a page with no works. For canonical tags the works from the subcollections are included, so we should do the same here

I see, is there a quick way of having a list of collection from a parent collection? (since I assume child of child are also included)

Fortunately, child collections can't have children -- subcollections only go one level deep.

Could you also check TV (2) here, so we're sure that the syn is counted under the fandom?

Is this still required or can it be replaced with a well placed the collection counts have expired?

This has to load all works in the collection and all fandoms of those works. This is not great, especially in the context in which this was implemented to be done via elasticsearch: https://otwarchive.atlassian.net/browse/AO3-6116. Since I'm writing this during maintenance and you can't look it up right now: Yuletide has 42 806 works. So we'd be loading all that at once, and then also all the fandom tags of those works.

During the migration of collections to elatiscsearch we tried to optimise a similar query and despite that it was still quite slow: #5234 (comment). This here isn't as bad because it doesn't include bookmarks or the crossover check, but we will also run this code here regularly, so we have higher performance requirements than a one-time task.

Well, the above isn't quite correct: As is, query.search_results will return one page worth of works. If we were to fix that (so that we don't get wrong information), then we'd load all of it.

So unfortunately I think we can't go with this approach. If we are going to get this information, we should get it directly from elasticsearch, maybe with some computation in Ruby, not with additional large database queries.

I believe this issue was marked as blocked by collections in elasticsearch because the original plan there was to include fandom information on the collection, so we could have gotten information via that. However, that plan was changed because it was looking too performance intensive. So now we have to solve this issue here differently.

With the existing information we have on works in elasticsearch, the uncategorized fandoms are included in filter_ids, mixed with a bunch of other tag types. So if we were to use them here, we'd still have to get the tags from the db to get their type. That's slightly better because we don't have to get the works, but still not great.

This may work on the fandoms page, since we already have to get the fandoms there to display them, so if we're clever about it we wouldn't introduce another query. But for the count in the sidebar, that doesn't apply.

In the big picture, we should put the data into elasticsearch in a format we can use well (so doing the computation when indexing), not do lots of computation afterwards to use it. So I'm leaning towards putting this information into elasticsearch over trying to use filter_ids. But that means reindexing etc. So I don't know where to go from here. Do you have an idea?

While imperfect, an idea could be to combine both methods? That is, if a collection has a lot of work, we could go with the previous approach, but if it has less then we could have a more exact count using the tags?

Another would be to put the uncategorized fandom_ids in the fandom_id field of the query when they're added so we can go with the previous method

Could we add a property to query so that we can fetch only items with an uncategorized tag of some sort? That way we could do two queries (not ideal) then we do one query with fandom_ids for all, then one query that's more involved with the filter_ids when a work has an uncategorized tag then fetch the ones that are uncategorized fandoms.

if a collection has a lot of work, we could go with the previous approach, but if it has less then we could have a more exact count using the tags?

That's not viable because yuletide is one of the collections where it's especially important that uncategorized fandoms are shown, since it has many of those each year.

Another would be to put the uncategorized fandom_ids in the fandom_id field of the query

We can't modify what is in the field like that, it's used by more code than the counts here, like the facets (the filter sidebar) that expect only canonical tags

Could we add a property to query ...

Something like that is what I mean with "putting the right information into elasticsearch", yes. If we're adding a new property we should add it in a way that is directly useful here (e.g. all fandom tag ids or something), not something that needs another workaround in the queries. But that has the downsides I noted - we'd need to reindex completely, since the mapping changes, plus put a bunch of brainpower into designing that property. Which brings to the end of my last comment: I don't know what we should be doing here

I see some possibilities for that new propriety:

We could add a new property that is all fandom tag ids. This would have the downside of having to parse through the tags to check canonicality, which might be not as useful. Some fandoms have a lot of synonyms so I'm not sure if that's the correct way forward. Plus, this feels like something which would have limited usefulness.

I'm unsure about how true that would be, but could having multiple uncategorized properties be useful for tag wrangling? For instance, this could allow someone to get all of the uncategorized tags or fandoms from all work with a specific fandom. I'm unsure if we can have one property that's like a list of list. So like freeform tags, relationship tags, character tags, which I guess could be more generically a list of list with types as keys. This is what feels the more generic and possibly more justifying a huge reindex.

I'm not full certain what is allowed as return type of elastic search, but this feels like a potential idea?

Finally, I apologize for not having understood what you meant.

Could you also check the counts here