Add support for minimum_should_match to aggregations

[renatonascalves]

Description

We need a method to be able to include all items in an aggregation, even if no results match those items, as an opt-in feature.

For example, we should have a way of being able to run elasticsearch_extensions()->get_aggregation_by_query_var( 'taxonomy_custom' )->checkboxes() and have checkboxes show for all items in the aggregation, even if none of the results match (e.g., show all categories, even if no posts are in a particular category in the results).

Use Case

A site needs to show an aggregation but listing all the aggregation items available.

Acceptance Criteria

• Allow an aggregation to be registered with an optional property of the minimum number of items to match in order for the aggregation item to show. This could be set to 0 to show items that don't have any matches (e.g., a category that doesn't have any matches in the results) or a higher number to only show items that have many matches (e.g., 10 to exclude the long tail of matches).

Comments

This will likely need to be done as a combination of Elasticsearch queries and WordPress database queries, because Elasticsearch won't know about items that are empty (e.g., taxonomy terms that aren't assigned to anything).

[kevinfodness]

@renatonascalves can you give an example of where this feature would be useful? It may require some work on the WP end, like pulling the full list of taxonomy terms, since we're currently relying on what ES is returning for the aggregations, which excludes empty items.

[renatonascalves]

I think the use case was that if you make a search, the results will return only the terms/aggregation that's part of the search results/response instead of bringing all, unfiltered, terms.

If you have a search form UI where you need to show the unfiltered terms in a empty state, the current approach forces one to query the database for those terms whereas ES could maybe return them too?

which excludes empty items.

I think the current behavior goes beyond empty items. Some non-empty items could also be removed if they don't match the main DSL query.

[kevinfodness]

Yeah, aggregations are not going to show anything with zero matches among the returned results.

Sounds like this is possible by setting the minimum document count to 0 with the following caveat:

Setting min_doc_count=0 will also return buckets for terms that didn’t match any hit. However, some of the returned terms which have a document count of zero might only belong to deleted documents or documents from other types, so there is no warranty that a match_all query would find a positive document count for those terms.

ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

[kevinfodness]

imo this is better set as an option when adding support for an aggregation rather than in the select and checkboxes methods. For example:

$es_config->enable_taxonomy_aggregation( 'category', [ 'min_doc_count' => 0 ] );

We would need to add support for the min_doc_count option for taxonomy aggregations and any other aggregations that could conceivably support it. This would be a benefit in other ways too, because you could also use that option to have a higher minimum, which would be useful in hiding a long tail of terms.

[renatonascalves]

@kevinfodness I did some more reading, and some testing, and that seems like a good approach.

See 1, 2, and 3.

The main search is not impacted and the aggregation will return all terms, including from deleted docs, which in this case, it's not a problem.

       ...
        {
          "key" : "Product & Development",
          "doc_count" : 1
        },
        {
          "key" : "Design",
          "doc_count" : 0
        },
        {
          "key" : "Drivers",
          "doc_count" : 0
        },
        {
          "key" : "Engineering",
          "doc_count" : 0
        },
        {
          "key" : "Product and User Experience",
          "doc_count" : 0
        }
      ]
    }
  }
}