feat: datasetCount for dataverses

[vera]

What this PR does / why we need it:

This PR adds a datasetCount for each collection to the search index. The count includes all datasets (published, linked or harvested).

This allows users to filter collections using the datasetCount (e.g., datasetCount:[1000 TO *]). Also, the value is returned in Dataverse search results via the Search API.

As recommended by @pdurbin I've used the fileCount feature PRs as inspiration (#6623 + #10598).

Which issue(s) this PR closes:

• Closes Feature Request/Idea: Filter dataverses by number of datasets #10190

Special notes for your reviewer:

Regarding the points commented by @cmbz in the issue (#10190 (comment)):

• Some subtlety here: what gets counted? (e.g., linked datasets, harvested datasets)

For simplicity, I've decided to count all datasets that would be shown when visiting the collection in the UI. This means that the count includes published, linked and harvested datasets in the collection or in any subcollections.

If there is interest in further subdividing the dataset count, more subcounts could be added in the future (e.g. publishedDatasetCount, linkedDatasetCount, harvestedDatasetCount).

• Considerations: Does this add overhead to reindexing process?

I've had to add new dataverse indexing calls in the following locations:

• After a dataset is published, the owning dataverse and any linking dataverses are reindexed. (added to this PR after feat: allow linking unpublished datasets to collections #11491)
• After dataset destruction, previously only the owning dataverse was reindexed. Now, any linking dataverse is also reindexed.
• After linking a dataset to a dataverse, the dataverse is reindexed.
• After a harvesting of datasets is complete, the target dataverse is reindexed.
• After a harvest client is deleted, the target dataverse is reindexed.
• In general, if the reindexed dataverse is not the root dataverse, all of its parents up until the root are also reindexed.

I believe this overhead is not too large, but let me know what you think.

Suggestions on how to test this:

mvn test -Dtest="SearchIT#testDataverseDatasetCounts"

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

/

Is there a release notes update needed for this change?:

I wrote a short release note.

Additional documentation:

/

[coveralls]

coverage: 23.209% (-0.02%) from 23.229%
when pulling 88cff95 on vera:feat/dataset-count
into fdcdba2 on IQSS:develop.

[qqmyers]

It might be worth testing this versus a custom query at scale (or, alternatively, to just test this versus the current indexing on a large database). I think this should just result in a proxy list where calling size() won't try to instantiate a real list of dataset objects, but I'm not sure. If it does do extra work compared to just a select count(*) query for the right datasets, it could slow or increase memory requirements significantly.

[vera]

I have been testing this with our test system which includes a larger dataverse (around 30k datasets) and I think that you are correct and that I am seeing some significant slowing of API responses which require reindexing of that large dataverse. I will investigate this further and update this PR as soon as I am able.

[vera]

I've just pushed a commit using a NamedQuery to calculate the datasetCount. With our 30k dataset dataverse, reindexing the dataverse is now speedy again.

[ofahimIQSS]

to be discussed during next weeks triage.

[stevenwinship]

Please resolve the conflicts.

[vera]

@stevenwinship the conflicts are now resolved.

[ofahimIQSS]

approved version bump, tests are passing - merging.