Feature to limit file uploads to datasets

[stevenwinship]

What this PR does / why we need it: Need a way to prevent thousands of file from being uploaded due to the issues that causes.

Which issue(s) this PR closes: #11275

• Closes Feature Request: (internal request) Add quota-like limit on the number of files in a dataset #11275

Special notes for your reviewer:

Suggestions on how to test this: See IT tests

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?: Included

Additional documentation: native-api and config

[landreev]

Just a quick status update:
I did get to look into, build and play with the PR today. Will be adding comments/asking questions tomorrow.

[coveralls]

coverage: 23.141%. remained the same
when pulling 9cfa6be on 11275-add-limit-to-number-of-dataset-files
into 2fb4a0d on develop.

[landreev]

Very happy to see this implemented.
I may have found a small bug: if the limit is set to 20, and I have a dataset with 19 files, it appears that I can upload 2 more files. I cannot add any files to a dataset that's already at 20. (but please re-test/confirm)

[landreev]

It is super important to have this config setting implemented for both collections AND datasets. In real life, I'm guessing it's going to be a somewhat common case where a specific dataset will need to be given a higher limit, because of some respectable reason. It appears to be working consistently when defined on either level.

However - and this may be penny-pinching, admittedly - I'm wondering if we want this column to be in the DvObject table; seeing how most DvObjects are files. Please at least consider making it a DvObjectContainer-only element. (see dvObjectContainer.storageDriver for an example; it ends up being an extra column in the Dataverse and Dataset tables each).

[stevenwinship]

I'm confused about the penny-pinching comment. Columns in the database that are null take up no space and therefore add no pennies to all the DvObject Datafiles.

[qqmyers]

It does have to get loaded into the objects though. Further - it's just odd that a file has a datasetFileCountLimit, which wouldn't be the case if it's on DvObjectContainer.

[stevenwinship]

Ok. I'll move it

[landreev]

Yeah, phrasing it in terms of objects makes more sense. I just wanted to emphasize that it was possible, even though there is no dedicated table in the db for DvObjectContainer.

[stevenwinship]

I moved it

[stevenwinship]

Taking a look at this. Thanks for finding this

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:11275-add-limit-to-number-of-dataset-files

ghcr.io/gdcc/configbaker:11275-add-limit-to-number-of-dataset-files

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[ofahimIQSS]

great fix, merging!

[qqmyers]

Does this make sense:

dataverse/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

Lines 1995 to 1996 in 1c31486

	if (authenticatedUser.isSuperuser() || permissionService.hasPermissionsFor(authenticatedUser, dataset,
	EnumSet.of(Permission.EditDataset))) {

- if you can edit the dataset, you can change the limit yourself?