Skip to content

Limit number of files that can be submitted, other bugfixes#73

Merged
patrick-austin merged 2 commits into36_queuingfrom
36_limit_queue_files
Apr 1, 2025
Merged

Limit number of files that can be submitted, other bugfixes#73
patrick-austin merged 2 commits into36_queuingfrom
36_limit_queue_files

Conversation

@patrick-austin
Copy link

  • Limit number of files that can be submitted and remove the chunking into parts for /queue/files
  • Calculate a running total for the fileSize of queued Downloads and set this directly instead of using the IDS
  • 400 and 404 errors for submitting bad visitIds/files or those that do not exist/do not pass authz
  • Only call performCheck when there's a prepardId to use
  • Properly extract the functional sessionId used to prepare queued Downloads
  • Tests
  • Add options to the admin script for queue related things

response.missing.add(file);
}
}
JsonArray jsonArray = submitQuery(queryPrefix + chunkedFiles + querySuffix);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 7 lines are a copy of those just above, so should go in a separate method to avoid duplication. There was some duplication previously but it is now worse with the use of the new response object and the additional values being populated.

* @throws TopcatException
*/
public long getDatasetFileSize(long datasetId) throws TopcatException {
String query = "SELECT datafile.fileSize FROM Datafile datafile WHERE datafile.dataset.id = " + datasetId;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why you didn't use a SELECT SUM query here? I think the ICAT API supports them.

/**
* Queue download of Datafiles by location, splitting into part Downloads if
* needed.
* Queue download of Datafiles by location, up to a configurable limit.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the return value in the javadoc needs updating. If I understand correctly it is no longer an array of download IDs but a single download ID and an array containing any missing files.

@patrick-austin
Copy link
Author

Also includes fix for a NullPointer when anonUserName is undefined, with a warn when starting the API if anonUserName is undefined to say that this will mean we cannot distinguish anon and authenticated users, so everyone will use authenticated priority (which is at least as high a priority as default).

@patrick-austin patrick-austin merged commit 7ce214f into 36_queuing Apr 1, 2025
1 check failed
@patrick-austin patrick-austin deleted the 36_limit_queue_files branch April 1, 2025 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants