[DS-4312] Add skip list to transmitaip task

[dspace-bot]

Imported from JIRA [DS-4312] created by mspalti

Transmitting AIPs to DuraCloud that are larger than 5GB results in a 504 error due to limitations imposed by Amazon. This causes the backup to halt. There's a discussion of changing this behavior (see DS-4309) by allowing curation tasks to continue and reporting any failures. Even with that, I think there's a case here for adding a skip list for ignoring AIPs that are known to be too large (it would be even better to introduce chunking so that RTS could transmit these larger objects!).

[dspace-bot]

mspalti said:

I'll submit a PR for this proposal, implementing Mark H. Wood's suggestion of returning CURATE_SKIP and logging a message. (That PR will include code from an earlier bug fix PR https://github.com/DSpace/dspace-replicate/pull/27)

[dspace-bot]

mspalti said:

Errors seem to happen when backing up large collections to DuraCloud. (I just saw a 502 error for a small file). Right now, ContentStoreExceptions are re-thrown as an IOException and processing is halted.  Is there any reason not to simply log ContentStoreException messages and continue with processing? Any items that are not transmitted as part of the batch job could then be sent individually after the site backup completes.  

[dspace-bot]

tdonohue said:

Michael Spalti: Thanks for taking this on (as you've likely seen, the dspace-replicate project has not received many updates as of late...so I'm glad you are able to find time to chip in).  In my personal opinion, I see no reason why dspace-replicate should exit (halt processing) on a single error (no matter which error occurs).  So, I like your & Mark's suggestion of instead logging the error and/or finding some other way of reporting the error known to the user without halting the entire process.