OAI-PMH errors block our hourly updates

[fsteeg]

Occasionally, the DNB's OAI-PMH interface returns an error (apparently caused by a transformation error on a specific resource). Our update process then aborts with java.io.IOException: Server returned HTTP response code: 502 for URL: <URL with resumptionToken>. The updates are then re-tried every hour, using the same starting time, until either:

• the error is fixed in the DNB's OAI-PMH interface, or
• we manually trigger a data update with a starting time after the problematic update

Ideally, the problematic record should just be skipped, instead of aborting the updates and getting stuck at that point in time.

[Phu2]

Idea: in case of an error split up the interval of the original request and issue two subsequent requests. Original request: from-until. subsequent requests: from-mid and mid-until. Do this recursively until the interval is less than 2 seconds. This remaining interval includes invalid records which lead to the error response.