MetadataStore updateObject API fails occasionally with a conditional check error. #6
Description
The core of the updateObject API implementation is here.
In some rare cases (mostly observed when running a very high number of metadata only operations against the file system: e.g. restoring an HBase snapshot into a directory) this method will fail with a ConditionalCheck.
What seems to be happening is that an I/O error occurs, on the server side in DynamoDB the operation completes successfully and the client retries the operation (since from its perspective it failed with an I/O error). At this point the operation fails because it is issued with a ConditionalCheck which is not idempotent.
The proposed solution is to handle this particular case: when we receive a ConditionalCheck failed error we should read the state of the item from DynamoDB and check it against the item state that was received as an update. If the two are equal that means the operation was in fact executed successfully in DynamoDB and we should not raise the error up the call stack.