Checksum Verification

Strategy

The fastest way to verify that all the files from Scholarsphere 3 were correctly migrated is to compare the etag calculated by Amazon's S3 service with the original md5 checksum that was calculated by Fits when the file was added to Scholarsphere 3.

While the etag verification process can be used for other purposes, it will not the be the only checksum verification method in Scholarsphere. We will need to created separate checksums, such as sha256, and store those with the file's metadata for future reference.

Getting Fits Checksums

Find Deleted Files

Scholarsphere::Migration::Resource.where(model: 'FileSet').where('exception IS NULL').each do |resource|
  begin
    FileSet.find(resource.pid)
  rescue Ldp::Gone
    resource.update(exception: 'Ldp::Gone', error: nil)
  end
end

def find_file(id)
  FileSet.find(id)
rescue Ldp::Gone
  puts "Gone: #{id}"
end

missing_checksums = []
Scholarsphere::Migration::Resource.where(model: 'FileSet').each do |resource|
  fs = find_file(resource.pid)
  missing_checksums << resource.pid unless fs.try(:original_checksum).present?
end