Drop files that no longer exist #31
Description
The current paradigm of crawling metadata leaves stuff in the db that is removed on Myrient's end, which causes invalid entries in the search. The crawl should account for these and remove any rows from the db that go unmatched in a crawl.