Use leases instead of indefinite locks in migrator control document #83
Description
Right now we use a lock field to determine if we have exclusive access to migration. In normal operations this is fine because the lock will be taken, migrations happen, then released. However what does not get covered are situations where an error occurs. In example if the migrator fails to migrate down a version (say because it doesn't exist since it is a roll back in code) then the lock will be taken, the migrator will crash, and then the lock will never be released. Another scenario can also occur where the DB connection is broken and not re-established and thus the lock can not be released without first reconnecting (if possible).
Point is that a lock as a boolean type is a bad idea. A better solution would be to use a "lease" mechanism. It is similar to a lock except that instead of being a boolean it is instead a date in the future at which point the lease expires. This way if a migration fails then all future migrations will not be hamstrung until the document is fixed manually. We should be loud if we have to remove an expired lease but this is clearly an improvement over the current implementation.