HARMONY-2016: Fix race condition in work reaper.

[ygliuvt]

Jira Issue ID

HARMONY-2016

Description

I think the issue is caused by the race condition between old work items cleanup and old workflow steps cleanup. Currently, work reaper uses the current time minus the reapableWorkAgeMinutes configuration value to find and cleanup old work items and old workflow steps, but because it takes time to run old work items cleanup, by the time it gets to workflow steps cleanup, there might be more work items that are eligible for cleanup based on the current time, thus causes the error documented in this ticket.

The fix is to pre-calculate the cutoff time before executing the work items cleanup and the workflow steps cleanup to avoid the race condition.

Local Test Steps

npm run test

PR Acceptance Checklist

• Acceptance criteria met
• Tests added/updated (if needed) and passing
• Documentation updated (if needed)
• Harmony in a Box tested (if changes made to microservices or new dependencies added)

[indiejames]

Tested locally, verified that work-items and steps are deleted

[chris-durbin]

Good catch.