fix: Republish CloudFront Origin Switch

[Adam-C-Mitchell]

There is a race condition between the CloudFront blue/green origin switchover (#536) and the Fastly cache purge. Specifically, we make an UpdateDistribution API call to CloudFront to make this origin switchover. This is asynchronous — the operation takes ~5 minutes or less (source), but is never instant. Empirically, it takes ~30-40 seconds. We should bear in mind that it doesn't seem AWS makes any kind of guarantee as to how long this switchover should take (at least as far as I could find). Thoughts on this are welcome.

The problem with what we have currently is that immediately after calling UpdateDistribution, we call Fastly to invalidate its cache. As such, Fastly (usually) beats CloudFront to the punch, and winds up revalidating its cache with the contents of the old (blue) bucket, just as we're about to switch over to the new (green) bucket. You then end up with a discrepancy between what's served from Fastly and what's served at the CloudFront origin: the former remains effectively one state behind reality, if you will.

It's likely that this issue is worse in Production, because it actively gets hit with more traffic specifically after the site has updated with new content (alerts). As such, it's more likely that a revalidation gets triggered during that window between the Fastly cache purge, and the origin switchover. That would certainly explain why it's been more heisenbuggy in lower environments.

---

The approach taken to solving this issue is to update the current bucket SSM parameter, have a Lambda@Edge origin-request function use it to determine which origin should be used, and every 2 seconds following that, have the GovUK application poll the website for a _content-manifest file that proves the CloudFront origin has been switched over. Once that's confirmed, we issue the API call to purge the Fastly cache, as we are confident that CloudFront now consistently reflects the desired state to be displayed publicly.

localenv:feature/eas-3228-origin-selection

[jamesorlakin]

Does (now) line 134 already cover this? E.g. I'm assuming an alert naturally should always have an updated_at but even so it'll be covered by line 120.