Static Findaid Architectural work

[amywieliczka]

• Move the s3 cache interactions out of the controller and into an S3Cache module.
• Also relocate logic to try the cache or render dynamically into a StaticFindingAidService
• Cache just the HTML fragment of the main body, rather than the full HTML page.
• Wrap the dynamic rendering in a timeout, and use Rails' job framework to queue a job if timeout is reached (return a 503 page saying we're generating this content, try to refresh again later)
• Create a StaticFindingAidRenderJob - this is the job that is queued
• Make sure we prevent multiple jobs getting queued in a row from firing the same solr query several times in a row (due to, for example, a bot's retry on the static finding aid url), by wrapping the job in a Mutex - this ensures all jobs of this type occur in sequence - and by checking the cache at the start of each job (in case, for example, a previous job on the queue created a up-to-date static finding aid). This doesn't prevent different containers from triggering the same query, however, most of the time this will be inexpensive and fast, and also done at the end of the indexing step...
• Add a request for this URL at the end of the index_finding_aid dag - if it returns a 503 (generating...) or a 200, succeed, otherwise, fail the task.

There is some more UI changes Adrian and Christine have requested, but I'm wanting to encapsulate these architectural changes separately.