Migrate to new infrastructure

[sandbergja]

• Connect the staging environment to CAS IdP (emailed 10/6)
• Do a test migration to staging server
• Do a real migration to production server
• Write backup/restore storage script -- maybe something that can run as a rake task on a worker pod? (for restore, write a custom bulkrax parser based on CSV parser that can pull from Azure instead of browse_everything; for backup, use a POST query to the json API maybe, or just use it programmatically?)
• cron up a rake blacklight:delete_old_searches[days_old] (contribute to upstream) #72
• Set up the staging deployment in github actions to wipe all data then rebuild using the restore script
• Set up kubernetes to regularly run the backup script (through a cron job)?
• Set up github actions to push to production when there is a new semver release
• Get end-users to test the new version on the staging server
• Switch DNS
• Decommission old server
• deploy github action isn't quite working -- maybe to do with the redis password --set arg?
• Do I really need 3 solrs running? It is getting us close to our limit of filesystem mounts -- configuration options here: https://github.com/bitnami/charts/tree/master/bitnami/solr/
• Figure out billing
• Rewrite Dockerfile to be based on the samveralabs/hyrax dockerfile
• Figure out a good repository for the resulting container images, automatic pushes to the repo, versioning (probably github container registry)
• Change Values.service.type to LoadBalancer
• Should be on port 80/443, not 3000
• Make sure that secrets are being taken care of properly
• Create a deployment script that takes some args or envvars for which helm release (staging or prod, injecting different values.yml files) and which container version (aka commit hash)
• Update documentation, including expected response of basic kubectl commands
• Make sure the worker image runs under the app user, not root
• Upgrade to helm chart 1.0 https://github.com/orgs/samvera/packages/container/hyrax%2Fhyrax-helm/8093512?tag=1.0.0
• Make sure that the solr pods are getting the appropriate configuration from the repo (check the load-solr-config step in the hyrax helm chart)
• Change the dockerfiles to be FROM ghcr.io/samvera/hyrax:version
• Add bulkrax
• Production deployment should have more hyrax-worker pods, other deployments should have fewer
• Check .env file to make sure I'm not forgetting anything (like SMTP?)
• Make sure the new environment accomplishes all of these: https://github.com/samvera/hyrax/wiki/Hyrax-Management-Guide