unable to recover meeting room assets

[slalomsunil]

I need your help to recover the assets created for our meeting room.

We have set up the mozilla hubs environment and we had our assets stored in it.
One day, the environment suddenly starting throwing 503 error

Here are the steps I tried to bring it back online.

1. I checked for cloudtrail logs for errors, but I could not get any lead.
2. I referred to the discussions and documentation and tried to terminate the instances. The instances came back online but the error still remains.
3. I verified the domain registration status and that looks good. I checked the instance and load balancer health, It looks good.
4. When I was not able to get the existing environment back up and running, I decided to make use of the backups from this environment and stand up a new environment. I referred the documentation and created a new environment with new domains and provided all the information for the database and the backup storage.
5. The backup environment came up but I was not receiving the magic link to login. I made sure that my email ID is verified in SES and that it is in N.Virginia. I even raised a request to AWS and got the sandbox mode changed to production mode, though this may not have been necessary.
6. I searched the discussions for this issue but I could not find much answers, I have raised this issue in the discord and waiting for a solution.

Please advice

1. How do I recover my original environment from 503 error and get all the assets back?
2. How do I receive the magic link in the back up environment to be able to login and get the assets from the back up resource.
3. Where can I check for the application logs when there is an issue with the hubs platform

[pattersonbl2]

are you able to SSH into the EC2 ?
this would allow you to run the journalctl command to inspect logs related to the reticulum, which could source of the issues.

[slalomsunil]

checked the logs and this is the error we see.

37.530651] cloud-init[1268]: [0mWaiting for pgBouncer.
[ 540.541241] cloud-init[1268]: Waking up the database.
[ 540.922933] cloud-init[1268]: psql: ERROR: no such database: polycosm_production
[ 540.926864] cloud-init[1268]: Starting stream services.
[ 540.943053] cloud-init[1268]: [0m [0mSupervisor starting mozillareality/janus-gateway. See the Supervisor output for more details.
[ 540.950547] cloud-init[1268]: [0mWaiting for Janus.

[pattersonbl2]

Sorry for the delay. Based on these logs , I would check your database connection. For your assets, they are normally stored in S3 buckets and the EBS volume.