All files for hundreds of rooms deleted during stack update - how is this possible?

[Aaron-E-Walsh]

Hello, I am writing in hopes of getting our Cloud instance (hosted on AWS) back up and running -- name, to recover all of the files for our more than 100 rooms that appear to have been deleted (or moved?) during a routine stack update.

The stack was updated to increase the amount of space allocated for user files (content they place into their rooms). We ran out of space once recently, and that caused all rooms to become unavailable. Increasing database storage and server size via a stack update solved that problem.

This evening it appears that we ran out of database space again, since the rooms became unavailable again. We ran a stack update, increasing database storage once again, and the update seemed to be successful according to AWS. But in reality it look as if the files folder has been deleted (or moved to a location where it can't be found by browsers).

I notice this when going to our main Clouds homepage, where we usually log in as Admin. The images on the homepage were "broken" (not showing) and when I looked at their URL they all reference this "files" folder location relative to our stack:

https://ied-spaces-stack-v1-assets.our-domain.org/**files**/cfcd7a12-4aa9-454d-8b5d-9430c32dc77d.png

I tried to locate the files folder in S3, but there are only folders for "assets", "hubs", "spoke" -- but no "files" folder. I have attached a screenshot.

Is is possible that the stack update somehow deleted the files folder, or am I looking in the wrong place (and if so, how do I fix the problem?).

Our rooms are still offline, and our users are upset, so I'm in a bind.

What might cause the images on our Cloud homepage (where we log in as administrator) to be broken and not able to be displayed (as they are not where they browser expects them to be, or are missing)? And why would the rooms not be available now that we've increased store (by 3x as much as it was) and gone up another server size (about 2x what we actually need).

Do you think we need to restore from a backup? We only have the automated backups that Clouds does (in AWS) to backup from, but that should be adequate although we have never restored from a backup before and it seems like a radical step to take if not necessary. And, quite frankly, my concern is that making a mistake when restoring from backup could make the situation much worse.

Any help or advice you can give would be greatly appreciated. We have a lot of upset users right now, the pressure is on, and it would be a tremendous relief to restore the 100+ rooms that they have created and use daily.

Thank you!
Aaron

[biscuitsanddavey]

I can’t speak to our situation as well as Aaron articulated their situation, but all our hubs rooms across three separate AWS accounts and instances are experiencing a similar issue. No scenes will load and no content will appear on the admin account homepage or backend (avatar, scene previews) and no content appears in Spoke. The projects are there, not none of the content. We have a lot of upset users right now and need to repair this ASAP.

[rawnsley]

There was an update last night and you might want to check this discussion on Discord.

[NigelRidderhof]

Hi, I was actually typing a message explaining I have similar problems, but I started reading through the Hubs discord and there's been an update causing many people's Hubs Cloud te be down since ealier today. I recommend going to their discord. You'll need to do a stack update. But the fact that Aaron already has done one and is still running into problem doesn't make sense to me (I'm just a user like you probably). Hope this helps a bit thiough!

[biscuitsanddavey]

Seeing the updates in Discord now. Will test it on a test instance this morning. Thanks! 🙏

[pattersonbl2]

A stack updated shouldn't delete your files. The problem is the stack you updated is out of date by 4 months. We added some changes into version 1.1.2 that references assets files in a different way. We haven't been able to update the version number when we make changes due to issues with AWS requirements. This can be done via https://hubs.mozilla.com/docs/hubs-cloud-aws-updating-the-stack.html. You would do a manual update to 1.1.2

I would wait to due to this because we are waiting for another update to this template to be published unless you have a personal stack, which has been updated. This has been the communication in the discord channel.

[Aaron-E-Walsh]

Thank you, Brian (and all) for your responses. This has been something of a nail-biting experience, but we will wait and update the stack as you suggest. Is there a notification service (email or text) that we can subscribe to in order to get notices when important changes/updates (like the recent stack template update) are coming that could disrupt service? With thanks again, Aaron

…

On Tue, Feb 8, 2022, 3:28 PM Brandon Patterson ***@***.***> wrote: A stack updated shouldn't delete your files. The problem is the stack you updated is out of date by 4 months. We added some changes into version 1.1.2 that references assets files in a different way. We haven't been able to update the version number when we make changes due to issues with AWS requirements. This can be done via https://hubs.mozilla.com/docs/hubs-cloud-aws-updating-the-stack.html. You would do a manual update to 1.1.2 I would wait to due to this because we are waiting for another update to this template to be published unless you have a personal stack, which has been updated. This has been the communication in the discord channel. — Reply to this email directly, view it on GitHub <#5091 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPJ77WPRS5Q7Y5FOBFDCQLU2F4G7ANCNFSM5NZLTWCQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[pattersonbl2]

It was actually published 10 mins after I sent to message. so you just need to launch a stack using the 1.1.2 version.

[Aaron-E-Walsh]

Thanks again, Brandon (I mistakingly wrote "Brian" instead of Brandon in my last, overly hasty, reply -- sorry about that!).

Is there a way to tell from the admin page what version of the stack we're using (I can see "App client: 1.0.0.20220115001851" there, but I'm not sure if that's different from the stack version itself?). If not, perhaps in AWS or by looking at some files in the AWS S3 would let me know the stack version?

Aaron

[GBR-422777]

@Aaron-E-Walsh can you kindly advise in dummy steps how you fixed this? Appreciated!

[GBR-422777]

@pattersonbl2 any clue?