Spoke projects not accessible anymore, assets returning 403 Error

[pincy]

Hi,

on our AWS hosted instance, we cannot access previously created projects anymore.
I cannot tell since when, because the instance was pretty much on idle for some months.

The stack update is already applied, but did not resolve the issue.
Trying to access the project files in the admin, returns 403 Forbidden on download.
What i checked so far, the database entries are all present.

I would be glad on instructions, how i can further debug this or directly access our data to import the files as new projects.

Best regards

[pattersonbl2]

By chance, could you include a screenshot of the error? I would like to see the Developer console log.

[pincy]

@pattersonbl2 The dev console is not too informational on the download error.
Trying to download the spoke project e.g. there is even no entry added to the dev console and network tab, i assume this is for the link having target="_blank".

Our general setup is with subdomains, all working properly over the summer last year.
The stack was created from a backup
Main uri: meet.DOMAIN.TLD
Internal uri: int.DOMAIN.TLD
The configuration of the instance wasn't touched for months.

I get the following for each not accessible scene thumbnail in the admin in the dev console:

c790c765-1bba-4a54-be82-d94a155dc926:1          GET https://meet.DOMAIN.TLD/files/c790c765-1bba-4a54-be82-d94a155dc926 403

But on the network tab i get the following information from a single request:

General

Request URL: https://meet.DOMAIN.TLD/files/95d2d067-b0ad-435d-ab6b-594956be8934
Request Method: GET
Status Code: 403 
Remote Address: [MY_IPV6]:443
Referrer Policy: strict-origin-when-cross-origin

Response Headers

cache-control: max-age=0, private, must-revalidate
content-length: 0
content-security-policy: default-src 'none'; img-src 'self' https://ASSET_BUCKET.int.DOMAIN.TLD; style-src 'self' 'unsafe-inline'
cross-origin-window-policy: deny
date: Thu, 14 Apr 2022 07:56:17 GMT
server: Cowboy
strict-transport-security: max-age=31536000
vary: Origin
via: 1.1 b1dc6a0d7547e8d4ab339f8c4caf9ea8.cloudfront.net (CloudFront)
x-amz-cf-id: _YoMGxDhXX7hrIVhAch_8SyDks0Su3wIYH3KZNh6ASHfsMjIpv4Hlw==
x-amz-cf-pop: DUS51-P2
x-cache: Error from cloudfront
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-request-id: FuW0NQqUFfQ-khUAAMyh
x-xss-protection: 1; mode=block

Request Headers

:authority: meet.DOMAIN.TLD
:method: GET
:path: /files/95d2d067-b0ad-435d-ab6b-594956be8934
:scheme: https
accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
accept-encoding: gzip, deflate, br
accept-language: en-GB,en;q=0.9,de-DE;q=0.8,de;q=0.7,en-US;q=0.6
dnt: 1
referer: https://meet.DOMAIN.TLD/admin
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"
sec-fetch-dest: image
sec-fetch-mode: no-cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36

Finally, i checked with my scenes in spoke and saw CORS erros, not present half a year ago.
Spoke error on old scene

Errors loading project:

2 Errors:
  Error loading video YOUTUBE_WATCH_URL:
    Cause:
      Error resolving media "YOUTUBE_WATCH_URL":
        Cause:
          Failed to fetch "https://meet.DOMAIN.TLD/api/v1/media":
            Cause:
              Network Error: 500 Unknown Error. Possibly a CORS error.
  Error loading model "https://uploads-prod.reticulum.io/files/728260ab-4e8f-4052-8e1a-c7fae3492989.glb":
    Cause:
      Error resolving media "https://uploads-prod.reticulum.io/files/728260ab-4e8f-4052-8e1a-c7fae3492989.glb":
        Cause:
          Failed to fetch "https://meet.DOMAIN.TLD/api/v1/media":
            Cause:
              Network Error: 500 Unknown Error. Possibly a CORS error.

The YOUTUBE_WATCH_URL is in the ?watch= pattern and is valid to visit manually.

[pincy]

I just tried it with exporting the cudillero-diorama on hubs as a legacy spoke project and import it on our instance, this raised the following error:

Errors loading project:

1 Error:
  Error loading model "https://sketchfab.com/models/e051a32d21d6453c8fad408082492267":
    Cause:
      Error resolving media "https://sketchfab.com/models/e051a32d21d6453c8fad408082492267":
        Cause:
          Failed to fetch "https://NODE-cors-proxy.int.DOMAIN.TLD/https://sketchfab.com/models/e051a32d21d6453c8fad408082492267":
            Cause:
              Network Error: 502 Unknown Error. Possibly a CORS error.

Also i just wrote you a mail with the domain.

[pattersonbl2]

The only thing I am seeing is an issue with the content security policy. It looks like your subdomain isn't being recognized, which is stopping your request. I would take another look at these settings and also do more troubleshooting using the Developer console. This is about as far as I can help with this issue.

[pincy]

I tried to find time, digging deeper into your suspicion on csp.
But with the non existing documentation, what should direct where and how, checking whats wrong seems impossible.
Did something on the domain setup change with the update on the CloudFormation Template?
My suspicion lies here on some update in between, since the setup is with subdomains, which was not really documented, but worked in the past.

Does the template check the urls and configurations on stack update (turning the stack on/off)?

I'm really getting to the point, where i just want to backup all the data, clear the stack and all project resources and start with a new stack.
How can i decrypt and backup the data?
I can't tell my coworker he needs to rebuild all the scenes for the next weeks/months.

Right now, this looks like a vendor lock-in in a broken state of the application, to be honest. :(

[pattersonbl2]

Nothing has changed on the domain set up with the cloud formation template.
No, The template update doesn't change the URL and configuration. the template builds the infrastructure. Taking it offline only means stopping the ec2 and pausing the database. No changes are needed to do that.

This could help with understanding backups https://hubs.mozilla.com/docs/hubs-cloud-aws-backup-and-restore.html.

[pincy]

This still does not answer my question, how can i backup the decrypted data.

Also the page does not explain the behaviour of the aws backup mechanics if the current stack was already created from a backup.
Will the new backup be independent and contain all data from the initial backup as well?