Fully self-contained deployment

[guitmz]

I noticed that the container deployment reaches out to external urls (like CDN, etc) to load the emulators. This means that the application cannot be used offline, the machine hosting it requires an internet connection.

I suggest the dependencies to be vendored inside the container to allow for a fully self-contained experience without requiring an Internet connection.

[geoffyuen]

Yeah I think this would be great - especially the js dependencies. I was also thinking the the Libretro thumbnails could be copied locally so that they're cached.

[arianrhodsandlot]

I've also thought about this, but I put it on hold because the workload might be larger than expected, and the benefits aren't as obvious. The workload isn't due to technical difficulty. It's just that it would be quite tedious.

Out of curiosity, is this expectation due to privacy concerns, network environment issues, or simply an aesthetic preference?

[guitmz]

For me, I just want to be able to use the app without internet connectivity. It just feels right.

The core emulator JavaScript dependencies should be quite simple to vendor in with docker, and should be enough for gameplay functionality. Not sure about the others like cover images etc but those are aesthetic

[psonhai]

I've looked into this and found the file responsible for loading emulator cores: src/pages/library/utils/nostalgis.ts. I replaced the URL generator: getCoreCDNUrl(core: string) with my own and updated the shader prefix url. After those changes, I was able to load the emulators completely offline.

function getCoreCDNUrl(core: string) {
  const externalCores = ['a5200', 'prosystem', 'stella2014', 'mupen64plus_next']
    const segments = externalCores.includes(core)
      ? [
          'cores',
          'custom',
          `${core}_libretro.zip`,
        ]
      : [
          'cores', 
          'retroarch', 
          `${core}_libretro.zip`
        ]
    const path = segments.join('/')
    return new URL(path, window.location.origin)
}

The cores are served in public/cores/custom & public/cores/retroarch. This should be good enough as a quick fix for now. I'm also looking into a way to cache images and box art locally.

[arianrhodsandlot]

This issue is essentially a trade-off, but one that’s harder to make than it might seem.

Some might assume that we only load a few simple JavaScript files from external sources, but that’s not accurate. The external files we load fall into three categories:

1. Device-specific assets such as photos and icons, hosted at https://github.com/arianrhodsandlot/retroassembly-assets, totaling around 12 MB.
2. Emscripten-compiled emulator binaries from RetroArch’s official builds, located at https://github.com/arianrhodsandlot/retroarch-emscripten-build. This is where the bulk of the storage usage comes from. We support 32 emulators; assuming an average of 1.5 MB each, plus notably larger cores like FBNeo (~10 MB) and Mame2003Plus (~6 MB), the total easily exceeds 60 MB.
3. Custom-compiled cores not provided by RetroArch’s official builds, developed by RetroAssembly itself and hosted at https://github.com/arianrhodsandlot/retroassembly-custom-cores, adding roughly 3 MB.

Bundling these into a Docker image inflates its size by at least 75 MB. While this may seem modest compared to some larger images, it already exceeds the current size of RetroAssembly’s own Docker image (which stands at 64.5 MB).

[arianrhodsandlot]

I'm open to suggestions on how to solve this issue—feel free to share your thoughts. However, it's best to minimize the impact. Because a significant portion of users don't need all the emulators; they might only need NES and SNES, for example.

[psonhai]

I'm trying to intercept every call to getCDNUrl and redirect it to another function that calls the customized server API. The server would first check whether the requested image already exists locally. If it does, it simply returns the cached image. If it doesn't, the server downloads it, stores it for future use, and then returns it.

The only downside is that the first request is pretty slow because the image has to be downloaded and that the server has to have the connection to the internet for the first time. Since it's only a one-time cost, I don't really think it can be a problem, to me at least. Other than that I wrote some code to check if the request is from local network, and only serve those resources from server if they are from local network, as it might be faster, and less burden on the server, to serve from CDNs outside of local network.

For the emulator cores, I think the same approach would work. The server could download a core the first time it's needed and serve the cached copy afterward when accessed from the local network. Alternatively, I could create a page with buttons that let users download only the cores they want, and optionally remove any cores they later unselect.

I'm a novice developer, but I'll help with this issues if you want a hand.

[guitmz]

Multiple images then? One as today, another with the extra 75mb and things vendored in.

Potentially even other images with specific emulators vendored in.

Then the user can choose. In my case, I would easily download 75mb more over the convenience of being able to use the tool any time even without internet or outages from upstream providers.