Fetch all URLs from a single URL, fix #46

[illwieckz]

Only one URL is hardcoded, which is a json file telling everything else.

The format is meant to be able to update the paths after a release, and to improve the updater to support multiple mirrors in the future. See #46

[slipher]

The overall concept is good. Besides permitting flexibility in the server configuration, this will be helpful when testing the updater.

However, I think some more work will be needed to correctly deal with the asynchronous nature of the versions.json request: the results are not always available when you want them. For example, if the game update page is opened while we still don't have the news URL (this will be especially important with the feature I am working on for Windows to relaunch as administrator, in which the splash screen will be skipped on the second launch).

[illwieckz]

We may postpone the news part, right? For the updater and game URLs I don't believe this PR adds more issue than there was before since those URLs were computed from fetched version numbers anyway.

[slipher]

It doesn't introduce an issue for the updater URL, since updater update can only be initiated when the versions request has succeeded.

It does create new problems for the game update. The game URL actually doesn't depend on the version number currently.

[illwieckz]

The game URL actually doesn't depend on the version number currently.

Ah, yes… 😕

[illwieckz]

So I rebased, and race conditions should now be avoided.

Here is an example of current.json: https://cdn.unvanquished.net/current.json

{
	"news": {
		"version": "rolling",
		"mirrors": [ "https://unvanquished.net" ],
		"parcels": {
			"all-all": {
				"path": "api/get_recent_posts",
				"content": "raw",
				"container": "json"
			}
		}
	},
	"game": {
		"version": "0.54.1",
		"mirrors": [ "https://cdn.unvanquished.net" ],
		"parcels": {
			"all-all": {
				"path": "unvanquished_0.54.1.torrent",
				"content": "mixed-files",
				"container": "torrent"
			}
		}
	},
	"updater": {
		"version": "0.2.0",
		"mirrors": [ "https://github.com/Unvanquished/updater/releases/download/v0.2.0" ],
		"parcels": {
			"linux-amd64": {
				"path": "UnvUpdaterLinux.zip",
				"content": "single-file",
				"container": "zip"
			},
			"windows-i686": {
				"path": "UnvUpdaterWin.zip",
				"content": "single-file",
				"container": "zip"
			},
			"macos-amd64": {
				"path": "UnvUpdaterOSX.zip",
				"content": "single-directory",
				"container": "zip"
			}
		}
	}
}

This new file is already generated by default on release (I added generation code for it in release scripts in 2021 when I opened this PR).

Some command line changed, the --internalcommand updateupdater will reinstall the updater from the current known version if --updaterurl is not set, or the one from --updaterurl if provided.

The json file already supports multiple mirrors, but the updater code doesn't for now. In the future I want it to.

[slipher]

Revisiting this due to Unvanquished/Unvanquished#3457. The sticking point was that adding URLs into the former versions.json creates dependencies on the versions request that we didn't have before. So we would need the versions request for these things:

• Updater download link (on Github). This is not a big problem since it is only used if the version request succeeded. But for Windows admin elevation we would have to plumb the URL through the command line arguments, instead of just the version.
• News endpoint. This would be the most painful of all to engineer around, but also gives us the least benefit. This should be left hard-coded for now as previously suggested.
• Torrent URL for game download. It would be feasible to wait for this URL during the update process. But maybe we can avoid the extra request entirely by mirroring the torrent file to the same servers that the JSON is mirrored to? Now the torrent appears to have some dynamically populated content in the list of trackers, so I would like to know how it is generated. Is it generated once when publishing a release or is the tracker list part more frequent regularly updated?

[illwieckz]

Now the torrent appears to have some dynamically populated content in the list of trackers, so I would like to know how it is generated. Is it generated once when publishing a release or is the tracker list part more frequent regularly updated?

We have a script fetching current lists of known good trackers and merging them, then the release script use that merged list. Both scripts (fetching tracker list, and publishing a release) are part of the release process. So it is generated once when publishing a release.

[illwieckz]

Updater download link (on Github). This is not a big problem since it is only used if the version request succeeded. But for Windows admin elevation we would have to plumb the URL through the command line arguments, instead of just the version.

If we want the updater iterate a list of mirrors, we better not blumb the list of URLs through the command line arguments but let the elevated updater fetch the list of mirrors by itself, even if that means downloading twice the json file.

[illwieckz]

Torrent URL for game download. It would be feasible to wait for this URL during the update process. But maybe we can avoid the extra request entirely by mirroring the torrent file to the same servers that the JSON is mirrored to?

Peoples syncing the CDN with rsync are already syncing the torrent file and that json file (they're all stored in the same folder).

Unlike the original https://dl.unvanquished.net/versions.json, this was designed from the start to be hosted as https://cdn.unvanquished.net/current.json to make it purposely synced with the CDN.

[illwieckz]

The thing is that we may have torrent mirrors on places where there is no version json mirrored files, and the json mirrored file may be mirrored before the torrent is.

So we may save a request by looking at the torrent file on the same place the version file is, but we would still do a request as a fallback if for some reason the CDN is incompletely mirrored and we need to find another torrent location.

[illwieckz]

Here is the current generated https://cdn.unvanquished.net/current.json file:

{
	"news": {
		"version": "rolling",
		"mirrors": [ "https://unvanquished.net/" ],
		"parcels": {
			"all-all": {
				"path": "api/get_recent_posts",
				"content": "wordpress-posts",
				"container": "json"
			}
		}
	},
	"game": {
		"version": "0.55.5",
		"mirrors": [
			"https://github.com/Unvanquished/Unvanquished/releases/download/v0.55.5/",
			"https://master.dl.sourceforge.net/project/unvanquished/v0.55.5/",
			"https://cdn.unvanquished.net/",
			"https://cdn.illwieckz.net/unvanquished/",
			"https://unvcdn.viech.name/",
			"https://webseed.unv.kangz.net/",
			"https://fosstorrents.com/files/download.php?file="
		],
		"parcels": {
			"all-all": {
				"path": "unvanquished_0.55.5.torrent",
				"content": "mixed-files",
				"container": "torrent"
			}
		}
	},
	"updater": {
		"version": "0.2.1",
		"mirrors": [
			"https://github.com/Unvanquished/updater/releases/download/0.2.1/",
			"https://master.dl.sourceforge.net/project/unvanquished/Updaters/0.2.1/"
		],
		"parcels": {
			"linux-amd64": {
				"path": "UnvUpdaterLinux.zip",
				"content": "single-file",
				"container": "zip"
			},
			"windows-i686": {
				"path": "UnvUpdaterWin.zip",
				"content": "single-file",
				"container": "zip"
			},
			"macos-amd64": {
				"path": "UnvUpdaterOSX.zip",
				"content": "single-directory",
				"container": "zip"
			}
		}
	}
}

I verified that the updater succeeds to download the torrent from GitHub, SourceForge and FOSS Torrents, so redirections are properly handled in that part of the updater.

[slipher]

Updater download link (on Github). This is not a big problem since it is only used if the version request succeeded. But for Windows admin elevation we would have to plumb the URL through the command line arguments, instead of just the version.

If we want the updater iterate a list of mirrors, we better not blumb the list of URLs through the command line arguments but let the elevated updater fetch the list of mirrors by itself, even if that means downloading twice the json file.

We probably don't want a list of mirrors for this anyway since we just have to trust the download source (unless we add more code to check a hash). Are there any plans to host the updater binary somewhere besides Github? This would increase code complexity since for the updater update case we would have to add extra logic to wait indefinitely for the versions data, error if an update is not detected, etc.

The thing is that we may have torrent mirrors on places where there is no version json mirrored files, and the json mirrored file may be mirrored before the torrent is.

So we may save a request by looking at the torrent file on the same place the version file is, but we would still do a request as a fallback if for some reason the CDN is incompletely mirrored and we need to find another torrent location.

As mentioned in the other thread, we probably don't want to mirror the torrent everywhere since it needs to be trusted, like the JSON file.

What if we embed the torrent inside the JSON? Then there would be no problems with synchronization or chained downloads.

[illwieckz]

We probably don't want a list of mirrors for this anyway since we just have to trust the download source (unless we add more code to check a hash). Are there any plans to host the updater binary somewhere besides Github?

The updater is usually mirrored to SourceForge and we trust that.

[illwieckz]

I added a commit that can look for alternate current.json files and it works.

I also added a commit that attempts to also iterate the torrents, but it crashes. I guess we never took care of properly closing a download before, so starting a download while once have been tried first is crashing the updater.

I kept a current.json file online with obvious mistakes to trigger the iteration of mirrors (the current.json file isn't used by the released updater, so that's fine), this makes it easier to test.

[illwieckz]

Or maybe that's the contrary: we destroy too much things when stopping a download, and maybe it was never written to be able to do more than one download.

Actually, removing aria2::libraryDeinit(); from the AriaDownloader destructor makes it work.

[illwieckz]

For the updater url passed to the command line, we may implement an alternate command line option where we pass the current.json file url on the command line as a way to tell “download the updater from the urls in it, that's the one that told us to update”.

[illwieckz]

Or we may add as much as url there are on the command line.

[illwieckz]

Like this:

./updater --splashms 1 --internalcommand updateupdater \
 --updaterurl 'https://github.com/Unvanquished/updater/releases/download/0.2.1/UnvUpdaterLinux.zip' \
 --updaterurl 'https://master.dl.sourceforge.net/project/unvanquished/Updaters/0.2.1/UnvUpdaterLinux.zip'

[illwieckz]

Like this:

./updater --splashms 1 --internalcommand updateupdater \
 --updaterurl 'https://github.com/Unvanquished/updater/releases/download/0.2.1/UnvUpdaterLinux.zip' \
 --updaterurl 'https://master.dl.sourceforge.net/project/unvanquished/Updaters/0.2.1/UnvUpdaterLinux.zip'

I implemented that. I don't know if it works, but at least it builds.

[illwieckz]

It looks like everything is now implemented.

I verified that if torrent or updater mirror is wrong, the next in list is picked, and if all urls are wrong, an error message is displayed after having iterated all the mirrors.

[illwieckz]

The mirroring of the current.json file is also useful, when for some reasons our server isn't reachable. It would be better if that wouldn't happen, but history reminds us we should expect it: hoster bricking their motherboard while updating firmwares, natural forces shutting down power in whole areas, provider disappearing overnight…

For the current.json files we may think about doing subdomains of unvanquished.net, so we keep the ability to disable a mirror even when the name is hardcoded in the updater. As far as I know we never had strong domain issues, that looks very reliable. Even when the domain was transferred from one company to another at their initiative, we didn't suffer from that.

[Gireen]

current.json could also be placed in the unv repository or, in case of unicorn problems, a separate one with github pages

[illwieckz]

current.json could also be placed in the unv repository or, in case of unicorn problems, a separate one with github pages

It's a way to mirror it on a reliable host, but I would prefer it to be mirrored on less static places. One of the purpose of having such current.json is to be able to edit it after the release to adjust for unexpected events. Unlike a torrent file that is most likely once for all, the url of the said torrent can move, for example. So it's better if the replication of the current.json file is the most straightforward possible. Our set of CDN mirrors is a good candidate for such replication. Not all CDN mirrors may be good candidates for that, but having 2 or 3 trusted ones being replicated with rsync would fit the need.

We already have something similar with the master server, as we have master.unvanquished.net which is in north america, and master2.unvanquished.net which is in Europe. The master2 is just a domain name that points to my server, we can do the same as cdn2.unvanquished.net. That cdn server already exists and is already replicated, but unlike master2 it isn't accessible with an unvanquished.net subdomain. Not only this would protect us from hoster unavailability, but having that cdn2 hosted in Europe would also protect us from some peering issues.