Upgrade from Ubuntu 18.04 LTS (Bionic Beaver) to Ubuntu 20.04 LTS (Focal Fossa) ?

[fingolfin]

Perhaps we should update this base image from Ubuntu 18.04 to 20.04?

Pro: Newer versions of lots of dependencies, and the homalg team seems to need those (at least they run apt dist-upgrade on our Docker images, ping @zickgraf and @mohamed-barakat)
Con: Requires work for testing it; and as a result, GAP packages using our docker images now are only tested on a fairly new Linux distribution, while many users might still be on 18.04.

Perhaps we should just offer both images based on 18.04 and on 20.04? Thoughts?

[zickgraf]

Newer versions of lots of dependencies, and the homalg team seems to need those (at least they run apt dist-upgrade on our Docker images, ping @zickgraf and @mohamed-barakat)

We (probably) do not need the new versions. The reason why I do an apt dist-upgrade is that we afterwards install additional software, and coming from Arch Linux I'm used to the philosophy: "partial updates will break your system". I'm not sure if Ubuntu supports partial updates, but even if it does officially, I do not want to be the single person on this planet with this specific combination of versions installed, so I decided to simply update everything.

[fingolfin]

Actually most distros are built around the concept of "partial updates", as you call it; this is perfectly safe. It's in general really bad idea to use apt dist-upgrade when using a Docker image; in a sense, it turns the whole idea of downloading a preinstalled image on its head: now your builds are not reproducible anymore, because when I re-run it the next day, apt dist-upgrade may install different versions of stuff, and it's really difficult to figure this out. It also can easily double the amount of data you need to download (you first download the premade image, then potentially update everything in it, so you need to download it again). And it reduces the effectiveness of image/container caching at CI provides substantially for the same reason.

Really, Arch Linux is the oddball (together with Gentoo) :-)

[zickgraf]

it turns the whole idea of downloading a preinstalled image on its head

My primary motivation for using the preinstalled docker container is not having to compile all the dependencies myself.

now your builds are not reproducible anymore, because when I re-run it the next day, apt dist-upgrade may install different versions of stuff, and it's really difficult to figure this out.

My problem is that even without a dist-upgrade, simply installing packages already breaks the reproducibility, since Ubuntu does not provide a way to install old versions of packages (as far as I know). So I would have to create and maintain a separate docker container just to achieve reproducibility at the system level, while until now all problems I had with reproducibility have been caused by GAP packages, not by the underlying system.

It also can easily double the amount of data you need to download (you first download the premade image, then potentially update everything in it, so you need to download it again).

Yes, knowing that "partial updates" are supported, I will remove remove the dist-upgrade in our CI, so we can save this bandwidth and time :-) (Hoping that I will not run into errors with the very specific combination of old and new versions :D )

[fingolfin]

Ubuntu (like Debian) does allow installing older versions of packages; you can specify a package version (including revision) when install any package, and people in fact use this frequently (and also e.g. "pin" packages to a specific version to prevent it from upgrading).

[zickgraf]

I know that I can provide a fixed version to apt, but the old versions seem to be removed from the mirrors. For example, if I execute

sudo apt install vim

in the current gap-docker container, I get the following errors:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/xxd_8.0.1453-1ubuntu1.3_amd64.deb  404  Not Found [IP: 91.189.88.152 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim-common_8.0.1453-1ubuntu1.3_all.deb  404  Not Found [IP: 91.189.88.152 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim-runtime_8.0.1453-1ubuntu1.3_all.deb  404  Not Found [IP: 91.189.88.152 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.3_amd64.deb  404  Not Found [IP: 91.189.88.152 80]

as the mirror only has version 8.0.1453-1ubuntu1.4 available.

Are there "archive" mirrors for Ubuntu? I have found https://launchpad.net/ubuntu/+archivemirrors, but this does not seem to be what I want (as the mirrors do not seem to have fixed dates). If there are such "archive" mirrors I think it would make sense to use them in the docker images to always have the package versions of the day the image was created.