4.1.0 upgrade experience (macos)

[ssbarnea]

After brew upgraded podman from 4.0.3 to 4.1.0, I endup up with a broken podman:

$ podman version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:50691/run/user/501/podman/podman.sock) failed: dial tcp [::1]:50691: connect: connection refused
FAIL: 125

ssbarnea@m1: ~/c/a
$ podman system connection list
Name                         URI                                                         Identity                                     Default
new                          ssh://core@localhost:60794/run/user/501/podman/podman.sock  /Users/ssbarnea/.ssh/new                     false
new-root                     ssh://root@localhost:60794/run/podman/podman.sock           /Users/ssbarnea/.ssh/new                     false
podman-machine-default       ssh://core@localhost:50691/run/user/501/podman/podman.sock  /Users/ssbarnea/.ssh/podman-machine-default  true
podman-machine-default-root  ssh://root@localhost:50691/run/podman/podman.sock           /Users/ssbarnea/.ssh/podman-machine-default  false

ssbarnea@m1: ~/c/a
$ podman machine start
Migrating machine "podman-machine-default"
Migrating machine "my-machine"
Migrating machine "new"
Error: cannot start VM podman-machine-default. VM new is currently running: only one VM can be active at a time
FAIL: 125

If I understand correctly the migration script did not know to stop the podman-machine-default before migrating it.

[ssbarnea]

Ok, so I got bit confused by the fact that "new" machine name was in the middle of the sentence and I missed it, assuming that podman-machine-default was the one already running. Can we make it colored, bold or add some quotes to its name, so the message is more clear? VM named 'new' ... would be safer to read.

Now, after doing the cleanup, stopping and removing old vms, i endup with a single one, migrated. Still, it looks weird to see that that one is outdated, see output of podman version:

$ podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18.1
Built:        Thu May  5 21:07:47 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.16.14
Built:        Thu Mar  3 14:58:50 2022
OS/Arch:      linux/arm64

Basically this means that migration does not upgrade the server version, something that I was expected to happen, or at least to see a warning message telling me to recreate the vm (rm and new)?

What do you think? I did not raise a bug as I am not yet sure which changes we should make to improve the experience.

Even more interesting:

$ podman machine rm
Error: running vm "podman-machine-default" cannot be destroyed

So how am I supposed to effectively upgrade the vm/server?

[rhatdan]

The problem is the community updates the podman version on brew faster then we can update the machine. I think the machine should be updated yesterday or today.

[ssbarnea]

Maybe we should look into automating the release so they happen relatively at the same time? Do you think that would be possible? I am asking as I do not know how pipelines are configured now.

[rhatdan]

Sadly, It is difficult to do. Right now we can do it a little better since we are releasing the FCOS. We are stopping that soon. What we need to do is block the brew release before the FCOS is ready. Or better yet, stop breaking the API, so newer podman client can work well with older podman service.