package_update fails due to running before initial clock synchronisation

[davidje13]

At the risk of being told this is actually an upstream issue (!), I've noticed that setting package_update: true fails due to it running too early in the boot process (before Initial clock synchronization):

/var/log/cloud-init-output.log

Cloud-init v. 25.2 running 'modules:config' at Mon, 24 Nov 2025 02:07:38 +0000. Up 22.66 seconds.
Cloud-init v. 25.2 running 'modules:final' at Mon, 24 Nov 2025 02:07:38 +0000. Up 22.74 seconds.
2025-11-24 02:07:38,843 - modules.py[WARNING]: Could not find module named cc_netplan_nm_patch (searched ['cc_netplan_nm_patch', 'cloudinit.config.cc_netplan_nm_patch'])
Hit:1 http://deb.debian.org/debian trixie InRelease
Get:2 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Get:3 http://archive.raspberrypi.com/debian trixie InRelease [54.8 kB]
Get:4 http://deb.debian.org/debian-security trixie-security InRelease [43.4 kB]
Err:2 http://deb.debian.org/debian trixie-updates InRelease
  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-13T07:44:39Z Verifying signature:            Not live until 2025-12-13T07:45:11Z
Err:3 http://archive.raspberrypi.com/debian trixie InRelease
  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T16:16:43Z
Err:4 http://deb.debian.org/debian-security trixie-security InRelease
  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T12:01:29Z Verifying signature:            Not live until 2025-12-12T12:01:29Z
Fetched 146 kB in 0s (1,157 kB/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://deb.debian.org/debian trixie-updates InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-13T07:44:39Z Verifying signature:            Not live until 2025-12-13T07:45:11Z
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://archive.raspberrypi.com/debian trixie InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T16:16:43Z
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://deb.debian.org/debian-security trixie-security InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T12:01:29Z Verifying signature:            Not live until 2025-12-12T12:01:29Z
W: Failed to fetch http://deb.debian.org/debian/dists/trixie-updates/InRelease  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-13T07:44:39Z Verifying signature:            Not live until 2025-12-13T07:45:11Z
W: Failed to fetch http://deb.debian.org/debian-security/dists/trixie-security/InRelease  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T12:01:29Z Verifying signature:            Not live until 2025-12-12T12:01:29Z
W: Failed to fetch http://archive.raspberrypi.com/debian/dists/trixie/InRelease  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T16:16:43Z
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Note this actually booted on Dec 13 12:22 — the 24 Nov 2025 02:07:38 time matches the image I'm using: 2025-11-24-raspios-trixie-arm64-lite.img, so is presumably the "last known time" it uses before contacting the time server.

journalctl

Nov 24 02:07:41 pi5 cloud-init[668]: Reading package lists...
Nov 24 02:07:41 pi5 cloud-init[668]: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://deb.debian.org/debian trixie-updates InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-13T07:44:39Z Verifying signature:            Not live until 2025-12-13T07:45:11Z
Nov 24 02:07:41 pi5 cloud-init[668]: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://archive.raspberrypi.com/debian trixie InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T16:16:43Z
Nov 24 02:07:41 pi5 cloud-init[668]: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://deb.debian.org/debian-security trixie-security InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T12:01:29Z Verifying signature:            Not live until 2025-12-12T12:01:29Z
Nov 24 02:07:41 pi5 cloud-init[668]: W: Failed to fetch http://deb.debian.org/debian/dists/trixie-updates/InRelease  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-13T07:44:39Z Verifying signature:            Not live until 2025-12-13T07:45:11Z
Nov 24 02:07:41 pi5 cloud-init[668]: W: Failed to fetch http://deb.debian.org/debian-security/dists/trixie-security/InRelease  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T12:01:29Z Verifying signature:            Not live until 2025-12-12T12:01:29Z
Nov 24 02:07:41 pi5 cloud-init[668]: W: Failed to fetch http://archive.raspberrypi.com/debian/dists/trixie/InRelease  Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Not live until 2025-12-12T16:16:43Z
Nov 24 02:07:41 pi5 cloud-init[668]: W: Some index files failed to download. They have been ignored, or old ones used instead.
Nov 24 02:07:41 pi5 cloud-init[668]: Reading package lists...
Nov 24 02:07:41 pi5 cloud-init[668]: Building dependency tree...
Nov 24 02:07:41 pi5 cloud-init[668]: Reading state information...
Nov 24 02:07:42 pi5 cloud-init[668]: Calculating upgrade...
Nov 24 02:07:42 pi5 cloud-init[668]: 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[…]
Dec 13 12:22:13 pi5 systemd-timesyncd[416]: Contacted time server 178.62.68.79:123 (2.debian.pool.ntp.org).
Dec 13 12:22:13 pi5 systemd-timesyncd[416]: Initial clock synchronization to Sat 2025-12-13 12:22:13.715210 GMT.

I'm using dtoverlay=disable-bt,disable-wifi, I've applied patch #1 to fix issue #2, and I'm using an NVMe SSD for the OS rather than a memory card, all of which may be causing things to run a bit faster than usual, triggering these race conditions which may not otherwise be happening.

It also appears there are other risks from having the time sync while cloud-init is running: I see this log shortly after the clock sync occurs (which might be caused by me specifying packages: [vim], or by me using a few raspi-config commands in the runcmd section):

Dec 13 12:22:15 pi5 cloud-init[668]: ERROR: rpi-eeprom-update -d -i -f /tmp/tmp20d6wcmb/pieeprom.upd timeout

Which suggests the eeprom update is perhaps timing out based on wall clock time (I don't know what the timeout is supposed to be set to, but the system hasn't been running for very long at that point; maybe 30 seconds in total). Update: tracked down and reported this particular sub-issue on the eeprom repo raspberrypi/rpi-eeprom#790, but since many commands a user may add to runcmd may expect a somewhat stable wall clock time, it's probably best to make sure the time has synced (if possible) before running them anyway.

[lurch]

Hmm, interesting. I guess there's actually 3 different scenarios that need to be considered here:

1. Pi (with or without an RTC) doesn't have any network connection at all -> Pi's clock never gets updated by NTP (local time always monotonically increases)
2. Pi that has a correct RTC and a network connection -> Pi's clock might have a small jump when updated by NTP (local time effectively monotonically increases)
3. Pi without an RTC (or with an RTC set to the wrong time) but with a network connection -> Pi's clock might have a large jump (if Pi has been shut down for several days) when updated by NTP (local time has discontinuities)

Of these three different scenarios, only 3 is the one that's likely to have any problems, and seems to match with what you're describing above. Your issue in the rpi-eeprom repo shows that you're using a Raspberry Pi 5, so the good news is that you can easily move yourself into scenario 2 by buying an RTC battery 😉

but since many commands a user may add to runcmd may expect a somewhat stable wall clock time, it's probably best to make sure the time has synced (if possible) before running them anyway.

That's the "obvious" solution that would help in scenario 3, but it would also stall or slow down the boot in scenario 1 (or on any network where NTP is blocked).

If the user knows that they've added commands to runcmd which expect a stable wallclock time, and they know that the clock is likely to be changing during bootup because they've got a network connection, then they could also add commands to runcmd to wait for the NTP sync? (Looks like https://unix.stackexchange.com/questions/792705/how-to-prove-that-ntp-time-sync-is-checking-regularly has some useful suggestions)

[davidje13]

Note that even if the hardware supports an RTC (and has a battery), it will still encounter a large jump on the first boot (which is the one which is most likely to be using cloud-init!)

Having an RTC battery would only help for cases where existing hardware is being re-imaged, rather than set up for the first time.

A relatively easy fix (I think) would be to automatically retry the package update a few times (with a delay) if it fails for any reason (which would also be useful for avoiding network blips, for example).

If a user has specified package_update: true, it probably means they're fine to wait for a few retries on the first boot, rather than the alternative of the request failing and ending up effectively ignored. And if they've specified package_update: true but then install it on a Pi that doesn't have network access, that's the user's mistake and I think it's fair enough for the boot to get stuck for several minutes of retries before giving up.

It doesn't fix the runcmd side of things (which can also be much more sensitive to small changes of a few seconds like in scenario 2). But maybe it's worth chasing those issues down individually in the commands themselves; most issues would likely be timeouts using the wrong clock, like I encountered in raspi-config. But I'll note that it often isn't obvious that a command would be sensitive to wall clock time in the first place, so it's definiely something that's likely to catch people out and potentially be difficult to debug.

[lurch]

Note that even if the hardware supports an RTC (and has a battery), it will still encounter a large jump on the first boot (which is the one which is most likely to be using cloud-init!)

Having an RTC battery would only help for cases where existing hardware is being re-imaged, rather than set up for the first time.

Good point, I've added that scenario to my previous comment.

A relatively easy fix (I think) would be to automatically retry the package update a few times (with a delay) if it fails for any reason (which would also be useful for avoiding network blips, for example).

I've not looked at the code in this repository (or at cloud-init in general) yet, but rather than doing anything "automatically", I wonder if it might be better to have a package_update_retries: option? 🤔

[tdewey-rpi]

At the risk of being told this is actually an upstream issue (!)

That risk is well considered, because unfortunately, I think it is!

That said, I'll be sending a PR in their direction to try and correct this.

[davidje13]

@tdewey-rpi sounds good! Want me to open an issue in https://github.com/canonical/cloud-init for tracking and close this as a duplicate?

[tdewey-rpi]

@tdewey-rpi sounds good! Want me to open an issue in https://github.com/canonical/cloud-init for tracking and close this as a duplicate?

I'll try to fix it in our downstream first - I generally prefer to offer upstream a way out while reporting an issue