About Micro OS snapshots becoming outdated

[pat-s]

After using this module for some months now, I (finally) ran into this issue which I already expected from the beginning: As all new nodes use the static snapshot created when starting out, this snapshot becomes outdated over time.

With a working kured, new nodes spun up by the autoscaler will be checked by kured. At some point, kured will immediately apply updates. If these include a new kernel, it will also drain them and reboot.

With the default config, kured will operate at any time and hence, the just freshly launched node (with 'fresh' being debatable due to the outdated snapshot) will be automatically drained and rebooted a few minutes after creation.
This terminates the workload on the node, causing disruption for possible time-limited processes which were supposed to run and finish cleanly.

Possible Solutions

• Find a way to build new images centrally and allow to spin up hetzner nodes from them (unlikely/impossible)
• Emphasize the need to rebuild the snapshot frequently (e.g. once a week using CI/CD)
• Don't run kured 24/7 by default but only at a time where nodes are infrequently spun up or no temporary workloads are processed (maybe 6 am?)

[maggie44]

I think Kured can be configured to reboot on a schedule instead: https://kured.dev/docs/configuration/

[pat-s]

Yes, that's what I also do meanwhile. However the default of this model is to reboot immediately.

Besides, there is still a need to recreate the ISOs every once in a while.

[maggie44]

It is helpful to recreate the ISOs once in a while. I mentioned the reboot as although an old OS would come up with the autoscaler, setting the schedule would stop the scenario of Kured immediately triggering a restart during the peak load period.

Would be nice to have the process for updating the ISO images documented though, I am due an update I'm sure.

[pat-s]

export HCLOUD_TOKEN=
packer init hcloud-microos-snapshots.pkr.hcl
packer build --force hcloud-microos-snapshots.pkr.hcl

[pat-s]

👆️ I am running this now on schedule once per week for a while without issues. Fresh images every week.

[mysticaltech]

Great to know!