Rocky Linux 9 and Ubuntu 24.04 template with cloud-init doesn't reset password on next subsequent boots

[hanisirfan]

ISSUE TYPE

• Other

COMPONENT NAME

UI, cloud-init integration

CLOUDSTACK VERSION

CloudStack 4.19.1.1

OS / ENVIRONMENT

cloud-init.noarch 23.4-7.el9_4.5.0.1 on Rocky Linux 9.4 and Ubuntu 24.04

SUMMARY

STEPS TO REPRODUCE

The steps below are for Rocky Linux but somewhat identical for Ubuntu.

Basically I ran the commands below to prepare a working Rocky Linux 9.4 template with password management/reset. I'm just following the CloudStack documentation on cloud-init integration.

Install the necessary packages
```bash
# Verify the hostname is set to localhost
hostnamectl

# Update the system first
dnf -y upgrade && reboot now

# Enable EPEL repo
dnf config-manager --set-enabled crb
dnf -y install epel-release
crb enable

# Install common packages
dnf -y install curl wget nano tmux vim telnet acpid fastfetch

# Install guest agents
## QEMU agent: https://pve.proxmox.com/wiki/Qemu-guest-agent
dnf -y install qemu-guest-agent
systemctl enable --now qemu-guest-agent

## VMware agent: https://docs.vmware.com/en/VMware-Tools/12.4.0/com.vmware.vsphere.vmwaretools.doc/GUID-C48E1F14-240D-4DD1-8D4C-25B6EBE4BB0F.html
dnf -y install open-vm-tools

# Install cloud-init package
dnf -y install cloud-init

# Configure cloud-init to detect Cloudstack data source during runtime.
nano /etc/cloud/cloud.cfg.d/99_cloudstack.cfg
--------------------------------------------------------------------------------------------------------------
#cloud-config
datasource_list: [ ConfigDrive, CloudStack, None ]
datasource:
  CloudStack: {}
  None: {}
--------------------------------------------------------------------------------------------------------------

# Enable cloud-init without any aid from ds-identify
echo "policy: enabled" >  /etc/cloud/ds-identify.cfg

# Enable set_passwords module on every boot
sed -i s/" - set[_|-]passwords"/" - [set_passwords, always]"/g /etc/cloud/cloud.cfg

# Configures root password with cloud-init
## lock_passwd: false = Allow password login.
## disable_root: false = Allow root to remotely SSH
## ssh_pwauth: true = Accept password when logging into SSH
nano /etc/cloud/cloud.cfg.d/80_user.cfg
--------------------------------------------------------------------------------------------------------------
#cloud-config
system_info:
  default_user:
    name: root
    lock_passwd: false
disable_root: false
ssh_pwauth: true
--------------------------------------------------------------------------------------------------------------

# Enable Cloudstack reset SSH keys feature configure cloud-init ssh module to run on every boot.
sed -i s/" - ssh$"/" - [ssh, always]"/g /etc/cloud/cloud.cfg

# Disable cloud-init regenerating host certificates on boot.
echo "ssh_deletekeys: false" > /etc/cloud/cloud.cfg.d/49_hostkeys.cfg

# Partition management
## Install Growpart module
dnf -y install cloud-utils-growpart

## Locate the root partition.
lvs
vgs
pvs

## Every boot growpart will check and extend <PHYSICAL_VOLUME_PARTITION e.g. /dev/vda2> if there is change in size
nano /etc/cloud/cloud.cfg.d/50_growpartion.cfg
--------------------------------------------------------------------------------------------------------------
#cloud-config
growpart:
  mode: auto
  devices: ['/dev/vda2']
  ignore_growroot_disabled: false
--------------------------------------------------------------------------------------------------------------

## Extend volume group and root LV (runcmd is used) (this is for XFS filesystem)
nano /etc/cloud/cloud.cfg.d/51_extend_volume.cfg
--------------------------------------------------------------------------------------------------------------
#cloud-config
runcmd:
  - [ cloud-init-per, always, grow_VG, pvresize, /dev/vda2 ]
  - [ cloud-init-per, always, grow_LV, lvresize, -l, '+100%FREE', /dev/rocky/root ]
  - [ cloud-init-per, always, grow_FS, xfs_growfs, /dev/rocky/root ]
--------------------------------------------------------------------------------------------------------------

## Enable autoresize on every boot
sed -i s/" - runcmd"/" - [runcmd, always]"/g /etc/cloud/cloud.cfg
sed -i s/" - scripts_user"/" - [scripts_user, always]"/g /etc/cloud/cloud.cfg

## Network configuration with ConfigDrive
echo -e "\nnetwork: {}" >> /etc/cloud/cloud.cfg

# Configures cloud-init final message
#cloud-config
nano /etc/cloud/cloud.cfg.d/100_extend_volume.cfg
--------------------------------------------------------------------------------------------------------------
#cloud-config
final_message: |
  Welcome to Rocky Linux 9 running on Nebula Cloud!
--------------------------------------------------------------------------------------------------------------

# cloud-init clean up
cloud-init clean --machine-id
rm -rf /etc/sudoers.d/*

# Template clean up

## Remove the udev persistent device rules and DHCP leases
rm -f /etc/udev/rules.d/70*
rm -f /var/lib/dhclient/*
rm -f /var/lib/NetworkManager/*.lease

## Remove SSH Keys to ensure template instances doesn't have the same SSH keys.
rm -f /etc/ssh/*key*

## Cleaning log files
cat /dev/null > /var/log/audit/audit.log 2>/dev/null
cat /dev/null > /var/log/wtmp 2>/dev/null
logrotate -f /etc/logrotate.conf 2>/dev/null
rm -f /var/log/*-* /var/log/*.gz 2>/dev/null

## Set User password to expire
passwd --expire root

## Clearing User History
history -c
unset HISTFILE

## Shutdown the Instance
halt -p

EXPECTED RESULTS

When the template has "Password enabled" turned on, I'm expecting these results:
1. The instance uses the randomly generated password on the first boot.
2. The instance uses a new randomly generated password after shutting it down and using the "Reset password" icon in the UI.

ACTUAL RESULTS

1. This works.
2. The newly generated password is not being applied by cloud-init. But, on the same instance, I ran clean-init clean --logs and rebooted the instance. The new password is applied after that.

My initial thought after looking at the logs is that, the cc_set_passwords module did ran with always frequency as configured. But, it's using a cached password from CloudStack datastore.

This is my first time trying to create template especially with cloud-init being integrated. My previous tests with CloudStack is jus using the default CentOS 5 template. Tried following the issue #8767 , which resulted in the config I pasted on STEPS TO REPRODUCE section.

I have additional question, say a user change the CIDR of their guest network, will cloud-init still able to detect the metadata hosted on the VR with the other CIDR than default?

Sorry if I'm not supposed to post on GitHub and should use the mailing list instead.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

just for your information, I have tested the ubuntu 24.04 cloud images, the ssh and password reset worked fine
https://cloud-images.ubuntu.com/minimal/releases/noble/release/

it may be better to share the /var/log/cloud-init*log or journal logs

• journalctl -xeu cloud-init
• journalctl -xeu cloud-config

[top-secrett]

@weizhouapache @hanisirfan I faced with same problem. I create new template from iso image. After reboot new password doesn't apply. But if I do cloud-init clean --logs command before reboot, new password applies.
My configuration is similar

[weizhouapache]

@weizhouapache @hanisirfan I faced with same problem. I create new template from iso image. After reboot new password doesn't apply. But if I do cloud-init clean --logs command before reboot, new password applies. My configuration is similar

thanks @top-secrett
early last week I faced a similar issue, same as you, I have to run cloud-init clean

maybe cloud-init clean should be the last step before creating a vm template

[top-secrett]

@weizhouapache it works only once.
I've tryed to deal with this script https://github.com/apache/cloudstack/blob/main/setup/bindir/cloud-set-guest-password.in
After resetting password in CS new password applyed in instance, BUT if I reboot intance again password returns to the previous one.

[hanisirfan]

Was the cloud-set-guest-password not required anymore for new distro versions? Btw @weizhouapache, I tested on both minimal Rocky Linux 9 and 24.04 with normal ISOs. Not the cloud images. I can try to reproduce the error and attach the logs. My first suspicion after reading some portion of my logs is that the cloud-init set-passwords module runs on every boot, but not fetching new passwords from the VR.

…

On Mon, 9 Sept 2024, 22:12 top-secrett, ***@***.***> wrote: @weizhouapache <https://github.com/weizhouapache> it works only once. I've tryed to deal with this scrypt https://github.com/apache/cloudstack/blob/main/setup/bindir/cloud-set-guest-password.in After resetting password in CS new password applyed in instance, BUT if I reboot intance again password returns to the previous one. — Reply to this email directly, view it on GitHub <#9630 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APZMOVJIHTZ6VQVVEKI7K7DZVWUEZAVCNFSM6AAAAABNTJD5XGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYGIZTSMJWGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[top-secrett]

@hanisirfan @weizhouapache I think I found a solution. Maybe It is not the best, but it works on Ubuntu 22.

1. In /etc/cloud/cloud.cfg do not touch - set_passwords.
2. Create service based on scipt https://github.com/apache/cloudstack/blob/main/setup/bindir/cloud-set-guest-password.in
   2.1. Place it in folder /etc/init.d/ folder with name for example "reset-cloud-password"
   2.2. chmod +x /etc/init.d/reset-cloud-password
   2.3. Add lines in script after #!/bin/bash

### BEGIN INIT INFO
# Provides:          password
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: your description here
### END INIT INFO

And change user in script if it is necessary

2.4. Execute command to create service systemctl enable reset-cloud-password
2.5. Edit service systemctl edit --full reset-cloud-password
Add lines in the end

[Install]
WantedBy=multi-user.target

2.6. Enable service systemctl enable reset-cloud-password

[top-secrett]

On Almalinux 9 all works fine without script. You just neet to install wget to get password from virtual router

[tdtmusic2]

On Almalinux 9 all works fine without script. You just neet to install wget to get password from virtual router

For me it doesn't work, not even at first boot, I tried via user-data with packages: - wget in the cloud-config script. Should I manually install wget on the cloud image and then convert it to a template?

[top-secrett]

Yes, preinstall it in the template

[tdtmusic2]

That didn't work either... I got wget on the generic cloud image with virt-customize, registered the template in ACS then deployed an instance with it. No password reset, cannot inject any ssh key (I don't want a predefined ssh key in cloud-init cfg). This cloud-init thing has proven to be a nightmare for me with those rhel based distros, yet it works flawlessly in anyhting debian/ubuntu. Get the cloud image, attach the user-data to the template and that's it, everything works.

[top-secrett]

@hanisirfan On Ubuntu 22 instead service with script above, I created service with script in /etc/init.d/ and enabled it. In cloud.cfg - [set_passwords, always]

### BEGIN INIT INFO
# Provides:          password
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: your description here
### END INIT INFO
cloud-init clean

[weizhouapache]

@hanisirfan On Ubuntu 22 instead service with script above, I created service with script in /etc/init.d/ and enabled it. In cloud.cfg - [set_passwords, always]

### BEGIN INIT INFO
# Provides:          password
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: your description here
### END INIT INFO
cloud-init clean

@top-secrett
that could be a good solution. will it run before cloud-init ?

It looks like the issue is, without cloud-init clean, cloud-init will not re-fetch userdata/metadata/password information from cloudstack VR, therefore new sshkey/password are not applied.

[top-secrett]

will it run before cloud-init ?

I don't know exactly, but it works. BUT I try to find another solutions, because after cloud-init clean my runcmd scripts lvresize and resize2fs don't work :(

[hanisirfan]

I was having problem with Rocky Linux 9.4 and Ubuntu 24.04. Rocky now fixed but Ubuntu haven't. It's due to a bug in cloud-init. Can refer here: canonical/cloud-init#5486

[top-secrett]

@hanisirfan thank you
For now I found working solution. All my features like password and ssh key reset, lvresize and resize2fs work right

### BEGIN INIT INFO
# Provides:          password
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: your description here
### END INIT INFO
cloud-init clean
cloud-init init --local
cloud-init init
cloud-init modules --mode config
cloud-init modules --mode final

[weizhouapache]

@hanisirfan @top-secrett
thank you both for the sharing

[DaanHoogland]

thanks guys, it sounds like this does not need fixing in CloudStack, is that right?

[weizhouapache]

thanks guys, it sounds like this does not need fixing in CloudStack, is that right?

I think yes, you are right
you can move to a discussion

[hanisirfan]

cloud init 24.3.1 is now in the updates channel for Ubuntu 🎉 I'll try it out later.

[DaanHoogland]

@hanisirfan is this still a problem or can we close this discussion item?

[hanisirfan]

Yep we can close this discussion.