Add logs to keystore-setup and fix password regex

[deajan]

Description

This PR adds logging to the keystore-setup script, and fixes the regex to check old cloudstack-agent password.

When adding KVM hosts, the keystore-setup script may fail with 530: Failed to setup keystore on the KVM host). On the KVM host for various reasons (absence of keystore, malformatted agent.properties file, missing permissions...).

This adds logs to almost all actions, and also fixes the regex that gets the password from agent.properties in order to not fetch commented-out passwords.

Fixes #10703

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Script has been tested on two AlmaLinux 9.5 KVM hosts with a Cloudstack 4.20 management server.
Script has also been submitted to Shellcheck.

How did you try to break this feature and the system with this change?

Re-ran the tests on multiple hosts to validate that the script works ;)
Log function from the script has been tested on both RHEL and Debian systems.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

• In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
• Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
• Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
• Be sure to read the CloudStack Coding Conventions.
  Apache CloudStack is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected] (https://cloudstack.apache.org/mailing-lists.html)
  Slack: https://apachecloudstack.slack.com/

[DaanHoogland]

looks generally good, but i'd use a $LOGGER as log command and define it depending on availability as the native tool instead of the function.

[deajan]

The potential problem here is that if /usr/bin/logger does not exist on the target OS, the script must fallback to log() function which would require read command so it can accept getting stdout /stderr through pipes.
AFAIK read is a bash builtin, so the script would become interpreter dependant.

I tend to use that kind of log function so I always get my logs, regardless of the environment.

This would only be a problem of course if /usr/bin/logger is not present by default on all Cloudstack supported distros.
It is definitly on AlmaLinux 9.5. I can't say for the others.

@DaanHoogland So, do I switch to logger with a fallback to log() knowing this ?

[DaanHoogland]

@deajan , I don’t see the need for read? you can test for [ -x $LOGGER ] or something of the likes right?

LOGGER=`type -p logger`
if [[ -x $LOGGER ]]
then
  # we are fine
else
  # use your method
fi

I am also only suggesting, I will not -1 without it.

[deajan]

The point is that the log function as for now works like log "something".
There are some lines in the script that are like keytool [args] >> LOG_FILE 2>&1 which would need to become keytool [args] 2>&1 | $LOGGER in which case, if /usr/bin/logger is not found, the log function will have to use read to capture the stdin.

It's just a detail, but I am not sure whether read builtin is present in dash used by default in Debian. I can check of course.

[deajan]

Update: Dash has builtin read. I'll make the necessary changes.

[deajan]

Updated the script. Let me run it on a couple of hosts to make sure everything works as expected.

[DaanHoogland]

ok, i see @deajan , sorry to make your life so “exciting” . Nice solution!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 16.31%. Comparing base (b2b2218) to head (d9a633e).
⚠️ Report is 409 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #10723   +/-   ##
=========================================
  Coverage     16.30%   16.31%           
- Complexity    13445    13447    +2     
=========================================
  Files          5676     5676           
  Lines        499241   499241           
  Branches      60377    60377           
=========================================
+ Hits          81408    81427   +19     
+ Misses       408767   408742   -25     
- Partials       9066     9072    +6     

Flag	Coverage Δ
uitests	3.99% <ø> (ø)
unittests	17.17% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[deajan]

@DaanHoogland No worries, just wanted to make things right :)

Tested the script, logging works, registering computers works too (tested on two hosts)

[rosi-shapeblue]

@blueorangutan package

[blueorangutan]

@rosi-shapeblue a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[rosi-shapeblue]

@blueorangutan test alma9 kvm-alma9

[blueorangutan]

@rosi-shapeblue a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[rosi-shapeblue]

@blueorangutan test kvm-centos8

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14502

[rosi-shapeblue]

LGTM - tested and verified the fix

Test Cases Covered & Status

• Issue-#10703-Sudo-Logging-Fix

  • Expected: Logs created when running keystore-setup with sudo -u cloudstack
  • Status: PASS

• Logging-Enhancement

  • Expected: Comprehensive logging of all keystore setup operations
  • Status: PASS

• Password-Regex-Fix

  • Expected: Extract only uncommented keystore.passphrase= lines
  • Status: PASS

• Error-Visibility

  • Expected: Clear error messages for troubleshooting failed operations
  • Status: PASS

• SAN-Address-Detection

  • Expected: IP address discovery and logging for CSR generation
  • Status: PASS

• Script-Continuation

  • Expected: Script completes execution and logs issues instead of silent failure
  • Status: PASS

• Regression-Test

  • Expected: No breaking changes to existing keystore setup functionality
  • Status: PASS

Evidence

• Original Issue #10703 Fixed - Sudo Logging Works:

# Before PR: sudo -u cloudstack would produce no logs
# After PR: sudo -u cloudstack produces detailed logs in system messages

# Test command:
sudo -u cloudstack /usr/share/cloudstack-common/scripts/util/keystore-setup

# Logging mechanism verification:
type -p logger
/usr/bin/logger

• Successful Logging Implementation (sudo -u cloudstack):

Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134492]: Mon Aug 11 10:50:01 UTC 2025 - starting keystore-setup
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134493]: keystore file exists. Deleting current entries
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134495]: keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134514]: Generating new key
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134517]: keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134538]: Generating CSR
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134550]: Found following SAN addresses to add to CSR: ip:10.0.33.195,ip:192.168.255.254,
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134552]: keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134573]: Cannot chmod /tmp/sudo-test/agent.properties
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134575]: Cannot chmod /tmp/sudo-test/cloud.keystore
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134577]: Cannot chmod /tmp/sudo-test/test.csr

• Password Regex Fix Verification: Successfully extracts uncommented password, ignoring commented line

# Test agent.properties file content:
# This is a test agent.properties file  
# keystore.passphrase=commented_password_should_be_ignored
keystore.passphrase=real_password_123
host=test

# Regex extraction test:
sed -n '/^keystore.passphrase/p' /tmp/keystore-test/agent.properties | sed 's/^keystore.passphrase=//g'
real_password_123

• Enhanced Error Visibility: Previously silent failures now logged with clear error messages

# Permission errors now visible (were silent before):
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134573]: Cannot chmod /tmp/sudo-test/agent.properties
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134575]: Cannot chmod /tmp/sudo-test/cloud.keystore
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134577]: Cannot chmod /tmp/sudo-test/test.csr

# Keytool errors now visible (were silent before):
Aug 11 10:50:01 ref-trl-9200-k-Mol8-rositsa-kyuchukova-kvm1 cloudstack-keystore-setup[134495]: keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14596

[DaanHoogland]

@blueorangutan test alma9 kvm-alma9

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-14053)

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-14055)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52744 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10723-t14055-kvm-ol8.zip
Smoke tests completed. 146 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[boring-cyborg]

Awesome work, congrats on your first merged pull request!