Validate the direct downloaded template file format (QCOW2) if the template file exists #10332

sureshanaparti · 2025-02-06T06:26:33Z

Description

This PR validates the direct downloaded template file format (QCOW2) if the template file exists, otherwise skips validation & logs it.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

…mplate file exists

sureshanaparti · 2025-02-06T06:32:58Z

@blueorangutan package

blueorangutan · 2025-02-06T06:34:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-02-06T06:37:54Z

Codecov Report

Attention: Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.

Project coverage is 15.99%. Comparing base (c5afee2) to head (75eaa5b).
Report is 26 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java	0.00%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #10332      +/-   ##
============================================
- Coverage     16.00%   15.99%   -0.01%     
- Complexity    13062    13064       +2     
============================================
  Files          5644     5644              
  Lines        494915   494920       +5     
  Branches      59960    59962       +2     
============================================
- Hits          79187    79185       -2     
- Misses       406891   406897       +6     
- Partials       8837     8838       +1

Flag	Coverage Δ
uitests	`4.01% <ø> (ø)`
unittests	`16.83% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2025-02-06T07:49:11Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12356

sureshanaparti · 2025-02-06T08:18:09Z

@blueorangutan test

blueorangutan · 2025-02-06T08:20:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-02-07T01:34:50Z

[SF] Trillian test result (tid-12322)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 59569 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12322-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.29	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.37	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.37	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	`Failure`	397.35	test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated	`Error`	15.10	test_vm_life_cycle.py

rohityadavcloud

LGTM

DaanHoogland

clgtm

DaanHoogland · 2025-02-07T09:27:17Z

@blueorangutan test

blueorangutan · 2025-02-07T09:28:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-02-08T02:07:23Z

[SF] Trillian test result (tid-12339)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 57382 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12339-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.29	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.35	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.36	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	`Failure`	382.78	test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated	`Error`	14.88	test_vm_life_cycle.py

DaanHoogland · 2025-02-09T09:09:23Z

[SF] Trillian test result (tid-12339) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 57382 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12339-kvm-ol8.zip Smoke tests completed. 138 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:
Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 2.29 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.35 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.36 test_ipv4_routing.py
test_06_purge_expunged_vm_background_task Failure 382.78 test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated Error 14.88 test_vm_life_cycle.py

These errors seem consistent. I'm not sure wht the state of 4.20 is right now, though

weizhouapache · 2025-02-09T09:56:45Z

[SF] Trillian test result (tid-12339) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 57382 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12339-kvm-ol8.zip Smoke tests completed. 138 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:
Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 2.29 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.35 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.36 test_ipv4_routing.py
test_06_purge_expunged_vm_background_task Failure 382.78 test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated Error 14.88 test_vm_life_cycle.py

These errors seem consistent. I'm not sure wht the state of 4.20 is right now, though

#10252

kiranchavala · 2025-02-10T12:33:57Z

@blueorangutan package

blueorangutan · 2025-02-10T12:36:03Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-10T13:47:21Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12398

kiranchavala · 2025-02-11T04:34:33Z

@blueorangutan test

blueorangutan · 2025-02-11T04:36:03Z

@kiranchavala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

kiranchavala

LGTM, was able to test it with a debugger attached

Register a template as direct download
Deploy a vm from the Template
Get the details of the template local path from the database

mysql > select * from template_spool_ref where template_id=<>;

*************************** 3. row ***************************
               id: 3
          pool_id: 1
      template_id: 203
          created: 2025-02-11 08:12:07
     last_updated: NULL
           job_id: NULL
     download_pct: 100
   download_state: DOWNLOADED
        error_str: NULL
       local_path: 53a5bace-b791-4e83-9d30-67848b0df3e5
     install_path: 53a5bace-b791-4e83-9d30-67848b0df3e5
    template_size: 2147418112
    marked_for_gc: 0
            state: Ready
     update_count: 0
          updated: NULL
deployment_option: NULL

Login to kvm host and delete the template

rm -rf /mnt/d94e3a76-d728-320c-8a27-b0ee6e03757b/53a5bace-b791-4e83-9d30-67848b0df3e5

During vm deployment cloudstack checks if the qcow2 file exists or not and fails the operation if the

Agent Logs

2025-02-11 08:12:07,293 WARN  [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-5:[]) (logid:) Skipped validation whether downloaded file is QCOW2 for template 53a5bace-b791-4e83-9d30-67848b0df3e5, due to downloaded template path is not valid: /mnt/d94e3a76-d728-320c-8a27-b0ee6e03757b/53a5bace-b791-4e83-9d30-67848b0df3e5

2025-02-11 08:17:12,441 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-1:[]) (logid:) Failed to create volume: com.cloud.utils.exception.CloudRuntimeException: Can't find volume:53a5bace-b791-4e83-9d30-67848b0df3e5
	at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.cloneVolumeFromBaseTemplate(KVMStorageProcessor.java:454)
2025-02-11 08:22:22,237 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:) Failed to create volume: com.cloud.utils.exception.CloudRuntimeException: Can't find volume:53a5bace-b791-4e83-9d30-67848b0df3e5

blueorangutan · 2025-02-11T22:56:11Z

[SF] Trillian test result (tid-12369)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 62950 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12369-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.25	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.37	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.37	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	`Failure`	388.54	test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated	`Error`	13.09	test_vm_life_cycle.py

sureshanaparti · 2025-02-12T19:13:55Z

@blueorangutan test

blueorangutan · 2025-02-12T19:16:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti · 2025-02-13T05:02:24Z

[SF] Trillian test result (tid-12369) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 62950 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12369-kvm-ol8.zip Smoke tests completed. 138 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 2.25 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 2.37 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 2.37 test_ipv4_routing.py
test_06_purge_expunged_vm_background_task Failure 388.54 test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated Error 13.09 test_vm_life_cycle.py

these failures are not related to this PR changes, same results in 4.20 health check as well here: #10006 (comment) (seems to be consistent, as per the above results cc @DaanHoogland @weizhouapache @kiranchavala)

rohityadavcloud · 2025-02-13T06:38:53Z

LGTM, thanks for the PR and testing @sureshanaparti @kiranchavala

blueorangutan · 2025-02-13T12:42:59Z

[SF] Trillian test result (tid-12379)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 59879 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10332-t12379-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.29	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.39	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	3.39	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	`Failure`	397.65	test_purge_expunged_vms.py
test_12_start_vm_multiple_volumes_allocated	`Error`	15.17	test_vm_life_cycle.py

…mplate file exists (apache#10332) * Validate the direct downloaded template file format (QCOW2) if the template file exists * string format not required

Validate the direct downloaded template file format (QCOW2) if the te…

5af5f64

…mplate file exists

boring-cyborg bot added the component:kvm label Feb 6, 2025

string format not required

75eaa5b

apache deleted a comment from blueorangutan Feb 6, 2025

sureshanaparti added this to the 4.20.1 milestone Feb 6, 2025

rohityadavcloud requested review from DaanHoogland and Pearl1594 February 7, 2025 09:06

rohityadavcloud approved these changes Feb 7, 2025

View reviewed changes

rohityadavcloud requested a review from borisstoyanov February 7, 2025 09:07

DaanHoogland approved these changes Feb 7, 2025

View reviewed changes

kiranchavala approved these changes Feb 11, 2025

View reviewed changes

sureshanaparti marked this pull request as ready for review February 12, 2025 09:28

rohityadavcloud merged commit 8c4a085 into apache:4.20 Feb 13, 2025
25 of 26 checks passed

rohityadavcloud deleted the direct_downloaded_template_qcow2_validation branch February 13, 2025 06:39

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

Validate the direct downloaded template file format (QCOW2) if the template file exists #10332

Validate the direct downloaded template file format (QCOW2) if the template file exists #10332

Uh oh!

Conversation

sureshanaparti commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Feb 6, 2025

Uh oh!

blueorangutan commented Feb 6, 2025

Uh oh!

codecov bot commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Feb 6, 2025

Uh oh!

sureshanaparti commented Feb 6, 2025

Uh oh!

blueorangutan commented Feb 6, 2025

Uh oh!

blueorangutan commented Feb 7, 2025

Uh oh!

rohityadavcloud left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Feb 7, 2025

Uh oh!

blueorangutan commented Feb 7, 2025

Uh oh!

blueorangutan commented Feb 8, 2025

Uh oh!

DaanHoogland commented Feb 9, 2025

Uh oh!

weizhouapache commented Feb 9, 2025

Uh oh!

kiranchavala commented Feb 10, 2025

Uh oh!

blueorangutan commented Feb 10, 2025

Uh oh!

blueorangutan commented Feb 10, 2025

Uh oh!

kiranchavala commented Feb 11, 2025

Uh oh!

blueorangutan commented Feb 11, 2025

Uh oh!

kiranchavala left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Feb 11, 2025

Uh oh!

sureshanaparti commented Feb 12, 2025

Uh oh!

blueorangutan commented Feb 12, 2025

Uh oh!

sureshanaparti commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rohityadavcloud commented Feb 13, 2025

Uh oh!

Uh oh!

blueorangutan commented Feb 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

sureshanaparti commented Feb 6, 2025 •

edited

Loading

codecov bot commented Feb 6, 2025 •

edited

Loading

sureshanaparti commented Feb 13, 2025 •

edited

Loading