Improve logging to include more identifiable information #9873

vishesh92 · 2024-10-30T11:22:02Z

Description

Github generated description

This pull request introduces several changes across multiple files to enhance logging, add UUID and name properties, and improve the toString methods for better debugging and clarity. Here are the most important changes:

Enhancements to `Agent` class:

Added uuid and name properties to the Agent class and updated the constructor to initialize these properties from the shell. [1] [2]
Updated logging statements to include uuid and name in various methods such as processStartupAnswer and processReadyCommand. [1] [2] [3]
Added getter and setter methods for uuid and name, including persistence logic.

Updates to `LoadBalancerTO` and related classes:

Added uuid property to CounterTO, ConditionTO, and AutoScalePolicyTO classes, along with corresponding getter methods. [1] [2] [3] [4] [5] [6]

Improvements to logging and exception handling:

Enhanced logging and exception messages in UpdateHostCmd and CreateSnapshotFromVMSnapshotCmd to include more detailed information. [1] [2] [3] [4]
Updated RemoveVpnUserCmd to use the Account object directly in method calls and logging.

Method signature changes:

Changed method signatures in Ipv6Service, RemoteAccessVpnService, and GlobalLoadBalancingRulesService to use more appropriate parameter types. [1] [2] [3]

Improved `toString` methods:

Added or enhanced toString methods in NetworkProfile, NicProfile, and UnmanagedInstanceTO classes for better debugging output. [1] [2] [3]

This PR updates the logging for some modules to ensure that we log more details to easily identify resources from the logs.
e.g. We use id at a lot of places in the logs and it makes it difficult to know about the resource without querying the database. This PR makes changes to include uuid or any other relevant fields to the objects in the logs so that the user can easily identify the resource.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2024-10-30T11:29:30Z

Codecov Report

Attention: Patch coverage is 7.09970% with 1845 lines in your changes missing coverage. Please review.

Project coverage is 16.04%. Comparing base (32af4a2) to head (8af0b13).
Report is 8 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...tack/engine/orchestration/NetworkOrchestrator.java	6.50%	114 Missing and 1 partial ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	3.19%	91 Missing ⚠️
...e/cloudstack/storage/volume/VolumeServiceImpl.java	7.40%	75 Missing ⚠️
...atastore/driver/ScaleIOPrimaryDataStoreDriver.java	13.51%	63 Missing and 1 partial ⚠️
...cloud/agent/manager/ClusteredAgentManagerImpl.java	0.00%	62 Missing ⚠️
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java	1.96%	49 Missing and 1 partial ⚠️
...bernetes/cluster/KubernetesClusterManagerImpl.java	4.87%	39 Missing ⚠️
...va/com/cloud/network/element/CiscoVnmcElement.java	2.56%	38 Missing ⚠️
.../cloudstack/storage/image/TemplateServiceImpl.java	2.63%	37 Missing ⚠️
...com/cloud/vm/VirtualMachinePowerStateSyncImpl.java	0.00%	35 Missing ⚠️
... and 237 more

Additional details and impacted files

@@            Coverage Diff             @@
##               4.20    #9873    +/-   ##
==========================================
  Coverage     16.04%   16.04%            
- Complexity    12835    12845    +10     
==========================================
  Files          5637     5639     +2     
  Lines        493651   493898   +247     
  Branches      59874    59883     +9     
==========================================
+ Hits          79205    79260    +55     
- Misses       405648   405832   +184     
- Partials       8798     8806     +8

Flag	Coverage Δ
uitests	`4.02% <ø> (ø)`
unittests	`16.89% <7.09%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

vishesh92 · 2024-11-05T12:17:44Z

@blueorangutan package

blueorangutan · 2024-11-05T12:20:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-05T13:43:12Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11508

vishesh92 · 2024-11-13T09:25:11Z

@blueorangutan package

blueorangutan · 2024-11-13T09:26:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-13T10:52:13Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11556

sureshanaparti · 2024-11-14T10:24:18Z

@blueorangutan package

blueorangutan · 2024-11-14T10:26:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

...eio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManagerImpl.java

blueorangutan · 2024-11-14T11:30:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11564

vishesh92 · 2024-11-14T12:53:23Z

@blueorangutan package

blueorangutan · 2024-11-14T12:54:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

.../apache/cloudstack/network/opendaylight/agent/OpenDaylightControllerResourceManagerImpl.java

...n/java/org/apache/cloudstack/storage/datastore/lifecycle/AdaptiveDataStoreLifeCycleImpl.java

server/src/main/java/com/cloud/ha/AbstractInvestigatorImpl.java

vishesh92 · 2025-01-02T08:35:42Z

@blueorangutan package

blueorangutan · 2025-01-02T08:38:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

vishesh92 · 2025-01-02T08:46:57Z

you are going to hate me @vishesh92 , but can we use capitals for acronyms please? I marked it in a few places but it is all over the place. It is more in line with other discussions on the same subject (see amongst others #9132) otherwise please involve yourself in the discussion on the topic.

@DaanHoogland Making this change in this PR would be difficult due to the size of the changes. It would be better to tackle this separately once we have reached a consensus in #9132.

Also, I have raised this PR against 4.20. There might be some merge conflicts while forward merging 4.19 to 4.20 but those were going to come anyway due to the logger variable rename in 4.20 branch.

blueorangutan · 2025-01-02T09:42:22Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11970

vishesh92 · 2025-01-02T10:52:45Z

@blueorangutan test

blueorangutan · 2025-01-02T10:54:04Z

@vishesh92 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-01-03T03:52:58Z

[SF] Trillian test result (tid-12028)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58558 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9873-t12028-kvm-ol8.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_10_bgp_peers	`Error`	0.07	test_ipv4_routing.py
test_11_isolated_network_with_dynamic_routed_mode	`Error`	0.06	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	1.07	test_ipv4_routing.py
test_create_pvlan_network	`Error`	0.09	test_pvlan.py
test_02_redundant_VPC_default_routes	`Failure`	427.72	test_vpc_redundant.py

kiranchavala

LGTM , the logs contains the UUID of the resources during various operations

Test Case Execution	Result
Smoketests	Pass
Test powerflex setup functionality	Pass
Verify changes are applicable for agent logs	Pass
Verify changes are applicable for management logs	Pass
Display uuid or name of entity for debugging purposes	Pass

For example

Before

2025-01-03 09:41:18,961 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:null) (logid:) vm id: 3 - time since last state update(21960ms) has not passed graceful period yet

After

2025-01-03 08:17:42,909 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) vm: VM instance {"id":6,"instanceName":"i-2-6-VM","type":"User","uuid":"ef5cf0d8-b63b-411c-a88f-94e57a1b6f97"} - time since last state update(50906 ms) has not passed graceful period yet

kiranchavala · 2025-01-03T10:56:21Z

@blueorangutan test keepEnv

blueorangutan · 2025-01-03T10:58:04Z

@kiranchavala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-01-04T02:34:44Z

[SF] Trillian test result (tid-12036)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53744 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9873-t12036-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.29	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.38	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.38	test_ipv4_routing.py

harikrishna-patnala

LGTM

* 4.20: merge errors fixed Restrict the migration of volumes attached to VMs in Starting state (#9725) server, plugin: enhance storage stats for IOPS (#10034) Introducing granular command timeouts global setting (#9659) Improve logging to include more identifiable information (#9873)

* Improve logging to include more identifiable information for kvm plugin * Update logging for scaleio plugin * Improve logging to include more identifiable information for default volume storage plugin * Improve logging to include more identifiable information for agent managers * Improve logging to include more identifiable information for Listeners * Replace ids with objects or uuids * Improve logging to include more identifiable information for engine * Improve logging to include more identifiable information for server * Fixups in engine * Improve logging to include more identifiable information for plugins * Improve logging to include more identifiable information for Cmd classes * Fix toString method for StorageFilterTO.java

PR apache#9873 changed the default implemention of hostConnect on the DefaultHostListener, and Linstor tried to call the none existant default implementation. With the change in the PR a recursion was triggered that triggered a StackOverflow and stopped adding Linstor primary storage.

PR apache#9873 changed the default implementation of hostConnect on the DefaultHostListener, and Linstor tried to call the none existent default implementation. With the change in the PR a recursion was triggered that triggered a StackOverflow and stopped adding Linstor primary storage.

boring-cyborg bot added component:agent component:api component:compute component:networking component:orchestration component:storage component:vpc labels Oct 30, 2024

vishesh92 force-pushed the improve-logging branch 2 times, most recently from 355b1e5 to 6a0dc33 Compare November 5, 2024 05:35

apache deleted a comment from blueorangutan Nov 5, 2024

vishesh92 force-pushed the improve-logging branch from 6a0dc33 to 5672f7f Compare November 5, 2024 12:17

apache deleted a comment from blueorangutan Nov 5, 2024

vishesh92 force-pushed the improve-logging branch from a0e0e89 to c84082e Compare November 13, 2024 10:18

sureshanaparti reviewed Nov 14, 2024

View reviewed changes

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Nov 14, 2024

View reviewed changes

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Nov 14, 2024

View reviewed changes

...eio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManagerImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Jan 1, 2025

View reviewed changes

.../apache/cloudstack/network/opendaylight/agent/OpenDaylightControllerResourceManagerImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Jan 1, 2025

View reviewed changes

...n/java/org/apache/cloudstack/storage/datastore/lifecycle/AdaptiveDataStoreLifeCycleImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Jan 1, 2025

View reviewed changes

server/src/main/java/com/cloud/ha/AbstractInvestigatorImpl.java Outdated Show resolved Hide resolved

Address comments

f0ee884

Fix toString method for StorageFilterTO.java

8af0b13

kiranchavala approved these changes Jan 3, 2025

View reviewed changes

vishesh92 marked this pull request as ready for review January 6, 2025 11:08

harikrishna-patnala approved these changes Jan 6, 2025

View reviewed changes

harikrishna-patnala merged commit a4224e5 into apache:4.20 Jan 6, 2025
25 of 26 checks passed

harikrishna-patnala deleted the improve-logging branch January 6, 2025 11:12

Dajeong-Park mentioned this pull request Jan 17, 2025

[Mold] Glue 이미지를 이용한 가상머신 생성 오류로 스택 소스 임시 주석처리 ablecloud-team/ablestack-cloud#590

Merged

12 tasks

winterhazel mentioned this pull request Feb 2, 2025

Refactor alert email generation method #8831

Merged

13 tasks

winterhazel mentioned this pull request Mar 11, 2025

Fix secondary storage selectors feature #10546

Merged

12 tasks

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

rp- mentioned this pull request May 16, 2025

linstor: fix host connect recursion regression #10878

Merged

14 tasks

Improve logging to include more identifiable information #9873

Improve logging to include more identifiable information #9873

Uh oh!

Conversation

vishesh92 commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Enhancements to Agent class:

Updates to LoadBalancerTO and related classes:

Improvements to logging and exception handling:

Method signature changes:

Improved toString methods:

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vishesh92 commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

vishesh92 commented Nov 13, 2024

Uh oh!

blueorangutan commented Nov 13, 2024

Uh oh!

blueorangutan commented Nov 13, 2024

Uh oh!

sureshanaparti commented Nov 14, 2024

Uh oh!

blueorangutan commented Nov 14, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Nov 14, 2024

Uh oh!

vishesh92 commented Nov 14, 2024

Uh oh!

blueorangutan commented Nov 14, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vishesh92 commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

vishesh92 commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

vishesh92 commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 3, 2025

Uh oh!

kiranchavala left a comment

Choose a reason for hiding this comment

Uh oh!

kiranchavala commented Jan 3, 2025

Uh oh!

blueorangutan commented Jan 3, 2025

Uh oh!

blueorangutan commented Jan 4, 2025

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

vishesh92 commented Oct 30, 2024 •

edited

Loading

Enhancements to `Agent` class:

Updates to `LoadBalancerTO` and related classes:

Improved `toString` methods:

codecov bot commented Oct 30, 2024 •

edited

Loading