Merge pull request #235491 from Lukeout/patch-31

Update agent-windows-troubleshoot.md

Author: tfosmark

articles/azure-monitor/agents/agent-windows-troubleshoot.md

@@ -131,3 +131,365 @@ If the query returns results, you need to determine if a particular data type is
     |8000 |HealthService |This event will specify if a workflow related to performance, event, or other data type collected is unable to forward to the service for ingestion to the workspace. | Event ID 2136 from source HealthService is written together with this event and can indicate the agent is unable to communicate with the service. Possible reasons might be misconfiguration of the proxy and authentication settings, network outage, or the network firewall or proxy doesn't allow TCP traffic from the computer to the service.|
     |10102 and 10103 |Health Service Modules |Workflow couldn't resolve the data source. |This issue can occur if the specified performance counter or instance doesn't exist on the computer or is incorrectly defined in the workspace data settings. If this is a user-specified [performance counter](data-sources-performance-counters.md#configure-performance-counters), verify the information specified follows the correct format and exists on the target computers. |
     |26002 |Health Service Modules |Workflow couldn't resolve the data source. |This issue can occur if the specified Windows event log doesn't exist on the computer. This error can be safely ignored if the computer isn't expected to have this event log registered. Otherwise, if this is a user-specified [event log](data-sources-windows-events.md#configure-windows-event-logs), verify the information specified is correct. |
+    
+## Pinned Certificate Issues with Older Microsoft Monitoring Agents - Breaking Change
+
+*Root CA Change Overview*
+
+As of 30 June 2023, Log Analytics back-end will no longer be accepting connections from MMA that reference an outdate root certificate. These MMAs are older versions prior to the Winter 2020 release (Log Analytics Agent) and prior to SCOM 2019 UR3 (SCOM). Any version, Bundle: 10.20.18053 / Extension: 1.0.18053.0, or greater will not have any issues, as well as any version above SCOM 2019 UR3. Any agent older than that will break and no longer be working and uploading to Log Analytics.
+
+*What exactly is changing?*
+
+As part of an ongoing security effort across various Azure services, Azure Log Analytics will be officially switching from the Baltimore CyberTrust CA Root to the [DigiCert Global G2 CA Root](https://www.digicert.com/kb/digicert-root-certificates.htm). This change will impact TLS communications with Log Analytics if the new DigiCert Global G2 CA Root certificate is missing from the OS, or the application is referencing the old Baltimore Root CA. **What this means is that Log Analytics will no longer accept connections from MMA that use this old root CA after it's retired.**
+
+*Solution products*
+
+You may have received the breaking change notification even if you have not personally installed the Microsoft Monitoring Agent. That is because various Azure products leverage the Microsoft Monitoring Agent. If you’re using one of these products, you may be affected as they leverage the Windows Log Analytics Agent. For those products with links below there may be specific instructions that will require you to upgrade to the latest agent.
+
+-	VM Insights
+-	[System Center Operations Manager (SCOM)](/system-center/scom/deploy-upgrade-agents)
+-	[System Center Service Manager (SCSM)](/system-center/scsm/upgrade-service-manager)
+-	[Microsoft Defender for Server](/microsoft-365/security/defender-endpoint/update-agent-mma-windows)
+-	[Microsoft Defender for Endpoint](/microsoft-365/security/defender-endpoint/update-agent-mma-windows)
+-	Azure Sentinel
+-	[Azure Automation Agent-based Hybrid Worker](../../automation/automation-windows-hrw-install.md#update-log-analytics-agent-to-latest-version)
+-	[Azure Automation Change Tracking and Inventory](../../automation/change-tracking/overview.md?tabs=python-2#update-log-analytics-agent-to-latest-version)
+-	[Azure Automation Update Management](../../automation/update-management/overview.md#update-windows-log-analytics-agent-to-latest-version)
+
+
+*Identifying and Remidiating Breaking Agents*
+
+For deployments with a limited number of agents, we highly recommend you upgrading your agent per node via [these management instructions](https://aka.ms/MMA-Upgrade).
+
+For deployments with multiple nodes, we've written a script that will detect any affected breaking MMAs per subscription and then subsequently upgrade them to the latest version. These scripts need to be run sequentially, starting with UpdateMMA.ps1 then UpgradeMMA.ps1. Depending on the machine, the script may take a while. PowerShell 7 or greater is needed to run to avoid a timeout.
+
+*UpdateMMA.ps1*
+This script will go through VMs in your subscriptions, check for existing MMAs installed and then generate a .csv file of agents that need to be upgraded.
+
+*UpgradeMMA.ps1*
+This script will use the .CSV file generated in UpdateMMA.ps1 to upgrade all breaking MMAs.
+
+Both of these scripts may take a while to complete.
+
+# [UpdateMMA](#tab/UpdateMMA)
+
+```powershell
+# UpdateMMA.ps1
+# This script is to be run per subscription, the customer has to set the az subscription before running this within the terminal scope.
+# This script uses parallel processing, modify the $parallelThrottleLimit parameter to either increase or decrease the number of parallel processes
+# PS> .\UpdateMMA.ps1 GetInventory
+# The above command will generate a csv file with the details of VM's and VMSS that require MMA upgrade. 
+# The customer can modify the the csv by adding/removing rows if needed
+# Update the MMA by running the script again and passing the csv file as parameter as shown below:
+# PS> .\UpdateMMA.ps1 Upgrade
+# If you don't want to check the inventory, then run the script wiht an additional -no-inventory-check
+# PS> .\UpdateMMA.ps1 GetInventory & .\UpdateMMA.ps1 Upgrade
+
+
+# This version of the script requires Powershell version >= 7 in order to improve performance via ForEach-Object -Parallel
+# https://docs.microsoft.com/powershell/scripting/whats-new/migrating-from-windows-powershell-51-to-powershell-7?view=powershell-7.1
+if ($PSVersionTable.PSVersion.Major -lt 7) 
+{
+    Write-Host "This script requires Powershell version 7 or newer to run. Please see https://docs.microsoft.com/powershell/scripting/whats-new/migrating-from-windows-powershell-51-to-powershell-7?view=powershell-7.1."
+    exit 1
+}
+
+$parallelThrottleLimit = 16
+$mmaFixVersion = [version]"10.20.18053.0"
+
+function GetVmsWithMMAInstalled
+{
+    param(
+        $fileName
+    )
+
+    $vmList = az vm list --show-details --query "[?powerState=='VM running'].{ResourceGroup:resourceGroup, VmName:name}" | ConvertFrom-Json
+    
+    if(!$vmList)
+    {
+        Write-Host "Cannot get the VM list, this script can only detect the running VM's"
+        return
+    }
+
+    $vmsCount = $vmList.Length
+    
+    $vmParallelThrottleLimit = $parallelThrottleLimit
+    if ($vmsCount -lt $vmParallelThrottleLimit) 
+    {
+        $vmParallelThrottleLimit = $vmsCount
+    }
+
+    if($vmsCount -eq 1)
+    {
+        $vmGroups += ,($vmList[0])
+    }
+    else
+    {
+        # split the vm's into batches to do parallel processing
+        for ($i = 0; $i -lt $vmsCount; $i += $vmParallelThrottleLimit) 
+        { 
+            $vmGroups += , ($vmList[$i..($i + $vmParallelThrottleLimit - 1)]) 
+        }
+    }
+
+    Write-Host "Detected $vmsCount Vm's running in this subscription."
+    $hash = [hashtable]::Synchronized(@{})
+    $hash.One = 1
+
+    $vmGroups | Foreach-Object -ThrottleLimit $parallelThrottleLimit -Parallel {
+        $len = $using:vmsCount
+        $hash = $using:hash
+        $_ | ForEach-Object {
+            $percent = 100 * $hash.One++ / $len
+            Write-Progress -Activity "Getting VM Inventory" -PercentComplete $percent
+            $vmName = $_.VmName
+            $resourceGroup = $_.ResourceGroup
+            $responseJson = az vm run-command invoke --command-id RunPowerShellScript --name $vmName -g $resourceGroup --scripts '@UpgradeMMA.ps1' --parameters "functionName=GetMMAVersion" --output json | ConvertFrom-Json
+            if($responseJson)
+            {
+                $mmaVersion = $responseJson.Value[0].message
+                if ($mmaVersion) 
+                {
+                    $extensionName = az vm extension list -g $resourceGroup --vm-name $vmName --query "[?name == 'MicrosoftMonitoringAgent'].name" | ConvertFrom-Json
+                    if ($extensionName) 
+                    {
+                        $installType = "Extension"
+                    }
+                    else 
+                    {
+                        $installType = "Installer"
+                    }
+                    $csvObj = New-Object -TypeName PSObject -Property @{
+                        'Name'           = $vmName
+                        'Resource_Group' = $resourceGroup
+                        'Resource_Type'  = "VM"
+                        'Install_Type'   = $installType
+                        'Version'        = $mmaVersion
+                        "Instance_Id"    = ""
+                    }
+                    $csvObj | Export-Csv $using:fileName -Append -Force
+                } 
+            } 
+        }
+    }
+}
+
+function GetVmssWithMMAInstalled
+{
+    param(
+        $fileName
+    )
+
+    # get the vmss list which are successfully provisioned
+    $vmssList = az vmss list --query "[?provisioningState=='Succeeded'].{ResourceGroup:resourceGroup, VmssName:name}" | ConvertFrom-Json   
+
+    $vmssCount = $vmssList.Length
+    Write-Host "Detected $vmssCount Vmss running in this subscription."
+    $hash = [hashtable]::Synchronized(@{})
+    $hash.One = 1
+
+    $vmssList | Foreach-Object -ThrottleLimit $parallelThrottleLimit -Parallel {
+        $len = $using:vmsCount
+        $hash = $using:hash
+        $percent = 100 * $hash.One++ / $len
+        Write-Progress -Activity "Getting VMSS Inventory" -PercentComplete $percent
+        $vmssName = $_.VmssName
+        $resourceGroup = $_.ResourceGroup
+
+        # get running vmss instance ids
+        $vmssInstanceIds = az vmss list-instances --resource-group $resourceGroup --name $vmssName --expand instanceView --query "[?instanceView.statuses[1].displayStatus=='VM running'].instanceId" | ConvertFrom-Json
+        if ($vmssInstanceIds.Length -gt 0) 
+        {
+            $isMMAExtensionInstalled = az vmss extension list -g $resourceGroup --vmss-name $vmssName --query "[?name == 'MicrosoftMonitoringAgent'].name" | ConvertFrom-Json
+            if ($isMMAExtensionInstalled ) 
+            {
+                # check an instance in vmss, if it needs an MMA upgrade. Since the extension is installed at VMSS level, checking for bad version in 1 instance should be fine.
+                $responseJson = az vmss run-command invoke --command-id RunPowerShellScript --name $vmssName -g $resourceGroup --instance-id $vmssInstanceIds[0] --scripts '@UpgradeMMA.ps1' --parameters "functionName=GetMMAVersion" --output json | ConvertFrom-Json
+                $mmaVersion = $responseJson.Value[0].message
+                if ($mmaVersion) 
+                {
+                    $csvObj = New-Object -TypeName PSObject -Property @{
+                        'Name'           = $vmssName
+                        'Resource_Group' = $resourceGroup
+                        'Resource_Type'  = "VMSS"
+                        'Install_Type'   = "Extension"
+                        'Version'        = $mmaVersion
+                        "Instance_Id"    = ""
+                    }
+                    $csvObj | Export-Csv $using:fileName -Append -Force
+                }
+            }
+            else 
+            {
+                foreach ($instanceId in $vmssInstanceIds) 
+                {
+                    $responseJson = az vmss run-command invoke --command-id RunPowerShellScript --name $vmssName -g $resourceGroup --instance-id $instanceId --scripts '@UpgradeMMA.ps1' --parameters "functionName=GetMMAVersion" --output json | ConvertFrom-Json
+                    $mmaVersion = $responseJson.Value[0].message
+                    if ($mmaVersion) 
+                    {
+                        $csvObj = New-Object -TypeName PSObject -Property @{
+                            'Name'           = $vmssName
+                            'Resource_Group' = $resourceGroup
+                            'Resource_Type'  = "VMSS"
+                            'Install_Type'   = "Installer"
+                            'Version'        = $mmaVersion
+                            "Instance_Id"    = $instanceId
+                        }
+                        $csvObj | Export-Csv $using:fileName -Append -Force
+                    }
+                }
+            }
+        }      
+    }
+}
+
+function Upgrade
+{
+    param(
+        $fileName = "MMAInventory.csv"
+    )
+    Import-Csv $fileName | ForEach-Object -ThrottleLimit $parallelThrottleLimit -Parallel {
+        $mmaVersion = [version]$_.Version
+        if($mmaVersion -lt $using:mmaFixVersion)
+        {
+            if ($_.Install_Type -eq "Extension") 
+            {
+                if ($_.Resource_Type -eq "VMSS") 
+                {
+                    # if the extension is installed with a custom name, provide the name using the flag: --extension-instance-name <extension name>
+                    az vmss extension set --name MicrosoftMonitoringAgent --publisher Microsoft.EnterpriseCloud.Monitoring --force-update --vmss-name $_.Name --resource-group $_.Resource_Group --no-wait --output none
+                }
+                else 
+                {
+                    # if the extension is installed with a custom name, provide the name using the flag: --extension-instance-name <extension name>
+                    az vm extension set --name MicrosoftMonitoringAgent --publisher Microsoft.EnterpriseCloud.Monitoring --force-update --vm-name $_.Name --resource-group $_.Resource_Group --no-wait --output none
+                }
+            }
+            else {
+                if ($_.Resource_Type -eq "VMSS") 
+                {
+                    az vmss run-command invoke --command-id RunPowerShellScript --name $_.Name -g $_.Resource_Group --instance-id $_.Instance_Id --scripts '@UpgradeMMA.ps1' --parameters "functionName=UpgradeMMA" --output none
+                }
+                else 
+                {
+                    az vm run-command invoke --command-id RunPowerShellScript --name $_.Name -g $_.Resource_Group --scripts '@UpgradeMMA.ps1' --parameters "functionName=UpgradeMMA" --output none
+                }
+            }
+        }
+    }
+}
+
+function GetInventory
+{
+    param(
+        $fileName = "MMAInventory.csv"
+    )
+
+    # create a new file 
+    New-Item -Name $fileName -ItemType File -Force
+    GetVmsWithMMAInstalled $fileName
+    GetVmssWithMMAInstalled $fileName
+}
+
+switch ($args.Count)
+{
+    0 {
+        Write-Host "The arguments provided are incorrect."
+        Write-Host "To get the Inventory: Run the script as: PS> .\UpdateMMA.ps1 GetInventory"
+        Write-Host "To update MMA from Inventory: Run the script as: PS> .\UpdateMMA.ps1 Upgrade"
+        Write-Host "To do the both steps together: PS> .\UpdateMMA.ps1 GetInventory & .\UpdateMMA.ps1 Upgrade"
+    }
+    1 {
+        $funcname = $args[0]
+        Invoke-Expression "& $funcname"
+    }
+    2 {
+        $funcname = $args[0]
+        $funcargs = $args[1]
+        Invoke-Expression "& $funcname $funcargs"
+    }
+}
+```
+
+# [UpgradeMMA](#tab/UpgradeMMA)
+
+```powershell
+#UpgradeMMA.ps1
+
+param(
+    $functionName
+)
+
+$mmaLatestVersion32bitDownloadUrl = "https://go.microsoft.com/fwlink/?LinkId=828604"
+$mmaLatestVersion64bitDownloadUrl = "https://go.microsoft.com/fwlink/?LinkId=828603"
+$mmaName = 'Microsoft Monitoring Agent'
+$mmaFixVersion = [version]"10.20.18053.0"
+$regPath = 'HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*'
+
+function GetMMAVersion
+{
+    $mmaVersion = (Get-ItemProperty $regPath | Where-Object { $_.DisplayName -eq $mmaName }).DisplayVersion
+    return $mmaVersion
+}
+
+function MMAUpgradeRequirementCheck
+{
+    $mmaVersion = [version](GetMMAVersion)
+    if ($mmaVersion -and ($mmaVersion -lt $mmaFixVersion)) 
+    {
+        return $TRUE
+    }
+    return $FALSE
+}
+
+function GetMMAUpgradeUrl
+{
+    $osArchitecture = (Get-WmiObject Win32_OperatingSystem).OSArchitecture
+    if ($osArchitecture -eq "64-bit") 
+    {
+        $newMMADownloadUrl = $mmaLatestVersion64bitDownloadUrl
+    }
+    else 
+    {
+        $newMMADownloadUrl = $mmaLatestVersion32bitDownloadUrl
+    }
+        
+    return $newMMADownloadUrl
+}
+
+function UpgradeMMA
+{
+    $mmaUpgradeRequired = MMAUpgradeRequirementCheck
+    if ($mmaUpgradeRequired) 
+    {
+        $mmaDownloadUrl = GetMMAUpgradeUrl
+        if ($mmaDownloadUrl) 
+        {
+            $downloadedFile = "MMASetup.exe"
+            # Download mma exe files
+            Invoke-WebRequest "$mmaDownloadUrl" -OutFile $downloadedFile
+            if(Test-Path $PSScriptRoot\MMA)
+            {
+                Remove-Item $PSScriptRoot\MMA -Recurse -Force
+            }
+            # Extract MMA exe file
+            Start-Process -Wait -NoNewWindow -FilePath "$PSScriptRoot\$downloadedFile" -ArgumentList "/c /t:$PSScriptRoot\MMA"
+            # Run Setup.exe
+            Start-Process -Wait -NoNewWindow -FilePath "MMA\Setup.exe" -ArgumentList "/qn /l*v AgentUpgrade.log AcceptEndUserLicenseAgreement=1"
+        }
+    }
+}
+
+if ($functionName -eq "GetMMAVersion") 
+{
+    GetMMAVersion
+}
+elseif ($functionName -eq "UpgradeMMA" ) 
+{
+    UpgradeMMA
+}
+else
+{
+    return "Wrong parameters"
+}
+```