Merge branch 'development' into jaspal_packageplugin

Author: alexandraBara

.github/workflows/update-plugin-docs.yml

@@ -1,9 +1,10 @@
-# Workflow to run plugin documentation generation then commit the updated changes
+# Workflow to run plugin documentation generation then create a PR with the updated changes
 
 name: Plugin Documentation Generator
 
 permissions:
   contents: write
+  pull-requests: write
 
 on:
   workflow_dispatch:
@@ -15,8 +16,15 @@ jobs:
     runs-on: [ self-hosted ]
     # To disable this workflow, set DISABLE_AUTO_DOCS to 'true' in repository variables
     if: vars.DISABLE_AUTO_DOCS != 'true'
+    env:
+      HOME: /tmp/github-actions-home
 
     steps:
+    - name: Setup HOME directory
+      run: |
+        mkdir -p /tmp/github-actions-home
+        export HOME=/tmp/github-actions-home
+
     - name: Checkout repository
       uses: actions/checkout@v4
       with:
@@ -37,10 +45,18 @@ jobs:
         source venv/bin/activate
         pre-commit run --files docs/PLUGIN_DOC.md || true
 
-    - name: Commit and push changes
-      run: |
-        git config user.name "github-actions[bot]"
-        git config user.email "github-actions[bot]@users.noreply.github.com"
-        git add docs/PLUGIN_DOC.md
-        git diff --staged --quiet || git commit --no-verify -m "docs: Update plugin documentation [automated]"
-        git push
+    - name: Create Pull Request
+      uses: peter-evans/create-pull-request@v6
+      with:
+        token: ${{ secrets.GITHUB_TOKEN }}
+        commit-message: "docs: Update plugin documentation [automated]"
+        committer: "github-actions[bot] <github-actions[bot]@users.noreply.github.com>"
+        author: "github-actions[bot] <github-actions[bot]@users.noreply.github.com>"
+        branch: automated-plugin-docs-update
+        delete-branch: true
+        title: "docs: Update plugin documentation [automated]"
+        body: |
+          Automated plugin documentation update generated by workflow.
+
+          This PR was automatically created by the Plugin Documentation Generator workflow.
+        labels: documentation,automated

docs/PLUGIN_DOC.md

@@ -4,7 +4,7 @@
 
 | Plugin | Collection | Analysis | DataModel | Collector | Analyzer |
 | --- | --- | --- | --- | --- | --- |
-| AmdSmiPlugin | amd-smi firmware --json<br>amd-smi list --json<br>amd-smi partition --json<br>amd-smi process --json<br>amd-smi static -g all --json<br>amd-smi version --json | **Analyzer Args:**<br>- `check_static_data`: bool<br>- `expected_gpu_processes`: Optional[int]<br>- `expected_max_power`: Optional[int]<br>- `expected_driver_version`: Optional[str]<br>- `expected_memory_partition_mode`: Optional[str]<br>- `expected_compute_partition_mode`: Optional[str]<br>- `expected_pldm_version`: Optional[str]<br>- `l0_to_recovery_count_error_threshold`: Optional[int]<br>- `l0_to_recovery_count_warning_threshold`: Optional[int]<br>- `vendorid_ep`: Optional[str]<br>- `vendorid_ep_vf`: Optional[str]<br>- `devid_ep`: Optional[str]<br>- `devid_ep_vf`: Optional[str]<br>- `sku_name`: Optional[str] | [AmdSmiDataModel](#AmdSmiDataModel-Model) | [AmdSmiCollector](#Collector-Class-AmdSmiCollector) | [AmdSmiAnalyzer](#Data-Analyzer-Class-AmdSmiAnalyzer) |
+| AmdSmiPlugin | firmware --json<br>list --json<br>partition --json<br>process --json<br>ras --cper --folder={folder}<br>static -g all --json<br>static -g {gpu_id} --json<br>version --json | **Analyzer Args:**<br>- `check_static_data`: bool<br>- `expected_gpu_processes`: Optional[int]<br>- `expected_max_power`: Optional[int]<br>- `expected_driver_version`: Optional[str]<br>- `expected_memory_partition_mode`: Optional[str]<br>- `expected_compute_partition_mode`: Optional[str]<br>- `expected_pldm_version`: Optional[str]<br>- `l0_to_recovery_count_error_threshold`: Optional[int]<br>- `l0_to_recovery_count_warning_threshold`: Optional[int]<br>- `vendorid_ep`: Optional[str]<br>- `vendorid_ep_vf`: Optional[str]<br>- `devid_ep`: Optional[str]<br>- `devid_ep_vf`: Optional[str]<br>- `sku_name`: Optional[str]<br>- `expected_xgmi_speed`: Optional[list[float]]<br>- `analysis_range_start`: Optional[datetime.datetime]<br>- `analysis_range_end`: Optional[datetime.datetime] | [AmdSmiDataModel](#AmdSmiDataModel-Model) | [AmdSmiCollector](#Collector-Class-AmdSmiCollector) | [AmdSmiAnalyzer](#Data-Analyzer-Class-AmdSmiAnalyzer) |
 | BiosPlugin | sh -c 'cat /sys/devices/virtual/dmi/id/bios_version'<br>wmic bios get SMBIOSBIOSVersion /Value | **Analyzer Args:**<br>- `exp_bios_version`: list[str]<br>- `regex_match`: bool | [BiosDataModel](#BiosDataModel-Model) | [BiosCollector](#Collector-Class-BiosCollector) | [BiosAnalyzer](#Data-Analyzer-Class-BiosAnalyzer) |
 | CmdlinePlugin | cat /proc/cmdline | **Analyzer Args:**<br>- `required_cmdline`: Union[str, list]<br>- `banned_cmdline`: Union[str, list] | [CmdlineDataModel](#CmdlineDataModel-Model) | [CmdlineCollector](#Collector-Class-CmdlineCollector) | [CmdlineAnalyzer](#Data-Analyzer-Class-CmdlineAnalyzer) |
 | DeviceEnumerationPlugin | lscpu \| grep Socket \| awk '{ print $2 }'<br>powershell -Command "(Get-WmiObject -Class Win32_Processor \| Measure-Object).Count"<br>lspci -d {vendorid_ep}: \| grep -i 'VGA\\|Display\\|3D' \| wc -l<br>powershell -Command "(wmic path win32_VideoController get name \| findstr AMD \| Measure-Object).Count"<br>lspci -d {vendorid_ep}: \| grep -i 'Virtual Function' \| wc -l<br>powershell -Command "(Get-VMHostPartitionableGpu \| Measure-Object).Count" | **Analyzer Args:**<br>- `cpu_count`: Optional[list[int]]<br>- `gpu_count`: Optional[list[int]]<br>- `vf_count`: Optional[list[int]] | [DeviceEnumerationDataModel](#DeviceEnumerationDataModel-Model) | [DeviceEnumerationCollector](#Collector-Class-DeviceEnumerationCollector) | [DeviceEnumerationAnalyzer](#Data-Analyzer-Class-DeviceEnumerationAnalyzer) |
@@ -14,7 +14,7 @@
 | JournalPlugin | journalctl --no-pager --system --output=short-iso | - | [JournalData](#JournalData-Model) | [JournalCollector](#Collector-Class-JournalCollector) | - |
 | KernelPlugin | sh -c 'uname -a'<br>wmic os get Version /Value | **Analyzer Args:**<br>- `exp_kernel`: Union[str, list]<br>- `regex_match`: bool | [KernelDataModel](#KernelDataModel-Model) | [KernelCollector](#Collector-Class-KernelCollector) | [KernelAnalyzer](#Data-Analyzer-Class-KernelAnalyzer) |
 | KernelModulePlugin | cat /proc/modules<br>wmic os get Version /Value | **Analyzer Args:**<br>- `kernel_modules`: dict[str, dict]<br>- `regex_filter`: list[str] | [KernelModuleDataModel](#KernelModuleDataModel-Model) | [KernelModuleCollector](#Collector-Class-KernelModuleCollector) | [KernelModuleAnalyzer](#Data-Analyzer-Class-KernelModuleAnalyzer) |
-| MemoryPlugin | free -b<br>wmic OS get FreePhysicalMemory /Value; wmic ComputerSystem get TotalPhysicalMemory /Value | - | [MemoryDataModel](#MemoryDataModel-Model) | [MemoryCollector](#Collector-Class-MemoryCollector) | [MemoryAnalyzer](#Data-Analyzer-Class-MemoryAnalyzer) |
+| MemoryPlugin | free -b<br>/usr/bin/lsmem<br>wmic OS get FreePhysicalMemory /Value; wmic ComputerSystem get TotalPhysicalMemory /Value | - | [MemoryDataModel](#MemoryDataModel-Model) | [MemoryCollector](#Collector-Class-MemoryCollector) | [MemoryAnalyzer](#Data-Analyzer-Class-MemoryAnalyzer) |
 | NvmePlugin | nvme smart-log {dev}<br>nvme error-log {dev} --log-entries=256<br>nvme id-ctrl {dev}<br>nvme id-ns {dev}{ns}<br>nvme fw-log {dev}<br>nvme self-test-log {dev}<br>nvme get-log {dev} --log-id=6 --log-len=512<br>nvme telemetry-log {dev} --output-file={dev}_{f_name} | - | [NvmeDataModel](#NvmeDataModel-Model) | [NvmeCollector](#Collector-Class-NvmeCollector) | - |
 | OsPlugin | sh -c '( lsb_release -ds \|\| (cat /etc/*release \| grep PRETTY_NAME) \|\| uname -om ) 2>/dev/null \| head -n1'<br>cat /etc/*release \| grep VERSION_ID<br>wmic os get Version /value<br>wmic os get Caption /Value | **Analyzer Args:**<br>- `exp_os`: Union[str, list]<br>- `exact_match`: bool | [OsDataModel](#OsDataModel-Model) | [OsCollector](#Collector-Class-OsCollector) | [OsAnalyzer](#Data-Analyzer-Class-OsAnalyzer) |
 | PackagePlugin | dnf list --installed<br>dpkg-query -W<br>pacman -Q<br>cat /etc/*release<br>wmic product get name,version | **Analyzer Args:**<br>- `exp_package_ver`: Dict[str, Optional[str]]<br>- `regex_match`: bool | [PackageDataModel](#PackageDataModel-Model) | [PackageCollector](#Collector-Class-PackageCollector) | [PackageAnalyzer](#Data-Analyzer-Class-PackageAnalyzer) |
@@ -42,25 +42,29 @@ Class for collection of inband tool amd-smi data.
 
 - **AMD_SMI_EXE**: `amd-smi`
 - **SUPPORTED_OS_FAMILY**: `{<OSFamily.LINUX: 3>}`
-- **CMD_VERSION**: `amd-smi version --json`
-- **CMD_LIST**: `amd-smi list --json`
-- **CMD_PROCESS**: `amd-smi process --json`
-- **CMD_PARTITION**: `amd-smi partition --json`
-- **CMD_FIRMWARE**: `amd-smi firmware --json`
-- **CMD_STATIC**: `amd-smi static -g all --json`
+- **CMD_VERSION**: `version --json`
+- **CMD_LIST**: `list --json`
+- **CMD_PROCESS**: `process --json`
+- **CMD_PARTITION**: `partition --json`
+- **CMD_FIRMWARE**: `firmware --json`
+- **CMD_STATIC**: `static -g all --json`
+- **CMD_STATIC_GPU**: `static -g {gpu_id} --json`
+- **CMD_RAS**: `ras --cper --folder={folder}`
 
 ### Provides Data
 
 AmdSmiDataModel
 
 ### Commands
 
-- amd-smi firmware --json
-- amd-smi list --json
-- amd-smi partition --json
-- amd-smi process --json
-- amd-smi static -g all --json
-- amd-smi version --json
+- firmware --json
+- list --json
+- partition --json
+- process --json
+- ras --cper --folder={folder}
+- static -g all --json
+- static -g {gpu_id} --json
+- version --json
 
 ## Collector Class BiosCollector
 
@@ -300,6 +304,7 @@ Collect memory usage details
 
 - **CMD_WINDOWS**: `wmic OS get FreePhysicalMemory /Value; wmic ComputerSystem get TotalPhysicalMemory /Value`
 - **CMD**: `free -b`
+- **CMD_LSMEM**: `/usr/bin/lsmem`
 
 ### Provides Data
 
@@ -308,6 +313,7 @@ MemoryDataModel
 ### Commands
 
 - free -b
+- /usr/bin/lsmem
 - wmic OS get FreePhysicalMemory /Value; wmic ComputerSystem get TotalPhysicalMemory /Value
 
 ## Collector Class NvmeCollector
@@ -646,10 +652,15 @@ Data model for amd-smi data.
 - **gpu_list**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.AmdSmiListItem]]`
 - **partition**: `Optional[nodescraper.plugins.inband.amdsmi.amdsmidata.Partition]`
 - **process**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.Processes]]`
+- **topology**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.Topo]]`
 - **firmware**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.Fw]]`
 - **bad_pages**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.BadPages]]`
 - **static**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.AmdSmiStatic]]`
 - **metric**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.AmdSmiMetric]]`
+- **xgmi_metric**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.XgmiMetrics]]`
+- **xgmi_link**: `Optional[list[nodescraper.plugins.inband.amdsmi.amdsmidata.XgmiLinks]]`
+- **cper_data**: `Optional[list[nodescraper.models.datamodel.FileModel]]`
+- **amdsmitst_data**: `nodescraper.plugins.inband.amdsmi.amdsmidata.AmdSmiTstData`
 
 ## BiosDataModel Model
 
@@ -763,6 +774,7 @@ Data model for journal logs
 
 - **mem_free**: `str`
 - **mem_total**: `str`
+- **lsmem_output**: `Optional[dict]`
 
 ## NvmeDataModel Model
 
@@ -915,7 +927,11 @@ Data model for in band syslog logs
 
 ## Data Analyzer Class AmdSmiAnalyzer
 
-**Bases**: ['DataAnalyzer']
+### Description
+
+Check AMD SMI Application data for PCIe, ECC errors, CPER data, and analyze amdsmitst metrics
+
+**Bases**: ['CperAnalysisTaskMixin', 'DataAnalyzer']
 
 **Link to code**: [amdsmi_analyzer.py](https://github.com/amd/node-scraper/blob/HEAD/nodescraper/plugins/inband/amdsmi/amdsmi_analyzer.py)
 
@@ -1213,6 +1229,9 @@ Check sysctl matches expected sysctl details
 - **devid_ep**: `Optional[str]`
 - **devid_ep_vf**: `Optional[str]`
 - **sku_name**: `Optional[str]`
+- **expected_xgmi_speed**: `Optional[list[float]]`
+- **analysis_range_start**: `Optional[datetime.datetime]`
+- **analysis_range_end**: `Optional[datetime.datetime]`
 
 ## Analyzer Args Class BiosAnalyzerArgs
 

nodescraper/plugins/inband/package/package_analyzer.py

@@ -44,7 +44,7 @@ def regex_version_data(
         package_data: dict[str, str],
         key_search: re.Pattern[str],
         value_search: Optional[Pattern[str]],
-    ) -> bool:
+    ) -> tuple[bool, list[tuple[str, str, str]]]:
         """Searches the package values for the key and value search patterns
 
         Args:
@@ -53,10 +53,12 @@ def regex_version_data(
             value_search (Optional[Pattern[str]]): a compiled regex pattern to search for the package version, if None then any version is accepted
 
         Returns:
-            bool: A boolean indicating if the value was found
+            tuple: (value_found, version_mismatches) where value_found is a bool and
+                   version_mismatches is a list of (package_name, expected_pattern, found_version) tuples
         """
 
         value_found = False
+        version_mismatches = []
         for name, version in package_data.items():
             self.logger.debug("Package data: %s, %s", name, version)
             key_search_res = key_search.search(name)
@@ -66,6 +68,7 @@ def regex_version_data(
                     continue
                 value_search_res = value_search.search(version)
                 if not value_search_res:
+                    version_mismatches.append((name, value_search.pattern, version))
                     self._log_event(
                         EventCategory.APPLICATION,
                         f"Package {key_search.pattern} Version Mismatch, Expected {value_search.pattern} but found {version}",
@@ -77,7 +80,7 @@ def regex_version_data(
                             "found_version": version,
                         },
                     )
-        return value_found
+        return value_found, version_mismatches
 
     def package_regex_search(
         self, package_data: dict[str, str], exp_package_data: dict[str, Optional[str]]
@@ -87,16 +90,23 @@ def package_regex_search(
         Args:
             package_data (dict[str, str]): a dictionary of package names and versions
             exp_package_data (dict[str, Optional[str]]): a dictionary of expected package names and versions
+
+        Returns:
+            tuple: (not_found_keys, regex_errors, version_mismatches) containing lists of errors
         """
         not_found_keys = []
+        regex_errors = []
+        version_mismatches = []
+
         for exp_key, exp_value in exp_package_data.items():
             try:
                 if exp_value is not None:
                     value_search = re.compile(exp_value)
                 else:
                     value_search = None
                 key_search = re.compile(exp_key)
-            except re.error:
+            except re.error as e:
+                regex_errors.append((exp_key, exp_value, str(e)))
                 self._log_event(
                     EventCategory.RUNTIME,
                     f"Regex Compile Error either {exp_key} {exp_value}",
@@ -108,10 +118,13 @@ def package_regex_search(
                 )
                 continue
 
-            key_found = self.regex_version_data(package_data, key_search, value_search)
+            key_found, mismatches = self.regex_version_data(package_data, key_search, value_search)
+
+            # Collect version mismatches
+            version_mismatches.extend(mismatches)
 
             if not key_found:
-                not_found_keys.append(exp_key)
+                not_found_keys.append((exp_key, exp_value))
                 self._log_event(
                     EventCategory.APPLICATION,
                     f"Package {exp_key} not found in the package list",
@@ -123,7 +136,8 @@ def package_regex_search(
                         "found_version": None,
                     },
                 )
-        return not_found_keys
+
+        return not_found_keys, regex_errors, version_mismatches
 
     def package_exact_match(
         self, package_data: dict[str, str], exp_package_data: dict[str, Optional[str]]
@@ -190,9 +204,43 @@ def analyze_data(
             return self.result
 
         if args.regex_match:
-            not_found_keys = self.package_regex_search(data.version_info, args.exp_package_ver)
-            self.result.message = f"Packages not found: {not_found_keys}"
-            self.result.status = ExecutionStatus.ERROR
+            not_found_keys, regex_errors, version_mismatches = self.package_regex_search(
+                data.version_info, args.exp_package_ver
+            )
+
+            # Adding details for err message
+            error_parts = []
+            if not_found_keys:
+                packages_detail = ", ".join(
+                    [
+                        f"'{pkg}' (expected version: {ver if ver else 'any'})"
+                        for pkg, ver in not_found_keys
+                    ]
+                )
+                error_parts.append(f"Packages not found: {packages_detail}")
+
+            if regex_errors:
+                regex_detail = ", ".join(
+                    [f"'{pkg}' pattern (version: {ver})" for pkg, ver, _ in regex_errors]
+                )
+                error_parts.append(f"Regex compile errors: {regex_detail}")
+
+            if version_mismatches:
+                version_detail = ", ".join(
+                    [
+                        f"'{pkg}' (expected: {exp}, found: {found})"
+                        for pkg, exp, found in version_mismatches
+                    ]
+                )
+                error_parts.append(f"Version mismatches: {version_detail}")
+
+            total_errors = len(not_found_keys) + len(regex_errors) + len(version_mismatches)
+            if total_errors > 0:
+                self.result.message = f"{'; '.join(error_parts)}"
+                self.result.status = ExecutionStatus.ERROR
+            else:
+                self.result.message = "All packages found and versions matched"
+                self.result.status = ExecutionStatus.OK
         else:
             self.logger.info("Expected packages: %s", list(args.exp_package_ver.keys()))
             not_found_match, not_found_version = self.package_exact_match(

test/unit/plugin/test_package_analyzer.py

@@ -89,4 +89,22 @@ def test_data_version_regex(package_analyzer, default_data_lib):
         regex_match=True,
     )
     res = package_analyzer.analyze_data(default_data_lib, args=args)
+    assert res.status == ExecutionStatus.OK
+    assert res.message == "All packages found and versions matched"
+
+
+def test_data_multiple_errors_regex(package_analyzer, default_data_lib):
+    """Test that detailed error messages are shown for multiple package errors"""
+    args = PackageAnalyzerArgs(
+        exp_package_ver={
+            "missing-package": None,
+            "test-ubuntu-package\\.x86_64": "2\\.\\d+",
+            "another-missing": "1\\.0",
+        },
+        regex_match=True,
+    )
+    res = package_analyzer.analyze_data(default_data_lib, args=args)
     assert res.status == ExecutionStatus.ERROR
+    assert "missing-package" in res.message
+    assert "another-missing" in res.message
+    assert len(res.events) == 3