-
Notifications
You must be signed in to change notification settings - Fork 621
Sysinfo: add support for AMD and NVIDIA GPUs #2238
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sarumaj
wants to merge
7
commits into
wavetermdev:main
Choose a base branch
from
sarumaj:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
b204f73
sysinfo: add support for amd and nvidia gpus
sarumaj be0ea10
wshremote: fix unused output
sarumaj 59257cd
refactor and apply suggestions
sarumaj 1aebc5b
get rid of estimations
sarumaj 14cfc5a
review tests
sarumaj 429a806
Merge branch 'main' into main
sarumaj 695f99f
Merge remote-tracking branch 'origin/main' into pr-2238
sawka File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| # GPU Monitoring Support | ||
|
|
||
| This document describes the GPU monitoring functionality added to the sysinfo system. | ||
|
|
||
| ## Overview | ||
|
|
||
| The GPU monitoring feature uses command-line tools to collect GPU metrics across multiple platforms and GPU vendors. Instead of relying on Go modules, it uses `exec.Command` to execute platform-specific GPU monitoring tools. | ||
|
|
||
| ## Supported Platforms and Tools | ||
|
|
||
| ### Linux | ||
| - **NVIDIA GPUs**: Uses `nvidia-smi` command | ||
| - Collects: GPU utilization, memory usage, memory total, temperature | ||
| - Command: `nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader,nounits` | ||
|
|
||
| - **AMD GPUs**: Uses `rocm-smi` command | ||
| - Collects: Memory usage, memory total, temperature | ||
| - Command: `rocm-smi --showproductname --showmeminfo vram --showtemp` | ||
|
|
||
| ### macOS | ||
| - Uses multiple commands for comprehensive GPU monitoring: | ||
| - `system_profiler SPDisplaysDataType` - Gets GPU names and VRAM information | ||
| - `iostat` - Attempts to get GPU utilization (if available) | ||
| - `vm_stat` - Gets memory pressure information | ||
| - `sysctl hw.memsize` - Gets system memory for estimation | ||
| - Collects: GPU names, estimated utilization, memory usage, memory total | ||
| - Automatically detects multiple GPUs (integrated + discrete) | ||
| - Estimates GPU memory based on system memory if VRAM info unavailable | ||
| - Provides more detailed information than basic system_profiler output | ||
|
|
||
| ### Windows | ||
| - Uses PowerShell commands for comprehensive GPU monitoring: | ||
| - `Get-WmiObject -Class Win32_VideoController` - Gets GPU names and memory information | ||
| - `Get-Counter "\GPU Engine(*)\Utilization Percentage"` - Gets GPU utilization | ||
| - `Get-Counter "\GPU Adapter Memory(*)\Dedicated Usage"` - Gets GPU memory usage | ||
| - Collects: GPU names, utilization, memory usage, memory total | ||
| - Automatically detects multiple GPUs (integrated + discrete) | ||
| - Filters out basic/standard display adapters to focus on dedicated GPUs | ||
| - Provides real-time GPU utilization using Windows Performance Counters | ||
|
|
||
| ## Data Structure | ||
|
|
||
| GPU data is collected in the following format: | ||
|
|
||
| ```go | ||
| type GpuData struct { | ||
| Index int `json:"index"` // GPU index | ||
| Util float64 `json:"util"` // GPU utilization percentage | ||
| MemUsed float64 `json:"mem_used"` // Memory used in GB | ||
| MemTotal float64 `json:"mem_total"` // Total memory in GB | ||
| Temp float64 `json:"temp"` // Temperature in Celsius | ||
| } | ||
| ``` | ||
|
|
||
| ## Metrics Collected | ||
|
|
||
| The system collects the following metrics for each GPU: | ||
|
|
||
| - `gpu` - Average GPU utilization across all GPUs | ||
| - `gpu:{index}:util` - GPU utilization for specific GPU | ||
| - `gpu:{index}:mem_used` - Memory used for specific GPU | ||
| - `gpu:{index}:mem_total` - Total memory for specific GPU | ||
| - `gpu:{index}:temp` - Temperature for specific GPU | ||
|
|
||
| ## Frontend Plot Types | ||
|
|
||
| The frontend supports the following GPU-related plot types: | ||
|
|
||
| - **GPU**: Shows average GPU utilization | ||
| - **All GPU**: Shows utilization for all individual GPUs | ||
| - **GPU Memory**: Shows memory usage for all GPUs | ||
| - **CPU + GPU**: Shows both CPU and GPU utilization | ||
|
|
||
| ## Implementation Details | ||
|
|
||
| ### Platform Detection | ||
| The system automatically detects the platform using `uname -s` and selects the appropriate GPU monitoring method. | ||
|
|
||
| ### Tool Availability Detection | ||
| Before attempting to collect GPU data, the system checks if the required tools (`nvidia-smi` or `rocm-smi`) are available on the system. | ||
|
|
||
| ### macOS Improvements | ||
| The macOS implementation has been significantly enhanced to provide more comprehensive GPU monitoring: | ||
|
|
||
| 1. **Multiple GPU Detection**: Parses `system_profiler` output to detect both integrated and discrete GPUs | ||
| 2. **VRAM Information**: Extracts VRAM size from system_profiler output using regex patterns | ||
| 3. **Memory Pressure**: Uses `vm_stat` to calculate memory usage and pressure | ||
| 4. **GPU Utilization**: Attempts to get GPU utilization from `iostat` output | ||
| 5. **Memory Estimation**: Falls back to estimating GPU memory based on system memory if VRAM info is unavailable | ||
| 6. **Error Handling**: Gracefully handles missing commands and parsing errors | ||
|
|
||
| ### Windows Implementation | ||
| The Windows implementation provides comprehensive GPU monitoring using PowerShell: | ||
|
|
||
| 1. **GPU Detection**: Uses `Get-WmiObject -Class Win32_VideoController` to detect all GPUs | ||
| 2. **Memory Information**: Extracts adapter RAM size and converts to GB | ||
| 3. **GPU Utilization**: Uses Windows Performance Counters to get real-time GPU utilization | ||
| 4. **Memory Usage**: Tracks dedicated GPU memory usage using performance counters | ||
| 5. **Multi-GPU Support**: Automatically detects and monitors multiple GPUs | ||
| 6. **Filtering**: Excludes basic/standard display adapters to focus on dedicated GPUs | ||
| 7. **Error Handling**: Gracefully handles PowerShell execution errors and missing counters | ||
|
|
||
| ### Error Handling | ||
| - If no GPU tools are available, the system gracefully continues without GPU data | ||
| - Timeouts are set for all command executions to prevent hanging | ||
| - Parsing errors are handled gracefully | ||
|
|
||
| ### Performance | ||
| - GPU data collection is integrated into the existing sysinfo loop | ||
| - Commands are executed with timeouts to prevent blocking | ||
| - Data is collected every second along with CPU and memory data | ||
|
|
||
| ## Usage | ||
|
|
||
| To use GPU monitoring: | ||
|
|
||
| 1. Ensure you have the appropriate GPU monitoring tools installed: | ||
| - For NVIDIA: Install NVIDIA drivers (includes `nvidia-smi`) | ||
| - For AMD: Install ROCm (includes `rocm-smi`) | ||
|
|
||
| 2. The GPU data will automatically appear in sysinfo blocks when available | ||
|
|
||
| 3. Select GPU plot types from the sysinfo view settings menu | ||
|
|
||
| ## Testing | ||
|
|
||
| Run the tests to verify GPU functionality: | ||
|
|
||
| ```bash | ||
| cd pkg/wshrpc/wshremote | ||
| go test -v | ||
| ``` | ||
|
|
||
| The tests will check: | ||
| - Platform detection | ||
| - Tool availability | ||
| - GPU data collection | ||
|
|
||
| ## Future Enhancements | ||
|
|
||
| - Windows GPU monitoring support | ||
sarumaj marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - More detailed macOS GPU monitoring | ||
| - GPU power consumption metrics | ||
| - GPU fan speed monitoring | ||
| - Support for additional GPU vendors (Intel, etc.) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.