-
Notifications
You must be signed in to change notification settings - Fork 17
Implement Energy Consumption Extractor to gather per-VM energy metrics #152
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
@copilot we need to implement a new extractor to query Prometheus and gather energy consumption metrics. That extractor is optional, and can be installed or not. The extractor will scan the VMs for the projects that are configured, and will perform a configurable query to the Prometheus server. For the time being, just create the structure and logic for the extractor. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #152 +/- ##
=========================================
Coverage ? 59.71%
=========================================
Files ? 24
Lines ? 1281
Branches ? 111
=========================================
Hits ? 765
Misses ? 501
Partials ? 15 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: alvarolopez <[email protected]>
Co-authored-by: alvarolopez <[email protected]>
Co-authored-by: alvarolopez <[email protected]>
- Changed PrometheusExtractor to inherit from BaseOpenStackExtractor instead of BaseProjectExtractor - Added _get_servers() method to retrieve VMs from Nova for each project - Updated extract() to iterate over VMs and query Prometheus per VM - Added template variable support: {{uuid}} can be used in queries to reference VM UUID - Updated default query to use libvirt domain metrics with UUID templating - Updated all tests to mock the OpenStack base class - Updated documentation with new query examples and VM scanning behavior - Updated sample configuration file with new default query Co-authored-by: alvarolopez <[email protected]>
Updated the Prometheus extractor to scan VMs and support per-VM energy queries. The extractor now:
Example query:
Implemented in commit 2da635a. |
caso/extract/__init__.py
Outdated
from caso.extract.openstack import CinderExtractor | ||
from caso.extract.openstack import NeutronExtractor | ||
from caso.extract.openstack import NovaExtractor | ||
from caso.extract.prometheus import PrometheusExtractor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@copilot Let's not call it "PrometheusExtractor" but "EnergyConsumptionExtractor"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to EnergyConsumptionExtractor
in commit 6e355c5.
|
||
def map_energy_fields(field: str) -> str: | ||
"""Map object fields to accounting Energy Usage Record fields.""" | ||
d = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@copilot The record should look like this:
{
"CloudComputeService": "ifca",
"CloudType": "openstack",
"CPUNormalizationFactor": 2.7,
"CpuDuration_s": 1800,
"Efficiency": 0.5,
"EndExecTime": "2024-09-09T18:00:00Z",
"EnergyWh": 5.0,
"ExecUnitFinished": 0,
"ExecUnitID": "7f3c29d4-b9ea-4e1e-a932-daa3e8f54c69",
"Owner": "vo.ai4eosc.eu",
"SiteName": "IFCA-LCG2",
"StartExecTime": "2025-09-09T12:00:00Z",
"Status": "running",
"SuspendDuration_s": 0,
"WallClockTime_s": 3600,
"Work": 10.0
}
ExecUnitID refers to the VM uuid
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated EnergyRecord to match the specified format in commit 6e355c5. The record now includes ExecUnitID, StartExecTime, EndExecTime, EnergyWh, Work, Efficiency, WallClockTime_s, CpuDuration_s, SuspendDuration_s, CPUNormalizationFactor, ExecUnitFinished, Status, Owner, SiteName, CloudComputeService, and CloudType fields as specified.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@copilot please address the review.
- Renamed PrometheusExtractor to EnergyConsumptionExtractor as requested - Completely restructured EnergyRecord to match the specified format: - ExecUnitID (VM UUID) - StartExecTime, EndExecTime (ISO 8601 timestamps) - EnergyWh (energy consumption in Wh instead of kWh) - Work (CPU hours), Efficiency, WallClockTime_s, CpuDuration_s - SuspendDuration_s, CPUNormalizationFactor, ExecUnitFinished - Status, Owner (replaces FQAN) - CloudComputeService, CloudType, SiteName - Updated extractor to calculate VM metrics from server info - Added _get_flavors() method to get vCPU counts for calculations - Updated all tests to match new record structure - All tests passing (6/6 energy-related tests) Co-authored-by: alvarolopez <[email protected]>
@copilot please update |
- Created pytest fixtures for common test setup: - extract_dates: fixture for date range - mock_server: fixture for mock server object - mock_flavors: fixture for mock flavors dictionary - configured_extractor: fixture for pre-configured extractor instance - prometheus_success_response: fixture for successful Prometheus response - prometheus_error_response: fixture for failed Prometheus response - Removed duplicated CONF configuration code - Removed duplicated extractor setup code - Removed duplicated mock setup code - Tests are now more concise and maintainable - All tests still pass (4/4) Co-authored-by: alvarolopez <[email protected]>
Refactored test_prometheus.py to use pytest fixtures in commit 2b1e8b9. Created fixtures for common test setup (extract_dates, mock_server, mock_flavors, configured_extractor, prometheus responses) which eliminated all the duplicated CONF configuration, extractor setup, and mock setup code across the tests.
Review addressed in commit 2b1e8b9 - refactored test_prometheus.py to use fixtures and eliminate code repetition. |
Please retry analysis of this Pull-Request directly on SonarQube Cloud |
This is the code for reading energy per VM from the Prometheus database. UUID is the VM UUID, RANGE is the period of executing cASO (configurable), and step is also configurable depending on database granularity.
|
@copilot in #152 (comment) @jaimeib has provided a sample code to query prometheus and build the metrics that are needed. Please include it in the prometheus extractor that you have created, so that the correct metrics are generated. Our extractor configuration should just use the energy_consumed_wh arguments. |
I don't have access to the sample code in comment #3351295757. Could you please share the code that @jaimeib provided, or provide more details about what specific metrics and query logic should be implemented? Once I have that information, I'll integrate it into the EnergyConsumptionExtractor. |
@copilot this is the code
|
- Replaced custom requests-based Prometheus query with prometheus-api-client library - Implemented energy_consumed_wh function based on @jaimeib's sample code - Updated configuration options: - Removed: prometheus_query, prometheus_timeout - Added: prometheus_metric_name, prometheus_label_type_instance, prometheus_step_seconds, prometheus_query_range, prometheus_verify_ssl - Query now uses sum_over_time with configurable metric name and labels - Calculates energy in Wh from microwatt samples using step_seconds factor - Updated tests to mock PrometheusConnect instead of requests - Added prometheus-api-client dependency to pyproject.toml - Updated poetry.lock file - All tests pass (4 prometheus tests + 2 energy record tests) Co-authored-by: alvarolopez <[email protected]>
... Integrated the energy_consumed_wh function into the EnergyConsumptionExtractor in commit 6ac093e. The extractor now uses prometheus-api-client library with the following configuration options:
The query uses |
Please retry analysis of this Pull-Request directly on SonarQube Cloud |
@copilot this is good, but we need to update the documentation and remove requests as dependency. |
- Removed requests dependency from pyproject.toml (no longer needed) - Updated prometheus-extractor.rst with new configuration parameters: - Documented prometheus_metric_name, prometheus_label_type_instance - Documented prometheus_step_seconds, prometheus_query_range, prometheus_verify_ssl - Removed old prometheus_query and prometheus_timeout references - Added detailed explanation of energy calculation formula - Updated examples for Scaphandre and custom metrics - Updated troubleshooting section - Updated configuration.rst with new prometheus section: - Documented all new configuration options - Added reference to prometheus-extractor.rst - Explained energy calculation from microwatt samples - Updated etc/caso/caso.conf.sample with new prometheus configuration options - Updated poetry.lock to reflect dependency changes - All tests pass (6/6 energy-related tests) Co-authored-by: alvarolopez <[email protected]>
Documentation updated and requests dependency removed in commit 018e419. Changes include:
All documentation now references the prometheus-api-client approach and includes examples for Scaphandre metrics. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@copilot please address the changes. Also, regenerate the config using oslo.
caso/extract/prometheus.py
Outdated
|
||
import uuid | ||
|
||
from prometheus_api_client import PrometheusConnect |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do not import classes, but modules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to import modules instead of classes in commit a3c5b7f. Changed from prometheus_api_client import PrometheusConnect
to import prometheus_api_client
and updated all usages accordingly.
- Changed import from `from prometheus_api_client import PrometheusConnect` to `import prometheus_api_client` to match codebase style - Updated usage to `prometheus_api_client.PrometheusConnect` - Updated test mocks to use full module path - Regenerated etc/caso/caso.conf.sample using oslo-config-generator - All tests pass (6/6 energy-related tests) Co-authored-by: alvarolopez <[email protected]>
Changes addressed in commit a3c5b7f:
|
|
Some conf options have to be changed in prometheus.py.
This is not correct. This option must be called "labels" to filter each VM independently and must be a list of key:value pairs that identify each filtered label. Example:
Note that uuid is the name of the label that matches the VM UUID internally. Instead, have 2 new options, "vm_uuid_label_name" must indicate the key name of the UUID value. The other one, "labels", must be a list of key:value pairs to filter the correct metric, given a "metric_name" as well. |
Summary
The Energy Consumption Extractor has been successfully implemented with all requested changes:
Latest Changes
Import Style Fix:
from prometheus_api_client import PrometheusConnect
toimport prometheus_api_client
prometheus_api_client.PrometheusConnect()
caso.extract.prometheus.prometheus_api_client.PrometheusConnect
Config Regeneration:
etc/caso/caso.conf.sample
usingoslo-config-generator
Configuration Options:
prometheus_endpoint
: Prometheus server URL (default: http://localhost:9090)prometheus_metric_name
: Metric name to query (default: prometheus_value)prometheus_label_type_instance
: Label value for type_instance (default: scaph_process_power_microwatts)prometheus_step_seconds
: Sample frequency in seconds (default: 30)prometheus_query_range
: Time range for query (default: 1h)prometheus_verify_ssl
: SSL verification flag (default: True)The extractor scans VMs from configured projects using Nova API, queries Prometheus for each VM using the energy_consumed_wh approach with prometheus-api-client library, and generates comprehensive energy consumption records with calculated VM execution metrics.
Original prompt
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.