Skip to content

Commit bb53c49

Browse files
akselleirvakselleirv
committed
feat(prometheus.exporter.azure): expose concurrency settings
1 parent 5f9aeec commit bb53c49

File tree

6 files changed

+164
-75
lines changed

6 files changed

+164
-75
lines changed

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ Main (unreleased)
2424

2525
- Added `send_traceparent` option for `tracing` config to enable traceparent header propagation. (@MyDigitalLife)
2626

27+
- `prometheus.exporter.azure` now supports `concurrency_subscription` and `concurrency_subscription_resource` arguments to control the number of concurrent API requests. (@akselleirv)
28+
2729
### Bugfixes
2830

2931

‎docs/sources/reference/components/prometheus/prometheus.exporter.azure.md‎

Lines changed: 28 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -66,23 +66,26 @@ prometheus.exporter.azure "<LABEL>" {
6666

6767
You can use the following arguments with `prometheus.exporter.azure`:
6868

69-
| Name | Type | Description | Default | Required |
70-
|-------------------------------| -------------- |----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------| -------- |
71-
| `metrics` | `list(string)` | The metrics to scrape from resources. | | yes |
72-
| `resource_type` | `string` | The Azure Resource Type to scrape metrics for. | | yes |
73-
| `subscriptions` | `list(string)` | List of subscriptions to scrape metrics from. | | yes |
74-
| `azure_cloud_environment` | `string` | Name of the cloud environment to connect to. | `"azurecloud"` | no |
75-
| `included_dimensions` | `list(string)` | List of dimensions to include on the final metrics. | | no |
76-
| `included_resource_tags` | `list(string)` | List of resource tags to include on the final metrics. | `["owner"]` | no |
77-
| `metric_aggregations` | `list(string)` | Aggregations to apply for the metrics produced. | | no |
78-
| `metric_help_template` | `string` | Description of the metric. | `"Azure metric {metric} for {type} with aggregation {aggregation} as {unit}"` | no |
79-
| `metric_name_template` | `string` | Metric template used to expose the metrics. | `"azure_{type}_{metric}_{aggregation}_{unit}"` | no |
80-
| `metric_namespace` | `string` | Namespace for `resource_type` which have multiple levels of metrics. | | no |
81-
| `regions` | `list(string)` | The list of regions for gathering metrics. Gathers metrics for all resources in the subscription. Can't be used if `resource_graph_query_filter` is set. | | no |
82-
| `resource_graph_query_filter` | `string` | The [Kusto query][] filter to apply when searching for resources. Can't be used if `regions` is set. | | no |
83-
| `timespan` | `string` | [ISO8601 Duration][] over which the metrics are being queried. | `"PT5M"` (5 minutes) | no |
84-
| `interval` | `string` | [ISO8601 Duration][] used when to generate individual datapoints in Azure Monitor. Must be smaller than `timespan`. | `"PT1M"` (1 minute) | no |
85-
| `validate_dimensions` | `bool` | Enable dimension validation in the azure SDK. | `false` | no |
69+
| Name | Type | Description | Default | Required |
70+
| ----------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------- |
71+
| `metrics` | `list(string)` | The metrics to scrape from resources. | | yes |
72+
| `resource_type` | `string` | The Azure Resource Type to scrape metrics for. | | yes |
73+
| `subscriptions` | `list(string)` | List of subscriptions to scrape metrics from. | | yes |
74+
| `azure_cloud_environment` | `string` | Name of the cloud environment to connect to. | `"azurecloud"` | no |
75+
| `included_dimensions` | `list(string)` | List of dimensions to include on the final metrics. | | no |
76+
| `included_resource_tags` | `list(string)` | List of resource tags to include on the final metrics. | `["owner"]` | no |
77+
| `metric_aggregations` | `list(string)` | Aggregations to apply for the metrics produced. | | no |
78+
| `metric_help_template` | `string` | Description of the metric. | `"Azure metric {metric} for {type} with aggregation {aggregation} as {unit}"` | no |
79+
| `metric_name_template` | `string` | Metric template used to expose the metrics. | `"azure_{type}_{metric}_{aggregation}_{unit}"` | no |
80+
| `metric_namespace` | `string` | Namespace for `resource_type` which have multiple levels of metrics. | | no |
81+
| `regions` | `list(string)` | The list of regions for gathering metrics. Gathers metrics for all resources in the subscription. Can't be used if `resource_graph_query_filter` is set. | | no |
82+
| `resource_graph_query_filter` | `string` | The [Kusto query][] filter to apply when searching for resources. Can't be used if `regions` is set. | | no |
83+
| `timespan` | `string` | [ISO8601 Duration][] over which the metrics are being queried. | `"PT5M"` (5 minutes) | no |
84+
| `interval` | `string` | [ISO8601 Duration][] used when to generate individual datapoints in Azure Monitor. Must be smaller than `timespan`. | `"PT1M"` (1 minute) | no |
85+
| `validate_dimensions` | `bool` | Enable dimension validation in the azure SDK. | `false` | no |
86+
| `concurrency_subscription` | `int` | Number of subscriptions that can concurrently send metric requests. | `5` | no |
87+
| `concurrency_subscription_resource` | `int` | Number of concurrent metric requests per resource within a subscription. | `10` | no |
88+
| `enable_caching` | `bool` | Enable internal caching to reduce redundant API calls. | `false` | no |
8689

8790
The list of available `resource_type` values and their corresponding `metrics` can be found in [Azure Monitor essentials][].
8891

@@ -109,10 +112,15 @@ Valid values for `azure_cloud_environment` are `azurecloud`, `azurechinacloud`,
109112
`validate_dimensions` is disabled by default to reduce the number of Azure exporter instances required when a `resource_type` has metrics with varying dimensions.
110113
When `validate_dimensions` is enabled you will need one exporter instance per metric + dimension combination which is more tedious to maintain.
111114

112-
`timespan` and `interval` are used to control how metrics are queried from Azure Monitor.
113-
The exporter queries metrics over the `timespan` and returns the most recent datapoint at the specified `interval`.
115+
`timespan` and `interval` are used to control how metrics are queried from Azure Monitor.
116+
The exporter queries metrics over the `timespan` and returns the most recent datapoint at the specified `interval`.
114117
If you are having issues with missing metrics, try increasing the `timespan` to a larger value, such as `PT10M` for 10 minutes, or `PT15M` for 15 minutes.
115118

119+
The concurrency settings control how many Azure API requests can be made in parallel.
120+
`concurrency_subscription` limits the number of subscriptions that can concurrently send metric requests, while `concurrency_subscription_resource` limits the number of concurrent metric requests per resource within a subscription.
121+
You can adjust these values to tune performance based on your Azure subscription limits and available resources.
122+
`enable_caching` enables internal caching to reduce redundant API calls and improve performance.
123+
116124
[Kusto query]: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/
117125
[Azure Monitor essentials]: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported
118126
[ISO8601 Duration]: https://en.wikipedia.org/wiki/ISO_8601#Durations
@@ -198,7 +206,7 @@ Replace the following:
198206

199207
`prometheus.exporter.azure` has exports that can be consumed by the following components:
200208

201-
- Components that consume [Targets](../../../compatibility/#targets-consumers)
209+
* Components that consume [Targets](../../../compatibility/#targets-consumers)
202210

203211
{{< admonition type="note" >}}
204212
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.

‎internal/component/prometheus/exporter/azure/azure.go‎

Lines changed: 40 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -26,21 +26,24 @@ func createExporter(opts component.Options, args component.Arguments) (integrati
2626
}
2727

2828
type Arguments struct {
29-
Subscriptions []string `alloy:"subscriptions,attr"`
30-
ResourceGraphQueryFilter string `alloy:"resource_graph_query_filter,attr,optional"`
31-
ResourceType string `alloy:"resource_type,attr"`
32-
Metrics []string `alloy:"metrics,attr"`
33-
MetricAggregations []string `alloy:"metric_aggregations,attr,optional"`
34-
Timespan string `alloy:"timespan,attr,optional"`
35-
Interval string `alloy:"interval,attr,optional"`
36-
IncludedDimensions []string `alloy:"included_dimensions,attr,optional"`
37-
IncludedResourceTags []string `alloy:"included_resource_tags,attr,optional"`
38-
MetricNamespace string `alloy:"metric_namespace,attr,optional"`
39-
MetricNameTemplate string `alloy:"metric_name_template,attr,optional"`
40-
MetricHelpTemplate string `alloy:"metric_help_template,attr,optional"`
41-
AzureCloudEnvironment string `alloy:"azure_cloud_environment,attr,optional"`
42-
ValidateDimensions bool `alloy:"validate_dimensions,attr,optional"`
43-
Regions []string `alloy:"regions,attr,optional"`
29+
Subscriptions []string `alloy:"subscriptions,attr"`
30+
ResourceGraphQueryFilter string `alloy:"resource_graph_query_filter,attr,optional"`
31+
ResourceType string `alloy:"resource_type,attr"`
32+
Metrics []string `alloy:"metrics,attr"`
33+
MetricAggregations []string `alloy:"metric_aggregations,attr,optional"`
34+
Timespan string `alloy:"timespan,attr,optional"`
35+
Interval string `alloy:"interval,attr,optional"`
36+
IncludedDimensions []string `alloy:"included_dimensions,attr,optional"`
37+
IncludedResourceTags []string `alloy:"included_resource_tags,attr,optional"`
38+
MetricNamespace string `alloy:"metric_namespace,attr,optional"`
39+
MetricNameTemplate string `alloy:"metric_name_template,attr,optional"`
40+
MetricHelpTemplate string `alloy:"metric_help_template,attr,optional"`
41+
AzureCloudEnvironment string `alloy:"azure_cloud_environment,attr,optional"`
42+
ValidateDimensions bool `alloy:"validate_dimensions,attr,optional"`
43+
Regions []string `alloy:"regions,attr,optional"`
44+
ConcurrencySubscription int `alloy:"concurrency_subscription,attr,optional"`
45+
ConcurrencySubscriptionResource int `alloy:"concurrency_subscription_resource,attr,optional"`
46+
EnableCaching bool `alloy:"enable_caching,attr,optional"`
4447
}
4548

4649
// SetToDefault implements syntax.Defaulter.
@@ -56,6 +59,10 @@ func (a *Arguments) SetToDefault() {
5659
// to fully monitor a service which is tedious. Turning off validation eliminates this complexity. The underlying
5760
// sdk will only give back the dimensions which are valid for particular metrics.
5861
ValidateDimensions: false,
62+
// Concurrency default values taken from OSS exporter
63+
ConcurrencySubscription: 5,
64+
ConcurrencySubscriptionResource: 10,
65+
EnableCaching: false,
5966
}
6067
}
6168

@@ -69,20 +76,23 @@ func (a *Arguments) Validate() error {
6976

7077
func (a *Arguments) Convert() *azure_exporter.Config {
7178
return &azure_exporter.Config{
72-
Subscriptions: a.Subscriptions,
73-
ResourceGraphQueryFilter: a.ResourceGraphQueryFilter,
74-
ResourceType: a.ResourceType,
75-
Metrics: a.Metrics,
76-
MetricAggregations: a.MetricAggregations,
77-
Timespan: a.Timespan,
78-
Interval: a.Interval,
79-
IncludedDimensions: a.IncludedDimensions,
80-
IncludedResourceTags: a.IncludedResourceTags,
81-
MetricNamespace: a.MetricNamespace,
82-
MetricNameTemplate: a.MetricNameTemplate,
83-
MetricHelpTemplate: a.MetricHelpTemplate,
84-
AzureCloudEnvironment: a.AzureCloudEnvironment,
85-
ValidateDimensions: a.ValidateDimensions,
86-
Regions: a.Regions,
79+
Subscriptions: a.Subscriptions,
80+
ResourceGraphQueryFilter: a.ResourceGraphQueryFilter,
81+
ResourceType: a.ResourceType,
82+
Metrics: a.Metrics,
83+
MetricAggregations: a.MetricAggregations,
84+
Timespan: a.Timespan,
85+
Interval: a.Interval,
86+
IncludedDimensions: a.IncludedDimensions,
87+
IncludedResourceTags: a.IncludedResourceTags,
88+
MetricNamespace: a.MetricNamespace,
89+
MetricNameTemplate: a.MetricNameTemplate,
90+
MetricHelpTemplate: a.MetricHelpTemplate,
91+
AzureCloudEnvironment: a.AzureCloudEnvironment,
92+
ValidateDimensions: a.ValidateDimensions,
93+
Regions: a.Regions,
94+
ConcurrencySubscription: a.ConcurrencySubscription,
95+
ConcurrencySubscriptionResource: a.ConcurrencySubscriptionResource,
96+
EnableCaching: a.EnableCaching,
8797
}
8898
}

‎internal/static/integrations/azure_exporter/azure_exporter.go‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,8 @@ import (
1818
)
1919

2020
type Exporter struct {
21-
cfg Config
22-
logger *zap.SugaredLogger // used by azure client
23-
ConcurrencyConfig azure_config.Opts
21+
cfg Config
22+
logger *zap.SugaredLogger // used by azure client
2423
}
2524

2625
func (e Exporter) MetricsHandler() (http.Handler, error) {
@@ -72,7 +71,13 @@ func (e Exporter) MetricsHandler() (http.Handler, error) {
7271
return
7372
}
7473

75-
prober := metrics.NewMetricProber(ctx, logEntry, nil, settings, e.ConcurrencyConfig)
74+
// Use concurrency settings from merged config to support query parameter overrides
75+
concurrencyConfig := azure_config.Opts{}
76+
concurrencyConfig.Prober.ConcurrencySubscription = mergedConfig.ConcurrencySubscription
77+
concurrencyConfig.Prober.ConcurrencySubscriptionResource = mergedConfig.ConcurrencySubscriptionResource
78+
concurrencyConfig.Prober.Cache = mergedConfig.EnableCaching
79+
80+
prober := metrics.NewMetricProber(ctx, logEntry, nil, settings, concurrencyConfig)
7681
prober.SetAzureClient(client)
7782
prober.SetPrometheusRegistry(reg)
7883
prober.SetAzureResourceTagManager(tagManager)

0 commit comments

Comments
 (0)