33This sample demonstrates three advanced Cadence worker features:
44
551 . ** Worker Poller Autoscaling** - Dynamic adjustment of worker poller goroutines based on workload
6- 2 . ** Prometheus Tally Reporter ** - Metrics collection using Tally and Prometheus
7- 3 . ** HTTP Endpoint for Prometheus Scraping ** - Exposing metrics for monitoring
6+ 2 . ** Integrated Prometheus Metrics ** - Real-time metrics collection using Tally with Prometheus reporter
7+ 3 . ** Autoscaling Metrics ** - Comprehensive autoscaling behavior metrics exposed via HTTP endpoint
88
99## Features
1010
@@ -18,10 +18,11 @@ The worker uses `worker.NewV2` with `AutoScalerOptions` to enable true autoscali
1818The worker automatically adjusts the number of poller goroutines between the min and max values based on the current workload.
1919
2020### Prometheus Metrics
21- The sample exposes standard Cadence metrics via Prometheus, including comprehensive autoscaling behavior metrics:
22- - Standard worker performance metrics (poller counts, task processing rates)
23- - Autoscaling-specific metrics (` cadence_concurrency_auto_scaler_* ` )
24- - All metrics automatically emitted by the Cadence Go client
21+ The sample uses Tally with Prometheus reporter to expose comprehensive metrics:
22+ - ** Real-time autoscaling metrics** - Poller count changes, quota adjustments, wait times
23+ - ** Worker performance metrics** - Task processing rates, poller utilization, queue depths
24+ - ** Standard Cadence metrics** - All metrics automatically emitted by the Cadence Go client
25+ - ** Sanitized metric names** - Prometheus-compatible metric names and labels
2526
2627### Monitoring Dashboards
2728When running the Cadence server locally with Grafana, you can access the client dashboards at:
@@ -41,7 +42,10 @@ When running the Cadence server locally with Grafana, you can access the client
4142./bin/autoscaling-monitoring -m worker
4243```
4344
44- ### 2. Start the Prometheus Server (Optional)
45+ The worker automatically exposes metrics at: http://127.0.0.1:8004/metrics
46+
47+ ### 2. Access Metrics (Optional)
48+ For dedicated metrics server:
4549``` bash
4650./bin/autoscaling-monitoring -m server
4751```
@@ -100,8 +104,8 @@ The configuration values are used throughout the sample:
1001043. **Activity Configuration** (`activities.go`) :
101105 - ` minProcessingTime` , `maxProcessingTime` → Activity processing time range
102106
103- 4. **Prometheus Configuration** (`prometheus_server.go` ) :
104- - ` listenAddress` → HTTP server port
107+ 4. **Prometheus Configuration** (integrated ) :
108+ - `listenAddress` → Metrics endpoint port (default : 127.0.0.1:8004)
105109
106110# ## Default Configuration
107111
@@ -128,6 +132,9 @@ autoscaling:
128132
129133# ## Metrics Endpoints
130134- **Prometheus Metrics**: http://127.0.0.1:8004/metrics
135+ - Exposed automatically when running worker or server mode
136+ - Real-time autoscaling and worker performance metrics
137+ - Prometheus-compatible format with sanitized names
131138
132139# ## Grafana Dashboard
133140Access the Cadence client dashboard at : http://localhost:3000/d/dehkspwgabvuoc/cadence-client
@@ -144,6 +151,8 @@ Access the Cadence client dashboard at: http://localhost:3000/d/dehkspwgabvuoc/c
144151 - ` cadence_worker_poller_count` - Number of active poller goroutines (key autoscaling indicator)
145152 - ` cadence_concurrency_auto_scaler_poller_quota` - Current poller quota for autoscaling
146153 - ` cadence_concurrency_auto_scaler_poller_wait_time` - Time pollers wait for tasks
154+ - ` cadence_concurrency_auto_scaler_scale_up_count` - Number of scale-up events
155+ - ` cadence_concurrency_auto_scaler_scale_down_count` - Number of scale-down events
147156
148157# # How It Works
149158
@@ -161,11 +170,11 @@ The worker uses `worker.NewV2` with `AutoScalerOptions` to:
161170- Automatically adjust based on task queue depth and processing time
162171
163172# ## Metrics Collection
164- The sample exposes standard Cadence metrics via Prometheus :
165- - Worker poller count and utilization
166- - Decision and activity task processing rates
167- - Task queue depth and processing times
168- - All standard Cadence worker metrics via Tally
173+ The sample uses Tally with Prometheus reporter for comprehensive metrics :
174+ - **Real-time autoscaling metrics** - Poller count changes, quota adjustments, scale events
175+ - **Worker performance metrics** - Task processing rates, poller utilization, queue depths
176+ - **Standard Cadence metrics** - All metrics automatically emitted by the Cadence Go client
177+ - **Sanitized metric names** - Prometheus-compatible format with proper character replacement
169178
170179# # Production Considerations
171180
@@ -207,9 +216,10 @@ The sample exposes standard Cadence metrics via Prometheus:
207216 - Review default values if config file is not found
208217
2092184. **Metrics Not Appearing** :
210- - Verify Prometheus server is running
211- - Check metrics endpoint is accessible
219+ - Verify worker is running (metrics are exposed automatically)
220+ - Check metrics endpoint is accessible : http://127.0.0.1:8004/metrics
212221 - Ensure Prometheus is configured to scrape the endpoint
222+ - Check for metric name sanitization issues
213223
2142245. **Dashboard Not Loading** :
215225 - Verify Grafana is running
0 commit comments