Merge pull request #568 from Beckn-One/feat/observability

Feat/observability

Author: nirmay

CONFIG.md

@@ -6,13 +6,14 @@
 3. [Top-Level Configuration](#top-level-configuration)
 4. [HTTP Configuration](#http-configuration)
 5. [Logging Configuration](#logging-configuration)
-6. [Plugin Manager Configuration](#plugin-manager-configuration)
-7. [Module Configuration](#module-configuration)
-8. [Handler Configuration](#handler-configuration)
-9. [Plugin Configuration](#plugin-configuration)
-10. [Routing Configuration](#routing-configuration)
-11. [Deployment Scenarios](#deployment-scenarios)
-12. [Configuration Examples](#configuration-examples)
+6. [Metrics Configuration](#metrics-configuration)
+7. [Plugin Manager Configuration](#plugin-manager-configuration)
+8. [Module Configuration](#module-configuration)
+9. [Handler Configuration](#handler-configuration)
+10. [Plugin Configuration](#plugin-configuration)
+11. [Routing Configuration](#routing-configuration)
+12. [Deployment Scenarios](#deployment-scenarios)
+13. [Configuration Examples](#configuration-examples)
 
 ---
 
@@ -70,6 +71,7 @@ The main configuration file follows this structure:
 ```yaml
 appName: "onix-local"
 log: {...}
+metrics: {...}
 http: {...}
 pluginManager: {...}
 modules: [...]
@@ -187,6 +189,120 @@ log:
 
 ---
 
+## Application-Level Plugins Configuration
+
+### `plugins`
+**Type**: `object`  
+**Required**: No  
+**Description**: Application-level plugin configurations. These plugins apply to the entire application and are shared across all modules.
+
+#### `plugins.otelsetup`
+**Type**: `object`  
+**Required**: No  
+**Description**: OpenTelemetry configuration controlling whether the Prometheus exporter is enabled.
+
+**Important**: This block is optional—omit it to run without telemetry. When present, the `/metrics` endpoint is exposed on a separate port (configurable via `metricsPort`) only if `enableMetrics: true`.
+
+##### Parameters:
+
+###### `id`
+**Type**: `string`  
+**Required**: Yes  
+**Description**: Plugin identifier. Must be `"otelsetup"`.
+
+###### `config`
+**Type**: `object`  
+**Required**: Yes  
+**Description**: Plugin configuration parameters.
+
+###### `config.enableMetrics`
+**Type**: `string` (boolean)  
+**Required**: No  
+**Default**: `"true"`  
+**Description**: Enables metrics collection and the `/metrics` endpoint. Must be `"true"` or `"false"` as a string.
+
+###### `config.serviceName`
+**Type**: `string`  
+**Required**: No  
+**Default**: `"beckn-onix"`  
+**Description**: Sets the `service.name` resource attribute.
+
+###### `config.serviceVersion`
+**Type**: `string`  
+**Required**: No  
+**Description**: Sets the `service.version` resource attribute.
+
+###### `config.environment`
+**Type**: `string`  
+**Required**: No  
+**Default**: `"development"`  
+**Description**: Sets the `deployment.environment` attribute (e.g., `development`, `staging`, `production`).
+
+###### `config.metricsPort`
+**Type**: `string`  
+**Required**: No  
+**Default**: `"9090"`  
+**Description**: Port on which the metrics HTTP server will listen. The metrics endpoint is hosted on a separate server from the main application.
+
+**Example - Enable Metrics** (matches `config/local-simple.yaml`):
+```yaml
+plugins:
+  otelsetup:
+    id: otelsetup
+    config:
+      serviceName: "beckn-onix"
+      serviceVersion: "1.0.0"
+      enableMetrics: "true"
+      environment: "development"
+      metricsPort: "9090"
+```
+
+### Accessing Metrics
+
+When `plugins.otelsetup.config.enableMetrics: "true"`, the metrics endpoint is hosted on a separate HTTP server. Scrape metrics at:
+
+```
+http://your-server:9090/metrics
+```
+
+**Note**: The metrics server runs on the port specified by `config.metricsPort` (default: `9090`), which is separate from the main application port configured in `http.port`.
+
+### Metrics Collected
+
+Metrics are organized by module for better maintainability and encapsulation:
+
+#### OTel Setup (from `otelsetup` plugin)
+- Prometheus exporter & `/metrics` endpoint on separate HTTP server
+- Go runtime instrumentation (`go_*`), resource attributes, and meter provider wiring
+
+#### Step Execution Metrics (from `telemetry` package)
+- `onix_step_executions_total`, `onix_step_execution_duration_seconds`, `onix_step_errors_total`
+
+#### Handler Metrics (from `handler` module)
+- `beckn_signature_validations_total` - Signature validation attempts
+- `beckn_schema_validations_total` - Schema validation attempts
+- `onix_routing_decisions_total` - Routing decisions taken by handler
+
+#### Cache Metrics (from `cache` plugin)
+- `onix_cache_operations_total`, `onix_cache_hits_total`, `onix_cache_misses_total`
+
+#### Plugin Metrics (from `telemetry` package)
+- `onix_plugin_execution_duration_seconds`, `onix_plugin_errors_total`
+
+#### Runtime Metrics
+- Go runtime metrics (`go_*`) and Redis instrumentation via `redisotel`
+
+Each metric includes consistent labels such as `module`, `role`, `action`, `status`, `step`, `plugin_id`, and `schema_version` to enable low-cardinality dashboards.
+
+**Note**: Metric definitions are now located in their respective modules:
+- OTel setup: `pkg/plugin/implementation/otelsetup`
+- Step metrics: `core/module/handler/step_metrics.go`
+- Handler metrics: `core/module/handler/handlerMetrics.go`
+- Cache metrics: `pkg/plugin/implementation/cache/cache_metrics.go`
+- Plugin metrics: `pkg/telemetry/pluginMetrics.go`
+
+---
+
 ## Plugin Manager Configuration
 
 ### `pluginManager`
@@ -1045,13 +1161,18 @@ routingRules:
 - Embedded Ed25519 keys
 - Local Redis
 - Simplified routing
+- Optional metrics collection (available on separate port when enabled)
 
 **Use Case**: Quick local development and testing
 
 ```yaml
 appName: "onix-local"
 log:
   level: debug
+metrics:
+  enabled: true
+  exporterType: prometheus
+  serviceName: onix-local
 http:
   port: 8081
 modules:
@@ -1063,6 +1184,8 @@ modules:
           config: {}
 ```
 
+**Metrics Access**: When enabled, access metrics at `http://localhost:9090/metrics` (default metrics port, configurable via `plugins.otelsetup.config.metricsPort`)
+
 ### 2. Local Development (Vault Mode)
 
 **File**: `config/local-dev.yaml`
@@ -1096,10 +1219,21 @@ modules:
 - Production Redis
 - Remote plugin loading
 - Pub/Sub integration
+- OpenTelemetry metrics enabled (available on separate port, default: 9090)
 
 **Use Case**: Single deployment serving both roles
 
 ```yaml
+appName: "onix-production"
+log:
+  level: info
+  destinations:
+    - type: stdout
+metrics:
+  enabled: true
+  exporterType: prometheus
+  serviceName: beckn-onix
+  serviceVersion: "1.0.0"
 pluginManager:
   root: /app/plugins
   remoteRoot: /mnt/gcs/plugins/plugins_bundle.zip
@@ -1122,6 +1256,9 @@ modules:
             topic: bapNetworkReciever
 ```
 
+**Metrics Access**: 
+- Prometheus scraping: `http://your-server:9090/metrics` (default metrics port, configurable via `plugins.otelsetup.config.metricsPort`)
+
 ### 4. Production BAP-Only Mode
 
 **File**: `config/onix-bap/adapter.yaml`

README.md

@@ -64,9 +64,45 @@ The **Beckn Protocol** is an open protocol that enables location-aware, local co
 ### 📊 **Observability**
 - **Structured Logging**: JSON-formatted logs with contextual information
 - **Transaction Tracking**: End-to-end request tracing with unique IDs
-- **Metrics Support**: Performance and business metrics collection
+- **OpenTelemetry Metrics**: Pull-based metrics exposed via `/metrics`
+  - RED metrics for every module and action (rate, errors, duration)
+  - Per-step histograms with error attribution
+  - Cache, routing, plugin, and business KPIs (signature/schema validations, Beckn messages)
+  - Native Prometheus exporter with Grafana dashboards & alert rules (`monitoring/`)
+  - Opt-in: add a `plugins.otelsetup` block in your config to wire the `otelsetup` plugin; omit it to run without metrics. Example:
+
+    ```yaml
+    plugins:
+      otelsetup:
+        id: otelsetup
+        config:
+          serviceName: "beckn-onix"
+          serviceVersion: "1.0.0"
+          enableMetrics: "true"
+          environment: "development"
+    ```
+  - **Modular Metrics Architecture**: Metrics are organized by module for better maintainability:
+    - OTel SDK wiring via `otelsetup` plugin
+    - Step execution metrics in `telemetry` package
+    - Handler metrics (signature, schema, routing) in `handler` module
+    - Cache metrics in `cache` plugin
+- **Runtime Instrumentation**: Go runtime + Redis client metrics baked in
 - **Health Checks**: Liveness and readiness probes for Kubernetes
 
+#### Monitoring Quick Start
+```bash
+./install/build-plugins.sh
+go build -o beckn-adapter ./cmd/adapter
+./beckn-adapter --config=config/local-simple.yaml
+cd monitoring && docker-compose -f docker-compose-monitoring.yml up -d
+open http://localhost:3000 # Grafana (admin/admin)
+```
+Resources:
+- `monitoring/prometheus.yml` – scrape config
+- `monitoring/prometheus-alerts.yml` – alert rules (RED, cache, step, plugin)
+- `monitoring/grafana/dashboards/beckn-onix-overview.json` – curated dashboard
+- `docs/METRICS_RUNBOOK.md` – runbook with PromQL recipes & troubleshooting
+
 ### 🌐 **Multi-Domain Support**
 - **Retail & E-commerce**: Product search, order management, fulfillment tracking
 - **Mobility Services**: Ride-hailing, public transport, vehicle rentals
@@ -356,6 +392,15 @@ modules:
 | POST | `/bpp/receiver/*` | Receives all BAP requests |
 | POST | `/bpp/caller/on_*` | Sends responses back to BAP |
 
+### Observability Endpoints
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/health` | Health check endpoint |
+| GET | `/metrics` | Prometheus metrics endpoint (when telemetry is enabled) |
+
+**Note**: The `/metrics` endpoint is available when `telemetry.enableMetrics: true` in the configuration file. It returns metrics in Prometheus format.
+
 ## Documentation
 
 - **[Setup Guide](SETUP.md)**: Complete installation, configuration, and deployment instructions

cmd/adapter/main.go

@@ -17,12 +17,19 @@ import (
 	"github.com/beckn-one/beckn-onix/core/module/handler"
 	"github.com/beckn-one/beckn-onix/pkg/log"
 	"github.com/beckn-one/beckn-onix/pkg/plugin"
+	"github.com/beckn-one/beckn-onix/pkg/telemetry"
 )
 
+// ApplicationPlugins holds application-level plugin configurations.
+type ApplicationPlugins struct {
+	OtelSetup *plugin.Config `yaml:"otelsetup,omitempty"`
+}
+
 // Config struct holds all configurations.
 type Config struct {
 	AppName       string                `yaml:"appName"`
 	Log           log.Config            `yaml:"log"`
+	Plugins       ApplicationPlugins    `yaml:"plugins,omitempty"`
 	PluginManager *plugin.ManagerConfig `yaml:"pluginManager"`
 	Modules       []module.Config       `yaml:"modules"`
 	HTTP          httpConfig            `yaml:"http"`
@@ -91,11 +98,39 @@ func validateConfig(cfg *Config) error {
 	return nil
 }
 
+// loadAppPlugin is a generic function to load and validate application-level plugins.
+func loadAppPlugin[T any](ctx context.Context, name string, cfg *plugin.Config, mgrFunc func(context.Context, *plugin.Config) (T, error)) error {
+	if cfg == nil {
+		log.Debugf(ctx, "Skipping %s plugin: not configured", name)
+		return nil
+	}
+
+	_, err := mgrFunc(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("failed to load %s plugin (%s): %w", name, cfg.ID, err)
+	}
+
+	log.Debugf(ctx, "Loaded %s plugin: %s", name, cfg.ID)
+	return nil
+}
+
+// initAppPlugins initializes application-level plugins including telemetry.
+// This function is designed to be extensible for future plugin types.
+func initAppPlugins(ctx context.Context, mgr *plugin.Manager, cfg ApplicationPlugins) error {
+	if err := loadAppPlugin(ctx, "OtelSetup", cfg.OtelSetup, func(ctx context.Context, cfg *plugin.Config) (*telemetry.Provider, error) {
+		return mgr.OtelSetup(ctx, cfg)
+	}); err != nil {
+		return fmt.Errorf("failed to initialize application plugins: %w", err)
+	}
+
+	return nil
+}
+
 // newServer creates and initializes the HTTP server.
 func newServer(ctx context.Context, mgr handler.PluginManager, cfg *Config) (http.Handler, error) {
 	mux := http.NewServeMux()
-	err := module.Register(ctx, cfg.Modules, mux, mgr)
-	if err != nil {
+
+	if err := module.Register(ctx, cfg.Modules, mux, mgr); err != nil {
 		return nil, fmt.Errorf("failed to register modules: %w", err)
 	}
 	return mux, nil
@@ -126,6 +161,11 @@ func run(ctx context.Context, configPath string) error {
 	closers = append(closers, closer)
 	log.Debug(ctx, "Plugin manager loaded.")
 
+	// Initialize plugins including telemetry.
+	if err := initAppPlugins(ctx, mgr, cfg.Plugins); err != nil {
+		return fmt.Errorf("failed to initialize plugins: %w", err)
+	}
+
 	// Initialize HTTP server.
 	log.Infof(ctx, "Initializing HTTP server")
 	srv, err := newServerFunc(ctx, mgr, cfg)

cmd/adapter/main_test.go

@@ -73,6 +73,11 @@ func (m *MockPluginManager) KeyManager(ctx context.Context, cache definition.Cac
 	return nil, nil
 }
 
+// TransportWrapper returns a mock implementation of the TransportWrapper interface.
+func (m *MockPluginManager) TransportWrapper(ctx context.Context, cfg *plugin.Config) (definition.TransportWrapper, error) {
+	return nil, nil
+}
+
 // SchemaValidator returns a mock implementation of the SchemaValidator interface.
 func (m *MockPluginManager) SchemaValidator(ctx context.Context, cfg *plugin.Config) (definition.SchemaValidator, error) {
 	return nil, nil
@@ -170,14 +175,19 @@ func TestRunFailure(t *testing.T) {
 
 			// Mock dependencies
 			originalNewManager := newManagerFunc
-			// newManagerFunc = func(ctx context.Context, cfg *plugin.ManagerConfig) (*plugin.Manager, func(), error) {
-			// 	return tt.mockMgr()
-			// }
-			newManagerFunc = nil
+			// Ensure newManagerFunc is never nil to avoid panic if invoked.
+			newManagerFunc = func(ctx context.Context, cfg *plugin.ManagerConfig) (*plugin.Manager, func(), error) {
+				_, closer, err := tt.mockMgr()
+				if err != nil {
+					return nil, closer, err
+				}
+				// Return a deterministic error so the code path exits cleanly if reached.
+				return nil, closer, errors.New("mock manager error")
+			}
 			defer func() { newManagerFunc = originalNewManager }()
 
-			originalNewServer := newServerFunc
-			newServerFunc = func(ctx context.Context, mgr handler.PluginManager, cfg *Config) (http.Handler, error) {
+		originalNewServer := newServerFunc
+		newServerFunc = func(ctx context.Context, mgr handler.PluginManager, cfg *Config) (http.Handler, error) {
 				return tt.mockServer(ctx, mgr, cfg)
 			}
 			defer func() { newServerFunc = originalNewServer }()