Merge pull request #605 from Beckn-One/feat/issues_585

Feat: Network Observability - OTelSetup plugin, Instrumentation in Core.

Author: nirmay

.gitignore

@@ -1,6 +1,7 @@
 # Logs
 .DS_Store
 logs
+.idea
 *.log
 npm-debug.log*
 yarn-debug.log*

CONFIG.md

@@ -199,9 +199,7 @@ log:
 #### `plugins.otelsetup`
 **Type**: `object`  
 **Required**: No  
-**Description**: OpenTelemetry configuration controlling whether the Prometheus exporter is enabled.
-
-**Important**: This block is optional—omit it to run without telemetry. When present, the `/metrics` endpoint is exposed on a separate port (configurable via `metricsPort`) only if `enableMetrics: true`.
+**Description**: OpenTelemetry (OTLP) configuration for metrics, traces, and logs. When configured, telemetry is exported to an OTLP collector endpoint. Omit this block to run without telemetry.
 
 ##### Parameters:
 
@@ -215,11 +213,10 @@ log:
 **Required**: Yes  
 **Description**: Plugin configuration parameters.
 
-###### `config.enableMetrics`
-**Type**: `string` (boolean)  
-**Required**: No  
-**Default**: `"true"`  
-**Description**: Enables metrics collection and the `/metrics` endpoint. Must be `"true"` or `"false"` as a string.
+###### `config.otlpEndpoint`
+**Type**: `string`  
+**Required**: Yes (when OtelSetup is used)  
+**Description**: OTLP gRPC endpoint (host:port) for exporting metrics, traces, and logs. Example: `"localhost:4317"`, `"otel-collector-bap:4317"`.
 
 ###### `config.serviceName`
 **Type**: `string`  
@@ -238,47 +235,114 @@ log:
 **Default**: `"development"`  
 **Description**: Sets the `deployment.environment` attribute (e.g., `development`, `staging`, `production`).
 
-###### `config.metricsPort`
+###### `config.domain`
 **Type**: `string`  
 **Required**: No  
 **Default**: `"9090"`  
 **Description**: Port on which the metrics HTTP server will listen. The metrics endpoint is hosted on a separate server from the main application.
 
-**Example - Enable Metrics** (matches `config/local-simple.yaml`):
+###### `config.enableMetrics`
+**Type**: `string` (boolean)  
+**Required**: No  
+**Default**: `"false"`  
+**Description**: Enables metrics collection and OTLP metric export. Use `"true"` or `"false"` as a string.
+
+###### `config.enableTracing`
+**Type**: `string` (boolean)  
+**Required**: No  
+**Default**: `"false"`  
+**Description**: Enables trace export via OTLP. Use `"true"` or `"false"` as a string.
+
+###### `config.enableLogs`
+**Type**: `string` (boolean)  
+**Required**: No  
+**Default**: `"false"`  
+**Description**: Enables log export via OTLP (e.g. audit logs). Use `"true"` or `"false"` as a string.
+
+###### `config.timeInterval`
+**Type**: `string` (integer)  
+**Required**: No  
+**Default**: `"5"`  
+**Description**: Time interval in seconds used for periodic metric export or batching.
+
+###### `config.auditFieldsConfig`
+**Type**: `string`  
+**Required**: No  
+**Description**: Path to a YAML file that defines which request/response fields are included in audit logs, per action. See [Audit fields configuration](#audit-fields-configuration). Example: `"/app/config/audit-fields.yaml"`.
+
+
+**Example - OTLP export with audit logs** (e.g. `config/local-beckn-one-bap.yaml`):
 ```yaml
 plugins:
   otelsetup:
     id: otelsetup
     config:
-      serviceName: "beckn-onix"
+      serviceName: "beckn-one-bap"
       serviceVersion: "1.0.0"
-      enableMetrics: "true"
       environment: "development"
-      metricsPort: "9090"
+      domain: "ev_charging"
+      otlpEndpoint: "otel-collector-bap:4317"
+      enableMetrics: "true"
+      enableTracing: "true"
+      enableLogs: "true"
+      timeInterval: "5"
+      networkMetricsGranularity: "2min"
+      networkMetricsFrequency: "4min"
+      auditFieldsConfig: "/app/config/audit-fields.yaml"
 ```
 
-### Accessing Metrics
 
-When `plugins.otelsetup.config.enableMetrics: "true"`, the metrics endpoint is hosted on a separate HTTP server. Scrape metrics at:
 
+### Audit fields configuration
+
+When `config.auditFieldsConfig` points to a YAML file, audit logs (emitted via OTLP when `enableLogs: "true"`) include only the fields you list per action. The file format:
+
+```yaml
+auditRules:
+  default:                           # Optional: fallback for actions without a specific list
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.domain
+    - context.bap_id
+    - context.bpp_id
+  discover:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.filters
+    - message.spatial
+  select:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - message.order.beckn:buyer.beckn:id
+    # ... more dot-path fields
 ```
-http://your-server:9090/metrics
-```
 
-**Note**: The metrics server runs on the port specified by `config.metricsPort` (default: `9090`), which is separate from the main application port configured in `http.port`.
+- **Top-level key**: `auditRules`.
+- **Action keys**: Use Beckn action names (e.g. `discover`, `select`, `init`, `confirm`, `update`, `track`, `cancel`, `rating`, `support`). Use `default` for actions that do not have a specific list.
+- **Values**: List of dot-path strings into the request/response JSON (e.g. `context.transaction_id`, `message.order.beckn:id`). Namespaced keys use colons (e.g. `beckn:id`).
+
+See `config/audit-fields.yaml` for a full example.
 
 ### Metrics Collected
 
+When OtelSetup is configured with `otlpEndpoint`, metrics and traces are exported via OTLP (no separate metrics HTTP server). Scrape metrics from your OTLP collector 
+
 Metrics are organized by module for better maintainability and encapsulation:
 
 #### OTel Setup (from `otelsetup` plugin)
-- Prometheus exporter & `/metrics` endpoint on separate HTTP server
-- Go runtime instrumentation (`go_*`), resource attributes, and meter provider wiring
+- OTLP export for metrics, traces, and logs (gRPC endpoint). 
+- Go runtime instrumentation (`go_*`), resource attributes, and meter/tracer provider wiring.
+- When `enableLogs: "true"` and `auditFieldsConfig` is set, audit logs are emitted via OTLP with fields defined in the audit-fields YAML.
 
 #### Step Execution Metrics (from `telemetry` package)
 - `onix_step_executions_total`, `onix_step_execution_duration_seconds`, `onix_step_errors_total`
 
 #### Handler Metrics (from `handler` module)
+- `onix_http_request_count` – HTTP requests by status class, route, method, role, sender, recipient (and optional network metric attributes).
 - `beckn_signature_validations_total` - Signature validation attempts
 - `beckn_schema_validations_total` - Schema validation attempts
 - `onix_routing_decisions_total` - Routing decisions taken by handler
@@ -752,12 +816,12 @@ publisher:
 middleware:
   - id: reqpreprocessor
     config:
-      uuidKeys: transaction_id,message_id
       role: bap
+      contextKeys: transaction_id,message_id,subscriber_id,module_id
 ```
 
 **Parameters**:
-- `uuidKeys`: Comma-separated list of fields to auto-generate UUIDs for if missing
+- `contextKeys`: Comma-separated list of fields to auto-generate UUIDs for if missing
 - `role`: BAP or BPP role for request processing
 
 ---

README.md

@@ -132,7 +132,7 @@ The **Beckn Protocol** is an open protocol that enables location-aware, local co
 - **Decrypter**: AES decryption for encrypted data processing
 - **ReqPreprocessor**: Request preprocessing (UUID generation, headers)
 - **ReqMapper**: Middleware to transform payload either between Beckn versions or against other platforms.
-- **OtelSetup**: Observability Setup to make metrics, traces and logs available
+- **OtelSetup**: Observability setup for metrics, traces, and logs (OTLP). Supports optional audit log configuration via `auditFieldsConfig` (YAML mapping actions to fields) . See [CONFIG.md](CONFIG.md) for details.
 
 
 ## Quick Start
@@ -330,10 +330,11 @@ modules:
 ### Deployment Modes
 
 1. **Combined Mode**: Single instance handling both BAP and BPP (`config/onix/`) - Uses `secretskeymanager` (HashiCorp Vault) for production key management
-2. **BAP-Only Mode**: Dedicated buyer-side deployment (`config/onix-bap/`) 
-3. **BPP-Only Mode**: Dedicated seller-side deployment (`config/onix-bpp/`) 
-4. **Local Development Combined Mode**: Simplified configuration (`config/local-simple.yaml`) - Uses `simplekeymanager` with embedded Ed25519 keys, no vault setup needed.
-5. **Local Development Combined Mode (Alternative)**: Development configuration (`config/local-dev.yaml`) - Uses `keymanager` vault setup needed
+2. **BAP-Only Mode**: Dedicated buyer-side deployment (`config/onix-bap/`)
+3. **BPP-Only Mode**: Dedicated seller-side deployment (`config/onix-bpp/`)
+4. **Local Development Combined Mode**: Simplified configuration (`config/local-simple.yaml`) - Uses `simplekeymanager` with embedded Ed25519 keys, no vault setup needed
+5. **Local Development Combined Mode (Alternative)**: Development configuration (`config/local-dev.yaml`) - Uses `keymanager`, vault setup needed
+6. **Local with Observability (BAP/BPP)**: Configs `config/local-beckn-one-bap.yaml` and `config/local-beckn-one-bpp.yaml` include OtelSetup (metrics, traces, audit logs) for use with an OTLP collector. Audit fields are configured via `config/audit-fields.yaml`. For a full stack (collectors, Grafana, Loki), see `install/network-observability/`
 
 ## API Endpoints
 
@@ -359,14 +360,6 @@ modules:
 | POST | `/bpp/receiver/*` | Receives all BAP requests |
 | POST | `/bpp/caller/on_*` | Sends responses back to BAP |
 
-### Observability Endpoints
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| GET | `/health` | Health check endpoint |
-| GET | `/metrics` | Prometheus metrics endpoint (when telemetry is enabled) |
-
-**Note**: The `/metrics` endpoint is available when `telemetry.enableMetrics: true` in the configuration file. It returns metrics in Prometheus format.
 
 ## Documentation
 

cmd/adapter/main.go

@@ -11,6 +11,7 @@ import (
 	"sync"
 	"time"
 
+	"github.com/beckn-one/beckn-onix/pkg/model"
 	"gopkg.in/yaml.v2"
 
 	"github.com/beckn-one/beckn-onix/core/module"
@@ -153,6 +154,9 @@ func run(ctx context.Context, configPath string) error {
 		return fmt.Errorf("failed to initialize logger: %w", err)
 	}
 
+	//to add the parent_id in the context value so it get passed to the logs
+	ctx = addParentIdCtx(ctx, cfg)
+
 	// Initialize plugin manager.
 	log.Infof(ctx, "Initializing plugin manager")
 	mgr, closer, err := newManagerFunc(ctx, cfg.PluginManager)
@@ -220,3 +224,40 @@ func shutdown(ctx context.Context, httpServer *http.Server, wg *sync.WaitGroup,
 		}
 	}()
 }
+
+func addParentIdCtx(ctx context.Context, config *Config) context.Context {
+	var parentID string
+	var podName string
+
+	if p := os.Getenv("POD_NAME"); p != "" {
+		log.Infof(ctx, "Adding POD name: %s", p)
+		podName = p
+	} else {
+		log.Info(ctx, "POD_NAME environment variable not set, falling back to hostname")
+		if hostname, err := os.Hostname(); err == nil {
+			log.Infof(ctx, "Setting POD name as hostname: %s", hostname)
+			podName = hostname
+		} else {
+			log.Info(ctx, "failed to get POD name")
+		}
+	}
+
+	for _, m := range config.Modules {
+		if m.Handler.Role == "" || m.Handler.SubscriberID == "" {
+			continue
+		}
+		candidate := string(m.Handler.Role) + ":" + m.Handler.SubscriberID + ":" + podName
+		if parentID == "" {
+			parentID = candidate
+		} else if candidate != parentID {
+			log.Warnf(ctx, "Multiple distinct role:subscriberID pairs found in modules (using %q, also saw %q); consider explicit parent_id config", parentID, candidate)
+		}
+	}
+
+	if parentID != "" {
+		ctx = context.WithValue(ctx, model.ContextKeyParentID, parentID)
+	} else {
+		log.Warnf(ctx, "Failed to find parent ID in config; add role and subscriber_id to the handler config")
+	}
+	return ctx
+}

config/audit-fields.yaml

@@ -0,0 +1,89 @@
+auditRules:
+  default:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.domain
+    - context.bap_id
+    - context.bpp_id
+
+  discover:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.filters
+    - message.spatial
+
+  select:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.order.beckn:buyer.beckn:id
+    - message.order.beckn:seller
+    - message.order.beckn:orderItems.beckn:acceptedOffer.beckn:id
+    - message.order.beckn:orderAttributes
+
+  init:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.order.beckn:id
+    - message.order.beckn:buyer.beckn:id
+    - message.order.beckn:orderValue.value
+    - message.order.beckn:payment.beckn:paymentStatus
+
+  confirm:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.order.beckn:id
+    - message.order.beckn:orderStatus
+    - message.order.beckn:buyer.beckn:id
+    - message.order.beckn:payment.beckn:txnRef
+    - message.order.beckn:payment.beckn:paymentStatus
+
+  update:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.order.beckn:id
+    - message.order.beckn:orderStatus
+    - message.order.beckn:fulfillment.beckn:deliveryAttributes.sessionStatus
+
+  track:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.order.beckn:id
+
+  cancel:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.order.beckn:id
+    - message.order.beckn:orderStatus
+    - message.order.beckn:buyer.beckn:id
+
+  rating:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.id
+    - message.value
+    - message.category
+
+  support:
+    - context.transaction_id
+    - context.message_id
+    - context.action
+    - context.timestamp
+    - message.ref_id
+    - message.ref_type

config/local-beckn-one-bap.yaml

@@ -16,6 +16,25 @@ http:
     idle: 30
 pluginManager:
   root: ./plugins
+
+# OpenTelemetry (OTLP) - metrics and traces sent to OTEL collector
+plugins:
+  otelsetup:
+    id: otelsetup
+    config:
+      serviceName: "beckn-one-bap"
+      serviceVersion: "1.0.0"
+      environment: "development"
+      domain: "ev_charging"
+      otlpEndpoint: "otel-collector-bap:4317"
+      enableMetrics: "true"
+      networkMetricsGranularity: "2min"
+      networkMetricsFrequency: "4min"
+      enableTracing: "true"
+      enableLogs: "true"
+      timeInterval: "5"
+      auditFieldsConfig: "/app/config/audit-fields.yaml"
+
 modules:
   - name: bapTxnReceiver
     path: /bap/receiver/