OTel working branch targeting OTel feature branch

docs/user-guide/api-mediation/observability/apiml-provided-observability-signals-and-attributes.md

@@ -0,0 +1,16 @@
+# API ML Provided Observability Signals and Attributes
+
+**TODO: Dev to provide Actual Signals and Attributes**
+
+<!-- This could be included in this topic. Please review -->
+
+## Custom Telemetry Template
+Use this template when requesting or defining new custom metrics for the API ML:
+
+* **Signal Type**: (Metric / Trace / Log)
+* **Name**: `zowe.apiml.[component].[functional_area]`
+* **Description**: What does this signal represent?
+* **Required Attributes**: 
+    * `route.id`: Identifier of the routed service.
+    * `client.id`: (Optional) The ID of the consuming application.
+    * `zos.smf.id`: Automatically inherited from Resource.

docs/user-guide/api-mediation/observability/configuring-otel-deployment-attributes.md

@@ -0,0 +1,26 @@
+# Configuring OpenTelemetry Deployment Attributes
+
+To configure deployment-specific resource attributes for the Zowe API ML. These attributes allow you to categorize telemetry data based on the lifecycle stage of the application, such as distinguishing between production, staging, or development environments.
+
+Unlike z/OS attributes which are often discovered automatically, deployment attributes are strictly informative and are typically defined manually. These attributes do not affect the unique identity of the service but are essential for filtering and grouping data within your observability backend. By explicitly labeling your environment, you ensure that performance anomalies in a test environment do not trigger false alerts in production monitoring views.
+
+## Deployment Attribute Reference
+
+The following attribute is used to describe the deployment of the single-service deployment of API ML:
+
+* **deployment.environment.name**  
+    Specifies the name of the deployment environment (Example: dev, test, staging, or production). Configuration Source: zowe.yaml
+
+## Configuration Example in zowe.yaml
+
+To set the deployment environment, add the `deployment.environment.name` key to the `resource.attributes` section of your zowe.yaml file.
+
+```
+zowe:
+  observability:
+    enabled: true
+    resource:
+      attributes:
+      # Deployment Attribute (Manual Entry)
+      deployment.environment.name: "production"
+```

docs/user-guide/api-mediation/observability/configuring-otel-service-attributes.md

@@ -0,0 +1,96 @@
+# Configure OpenTelemetry Service Attributes
+
+Services are identified via the service.name, service.namespace, and service.instance.id properties. Together, these attributes create a unique identity for API ML instances across your enterprise.
+
+In complex mainframe environments, you may have multiple API ML installations across different Sysplexes or data centers. To monitor these effectively, you must balance Logical Grouping (viewing all API ML traffic as one functional unit) with Instance Differentiation (identifying exactly which specific Address Space is experiencing an issue).
+
+## The Hierarchy of Identification
+OpenTelemetry uses a three-tier approach to define service identity:
+
+* **service.name** (The Service)  
+Identifies the logical name of the service. This property value should be identical for all instances across your entire organization that perform the same function (e.g., zowe-apiml). Expected to be globally unique if `namespace` is not defined.
+
+* **service.namespace** (The Environment/Site)  
+Groups services into logical sets. Use this property value to distinguish between different installations, such as sysplex-a vs. sysplex-b, or north-datacenter vs. south-datacenter. `service.name` is expected to be unique within the same `namespace`.
+
+* **service.instance.id** (The Unique Instance)  
+Identifies a specific running process or Address Space. This value must be globally unique for every instance. As multiple z/OS systems can run identical Job Names, ensure that you combine the Job Name with a unique identifier (such as the LPAR name or a UUID) to ensure the instance can be isolated during troubleshooting.
+
+<!-- Should we add service.version to this list of properties? -->
+
+## Configuration Examples
+
+**Example 1: Single API ML Installation (High Availability)**
+
+In this scenario, both instances share the same namespace because they belong to the same logical cluster on the same Sysplex.
+
+| Attribute | Instance 1 | Instance 2 |
+| :--- | :--- | :--- |
+| **service.name** | `zowe-apiml` | `zowe-apiml` |
+| **service.namespace** | `production-plex` | `production-plex` |
+| **service.instance.id** | `APIML01` | `APIML02` |
+
+**Instance 1 configuration**
+```
+zowe:
+  components:
+    api-mediation-layer:
+      observability:
+        enabled: true
+        resource:
+          attributes:
+            service.name: "zowe-apiml"
+            service.namespace: "production-plex"
+            service.instance.id: "APIML01"
+```            
+**Instance 2 configuration**
+```
+zowe:
+  components:
+    api-mediation-layer:
+      observability:
+        enabled: true
+        resource:
+          attributes:
+            service.name: "zowe-apiml"
+            service.namespace: "production-plex"
+            service.instance.id: "APIML02"
+```
+
+## Example of Multi-Site Deployment
+
+In this scenario, instances are separated by namespace to represent their physical data center locations.
+
+| Attribute | Site 1 (Instance A) | Site 1 (Instance B) | Site 2 (Instance C) |
+| :--- | :--- | :--- | :--- |
+| **service.name** | `zowe-apiml` | `zowe-apiml` | `zowe-apiml` |
+| **service.namespace** | `east-coast` | `east-coast` | `west-coast` |
+| **service.instance.id** | `ZOWE-E1` | `ZOWE-E2` | `ZOWE-W1` |
+
+**Site 1 (East Coast) Configuration:**
+
+```
+zowe:
+  components:
+    api-mediation-layer:
+      observability:
+        enabled: true
+        resource:
+          attributes:
+            service.name: "zowe-apiml"
+            service.namespace: "east-coast"
+            service.instance.id: "ZOWE-E1"
+```
+**Site 2 (West Coast) Configuration:**
+```
+zowe:
+  components:
+    api-mediation-layer:
+      observability:
+        enabled: true
+        resource:
+          attributes:
+            service.name: "zowe-apiml"
+            service.namespace: "west-coast"
+            service.instance.id: "ZOWE-W1"
+```

docs/user-guide/api-mediation/observability/configuring-otel-zos-attributes.md

@@ -0,0 +1,57 @@
+# Configure OpenTelemetry z/OS Attributes
+
+<!-- VALIDATE THIS CONTENT AFTER SUPPORT IS IMPLEMENTED. -->
+
+z/OS-specific resource attributes for API ML provide essential mainframe context to your telemetry data, allowing you to correlate metrics, traces, and logs with specific system identifiers such as SMF IDs, Sysplex names, and LPARs. By providing z/OS platform context, mainframe performance data can be integrated into distributed observability backends.
+
+## How system discovery works
+
+The z/OS attributes are primarily populated through an automated System Discovery process that occurs during the initialization of the API ML service. The integrated OpenTelemetry SDK executes platform-specific calls to query z/OS Control Blocks (such as the CVTSNAME or ECVT) and system variables.
+
+## z/OS Attribute Reference
+
+The following attributes are captured during system discovery to describe the mainframe environment:
+
+* **zos.smf.id**  
+The System Management Facility (SMF) Identifier that uniquely identifies a z/OS system within a SYSPLEX.
+Configuration Source: System discovery
+
+* **zos.sysplex.name**  
+The name of the SYSPLEX to which the z/OS system belongs.
+Configuration Source: System discovery
+
+* **mainframe.lpar.name**  
+Name of the LPAR that hosts the z/OS system.
+Configuration Source: System discovery
+
+* **os.type**  
+The operating system type, set to `zos`.
+Configuration Source: Static
+
+* **os.version**  
+The version string of the operating system (e.g., the release returned by `D IPLINFO`).
+Configuration Source: System discovery
+
+* **process.command**  
+The command or JOB name used to launch the Zowe process.
+Configuration Source: System discovery
+
+* **process.pid**  
+The Process Identifier, which on z/OS is set to the Address Space Identifier (ASID).
+Configuration Source: System discovery
+
+## Overriding Discovered Attributes in zowe.yaml
+
+While the discovery process handles most identifiers automatically, you may occasionally need to provide a manual override (for example, in shared environments where you wish to report a custom logical LPAR name). This is performed in the `resource.attributes` section of your zowe.yaml:
+
+```
+zowe:
+  observability:
+    enabled: true
+    resource:
+      attributes:
+        # Overriding discovered z/OS attributes
+        zos.smf.id: "MVS1"
+        zos.sysplex.name: "LOCALPLX"
+        mainframe.lpar.name: "PRODLPAR"
+```

docs/user-guide/api-mediation/observability/enabling-observability-in-zowe-yaml.md

@@ -0,0 +1,58 @@
+# Enabling API ML Observability in zowe.yaml
+
+Review how to enable and configure the OpenTelemetry (OTel) integration within the Zowe API Mediation Layer (API ML) single-service deployment. Configure these parameters in `zowe.yaml` to enable API ML to export metrics, traces, and logs to an OpenTelemetry Collector.
+
+## Configuration Overview
+
+The observability configuration is located under the API Mediation Layer `component` section of the zowe.yaml, under which there are three observability properties:
+
+* **enabled**  
+  Activates the OTel SDK. Set to `true` to initialize the OpenTelemetry SDK. 
+
+* **exporter**  
+Defines where the data is sent.  Sub-properties of `exporter` include the following:
+
+  * **exporter.otlp.protocol**  
+    The URL of your OTLP-compatible collector (e.g., z-Iris or Jaeger)
+
+  * **exporter.otlp.protocol**  
+    The protocol is either `grpc` or `http/protobuf`.
+    **Default:** `grcp`
+
+* **resource**  
+Defines the identity of the producer (Attributes).
+
+  * **resource.attributes**  
+    A collection of key-value pairs used to identify the telemetry source. See the following sub-properties of `resource.attributes`:
+
+    * **service.name**  
+    Logical name of the service. Must be the same for all instances within the same HA deployment. Expected to be globally unique if `namespace` is not defined.
+
+    * **service.namespace**  
+    The assigned value should help distinguish a group of services, such as the LPAR, or owner team. `service.name` is expected to be unique within the same `namespace`.
+
+    * **deployment.environment.name**  
+    Specifies the name of the deployment environment (Example: dev, test, staging, or production). Configuration Source: zowe.yaml
+
+To enable observability, configure the OpenTelemetry exporter and resource attributes within your `zowe.yaml` file with the following structure:
+
+```
+zowe:
+  observability:
+    enabled: true
+    exporter:
+      otlp:
+        endpoint: "http://otel-collector.your.domain:4317"
+        protocol: "grpc"
+        timeout: 10000
+    resource:
+      attributes:
+        service.name: "zowe-apiml"
+        service.namespace: "finance-production"
+        deployment.environment.name: "production"
+```
+
+## How the Export Works
+
+When `enabled: true` is set, the API ML single-service starts a background telemetry engine. This engine gathers all signals and bundles these signals with all Resource Attributes. These bundles are then pushed by means of the OTLP Exporter to your specified endpoint.
+

docs/user-guide/api-mediation/observability/observability-outline.md

@@ -0,0 +1,15 @@
+# Outline of API ML Observability Topics
+
+The following files will be presented under Advanced server-side configuration under the **Install** tab:
+
+* Configuring API ML Observability via OpenTelemetry
+  * [Configuring OpenTelemetry service attributes](configuring-otel-service-attributes.md)
+  * [Configuring OpenTelemetry deployment attributes](configuring-otel-deployment-attributes.md)
+  * [Configuring OpenTelemetry z/OS attributes](configuring-otel-zos-attributes.md)
+  * [Enabling Observability in zowe.yaml](enabling-observability-in-zowe.yaml.md)
+
+The following files will be presented under Using Zowe API Mediation Layer under the **Use** tab:
+
+* [Using your API ML OpenTelemetry metrics](using-your-otel-metrics.md)
+  * [API ML Provided Observability Signals and Attributes](apiml-provided-observability-signals-and-attributes.md)
+  * [Sample Output from API ML OpenTelemetry](sample-output-from-apiml-otel.md)

docs/user-guide/api-mediation/observability/sample-output-from-apiml-otel.md

@@ -0,0 +1,2 @@
+# Sample Output from API ML OpenTelemetry
+

docs/user-guide/api-mediation/observability/using-your-otel-metrics.md

@@ -0,0 +1,22 @@
+# Using Your API ML OpenTelemetry Metrics
+
+## Examples of Useability of Telemetry data in API ML
+
+How a system administrator interacts with this data depends on the visualization tool used (e.g., Grafana, Jaeger, or Broadcom WatchTower).
+
+### Example 1: High-Level Health Monitoring (Metrics)
+A system administrator views a Grafana dashboard. The administrator notices a spike in **`apiml.request.errors`**. 
+* **The View**: A red line graph shows a sudden jump from 0% to 15% error rate.
+* **The Insight**: By filtering the dashboard using the attribute **`zos.smf.id`**, the admin realizes the errors are only occurring on **LPAR1**, while **LPAR2** remains healthy. This suggests a local configuration or connectivity issue on a specific system rather than a global software bug.
+
+
+### Example 2: Latency Troubleshooting (Traces)
+A user reports that a specific API is "timing out." The admin finds the relevant **`traceId`** in the logs and opens it in a trace viewer.
+* **The View**: A "Gantt chart" style visualization of the request.
+* **The Insight**:
+    * `apiml.gateway.total`: 2005ms
+    * `apiml.auth.check`: 5ms
+    * `apiml.backend.proxy`: 2000ms
+* **The Action**: The admin sees that the Modulith itself only spent 5ms on logic, but waited 2 seconds for the backend mainframe service to respond. The admin can now confidently contact the specific backend service team.
+
+

sidebars.js

@@ -335,6 +335,16 @@ module.exports = {
                     "extend/extend-apiml/api-mediation-redis"
                   ]
                 },
+                {
+                  "type": "category",
+                  "label": "Configuring API ML Observability via OpenTelemetry",
+                  "items": [
+                    "user-guide/api-mediation/observability/configuring-otel-service-attributes",
+                    "user-guide/api-mediation/observability/configuring-otel-deployment-attributes",
+                    "user-guide/api-mediation/observability/configuring-otel-zos-attributes",
+					          "user-guide/api-mediation/observability/enabling-observability-in-zowe-yaml"
+                  ]
+                },
                 "user-guide/api-mediation/configuration-customizing-the-api-catalog-ui",
                 "user-guide/api-mediation/configuration-logging",
                 "user-guide/api-mediation/wto-message-on-startup",
@@ -527,6 +537,15 @@ module.exports = {
                 "user-guide/api-mediation-change-password-via-catalog",
               ],
             },
+            {
+              type: "category",
+              label: "Using your API ML OpenTelemetry metrics",
+              link: { "type": "doc", "id": "user-guide/api-mediation/observability/using-your-otel-metrics" },
+              items: [
+                "user-guide/api-mediation/observability/apiml-provided-observability-signals-and-attributes",
+                "user-guide/api-mediation/observability/sample-output-from-apiml-otel"
+              ],
+            },
             "user-guide/api-mediation/api-mediation-update-password",
             "user-guide/api-mediation/api-mediation-smf",
           ],