Merge remote-tracking branch 'upstream/main'

Author: dmitchsplunk

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 5.98
+current_version = 5.99
 commit = True
 Tag = True
 parse = v?(?P<major>\d+)\.(?P<minor>\d+)

README.md

@@ -22,5 +22,5 @@ Unless required by applicable law or agreed to in writing, software distributed
 To get started, please proceed to [The Splunk Observability Cloud Workshops Homepage](https://splunk.github.io/observability-workshop/latest/).
 
 Latest versions of the workshop are:
+- [v5.99](https://splunk.github.io/observability-workshop/v5.99/)
 - [v5.98](https://splunk.github.io/observability-workshop/v5.98/)
-- [v5.97](https://splunk.github.io/observability-workshop/v5.97/)

VERSION

@@ -1 +1 @@
-5.98
+5.99

content/en/conf/1-advanced-collector/1-agent-gateway/1-1-gateway.md

@@ -15,8 +15,9 @@ This makes your observability pipeline easier to manage, scale, and analyze—es
 Open or create your second terminal window and name it **Gateway**. Navigate to the first exercise directory `[WORKSHOP]/1-agent-gateway`
 then check the contents of the `gateway.yaml` file.
 
-This file outlines the core structure of the OpenTelemetry Collector as deployed in **Gateway** mode:
+This file outlines the core structure of the OpenTelemetry Collector as deployed in **Gateway** mode.
 
+<!--
 ```bash
  cat ./gateway.yaml
 ```
@@ -105,7 +106,7 @@ service:                          # Service configuration
       - debug                     # Debug exporter
       - file/logs
 ```
-
+-->
 {{% /notice %}}
 
 ### Understanding the Gateway Configuration
@@ -132,19 +133,20 @@ Let’s explore the `gateway.yaml` file that defines how the OpenTelemetry Colle
   The **Gateway** uses three file exporters to output telemetry data to local files. These exporters are defined as:
 
   ```yaml
-  exporters:
-    file/traces:
-      path: ./gateway-traces.out
-    file/metrics:
-      path: ./gateway-metrics.out
-    file/logs:
-      path: ./gateway-logs.out
+  exporters:                        # List of exporters
+    debug:                          # Debug exporter
+      verbosity: detailed           # Enable detailed debug output
+    file/traces:                    # Exporter Type/Name
+      path: "./gateway-traces.out"  # Path for OTLP JSON output for traces
+      append: false                 # Overwrite the file each time
+    file/metrics:                   # Exporter Type/Name
+      path: "./gateway-metrics.out" # Path for OTLP JSON output for metrics
+      append: false                 # Overwrite the file each time
+    file/logs:                      # Exporter Type/Name
+      path: "./gateway-logs.out"    # Path for OTLP JSON output for logs
+      append: false                 # Overwrite the file each time
   ```
 
-  Each exporter writes a specific signal type to its corresponding file:
-
-  * `gateway-traces.out`: stores span (trace) data
-  * `gateway-metrics.out`: stores metric data
-  * `gateway-logs.out`: stores log data
+  Each exporter writes a specific signal type to its corresponding file.
 
   These files are created once the gateway is started and will be populated with real telemetry as the agent sends data. You can monitor these files in real time to observe the flow of telemetry through your pipeline.

content/en/conf/1-advanced-collector/1-agent-gateway/1-2-send-metrics.md

@@ -1,26 +1,22 @@
 ---
-title: 1.2 Send Test Metrics
-linkTitle: 1.2 Send Metrics
+title: 1.2 Validate & Test Configuration
+linkTitle: 1.2 Validate & Test Configuration
 weight: 3
 ---
 
-Now, we can start the **Gateway** and the **Agent**, which is configured to automaticly send  **Host Metrics** at startup. We do this to  verify that data is properly routed from the **Agent** to the **Gateway**.
+Now, we can start the **Gateway** and the **Agent**, which is configured to automatically send **Host Metrics** at startup. We do this to verify that data is properly routed from the **Agent** to the **Gateway**.
 
 {{% notice title="Exercise" style="green" icon="running" %}}
 
-**Start the Gateway**: In the **Gateway terminal** window, run the following command to start the **Gateway**:
+**Gateway**: In the **Gateway terminal** window, run the following command to start the **Gateway**:
 
 ```bash {title="Start the Gateway"}
 ../otelcol --config=gateway.yaml
 ```
 
-If everything is configured correctly, the first and last lines of the output should look like:
+If everything is configured correctly, the collector will start and state `Everything is ready. Begin running and processing data.` in the output, similar to the following:
 
 ```text
-2025/06/09 09:22:11 settings.go:478: Set config to [gateway.yaml]
-...
-<snip to the end>
-...
 2025-06-09T09:22:11.944+0100    info    [email protected]/service.go:289 Everything is ready. Begin running and processing data. {"resource": {}}
 ```
 
@@ -43,22 +39,23 @@ Once the **Gateway** is running, it will listen for incoming data on port `5318`
 
 ```text
 <snip>
-NumberDataPoints #37
+NumberDataPoints #31
 Data point attributes:
-    -> cpu: Str(cpu0)
-    -> state: Str(system)
-StartTimestamp: 2024-12-09 14:18:28 +0000 UTC
-Timestamp: 2025-01-15 15:27:51.319526 +0000 UTC
-Value: 9637.660000
+     -> cpu: Str(cpu3)
+     -> state: Str(wait)
+StartTimestamp: 2025-07-07 16:49:42 +0000 UTC
+Timestamp: 2025-07-09 09:36:21.190226459 +0000 UTC
+Value: 77.380000
+        {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
 ```
 
-At this stage, the **Agent** continues to collect **CPU** metrics once per hour or upon each restart and sends them to the gateway. The **Gateway** processes these metrics and exports them to a file named `./gateway-metrics.out`. This file stores the exported metrics as part of the pipeline service.  
+At this stage, the **Agent** continues to collect **CPU** metrics once per hour or upon each restart and sends them to the gateway. The **Gateway** processes these metrics and exports them to a file named `gateway-metrics.out`. This file stores the exported metrics as part of the pipeline service.  
 
 **Verify Data arrived at Gateway**: To confirm that CPU metrics, specifically for `cpu0`, have successfully reached the gateway, we’ll inspect the `gateway-metrics.out` file using the `jq` command.
 
 The following command filters and extracts the `system.cpu.time` metric, focusing on `cpu0`. It displays the metric’s state (e.g., `user`, `system`, `idle`, `interrupt`) along with the corresponding values.
 
-Run the command below in the **Tests terminal** to check the `system.cpu.time` metric:
+Open or create your third terminal window and name it **Tests**. Run the command below in the **Tests terminal** to check the `system.cpu.time` metric:
 
 {{% tabs %}}
 {{% tab title="Check CPU Metrics" %}}
@@ -96,4 +93,7 @@ jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.ti
 {{% /tab %}}
 {{% /tabs %}}
 
+> [!IMPORTANT]
+> Stop the **Agent** and the **Gateway** processes by pressing `Ctrl-C` in their respective terminals.
+
 {{% /notice %}}

content/en/conf/1-advanced-collector/1-agent-gateway/_index.md

@@ -32,7 +32,7 @@ We will refer to these terminals as: **Agent**, **Gateway**, **Loadgen**, and **
     ├── agent.yaml
     └── gateway.yaml
     ```
-
+<!--
 3. Check the contents of the **agent.yaml** file. This file outlines the core structure of the OpenTelemetry Collector as deployed in **Agent** mode:
 
     ```bash
@@ -130,7 +130,7 @@ We will refer to these terminals as: **Agent**, **Gateway**, **Loadgen**, and **
           - file
           - otlphttp
     ```
-
+-->
 {{% /notice %}}
 
 ### Understanding the Agent configuration
@@ -179,18 +179,20 @@ The `receivers` section defines how the **Agent** ingests telemetry data. In thi
 
 #### Exporters
 
-* The `exporters` section controls where the collected telemetry data is sent:
+* **Debug Exporter**
 
   ```yaml
-  exporters:                             # Array of Exporters
     debug:                               # Exporter Type
       verbosity: detailed                # Enabled detailed debug output
+  ```
+
+* **OTLPHTTP Exporter**
+
+  ```yaml
     otlphttp:                            # Exporter Type
       endpoint: "http://localhost:5318"  # Gateway OTLP endpoint  
   ```
 
   The `debug` exporter sends data to the console for visibility and debugging during the workshop while the `otlphttp` exporter forwards all telemetry to the local **Gateway** instance.
 
-{{% notice title="Info" style="info" %}}
-This dual-export strategy ensures you can see the raw data locally while also sending it downstream for further processing and export.
-{{% /notice %}}
+  **This dual-export strategy ensures you can see the raw data locally while also sending it downstream for further processing and export.**

content/en/conf/1-advanced-collector/2-building-resilience/2-1-configuration.md

@@ -10,7 +10,18 @@ While these components do not process telemetry data directly, they provide valu
 
 {{% notice title="Exercise" style="green" icon="running" %}}
 
-**Update the `agent.yaml`**: In the **Agent terminal** window, add the `file_storage` extension and name it `checkpoint`:
+> [!IMPORTANT]
+> **Change _ALL_ terminal windows to the `2-building-resilience` directory and run the `clear` command.**
+
+Your directory structure will look like this:
+
+```text { title="Updated Directory Structure" }
+.
+├── agent.yaml
+└── gateway.yaml
+```
+
+**Update the `agent.yaml`**: In the **Agent terminal** window, add the `file_storage` extension under the existing `health_check` extension:
 
 ```yaml
   file_storage/checkpoint:             # Extension Type/Name
@@ -24,11 +35,9 @@ While these components do not process telemetry data directly, they provide valu
       max_transaction_size: 65536      # Max. size limit before compaction occurs
 ```
 
-**Add `file_storage` to existing `otlphttp` exporter**: Modify the `otlphttp:` exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur:
+**Add `file_storage` to the exporter**: Modify the `otlphttp` exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur. Add the following under the `endpoint: "http://localhost:5318"` and make sure the indentation matches `endpoint`:
 
 ```yaml
-  otlphttp: 
-    endpoint: "http://localhost:5318"
     retry_on_failure:
       enabled: true                    # Enable retry on failure
     sending_queue:                     # 
@@ -38,7 +47,7 @@ While these components do not process telemetry data directly, they provide valu
       storage: file_storage/checkpoint # File storage extension
 ```
 
-**Update the `services` section**: Add the `file_storage/checkpoint` extension to the existing `extensions:` section. This will cause the extension to be enabled:
+**Update the `services` section**: Add the `file_storage/checkpoint` extension to the existing `extensions:` section and the configuration needs to look like this:
 
 ```yaml
 service:
@@ -47,18 +56,18 @@ service:
   - file_storage/checkpoint            # Enabled extensions for this collector
 ```
 
-**Update the `metrics` pipeline**: For this exercise we are going to comment out the `hostmetrics` receiver from the Metric pipeline to reduce debug and log noise:
+**Update the `metrics` pipeline**: For this exercise we are going to comment out the `hostmetrics` receiver from the Metric pipeline to reduce debug and log noise, again the configuration needs to look like this:
 
 ```yaml
     metrics:
       receivers:
+      # - hostmetrics                    # Hostmetric reciever (cpu only)
       - otlp
-      # - hostmetrics                  # Hostmetrics Receiver
 ```
 
 {{% /notice %}}
 
-Validate the **Agent** configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `metrics:` section of your pipelines will look similar to this:
+<!-- Validate the **Agent** configuration using **[otelbin.io](https://www.otelbin.io/)**. For reference, the `metrics:` section of your pipelines will look similar to this:
 
 ```mermaid
 %%{init:{"fontFamily":"monospace"}}%%
@@ -76,16 +85,16 @@ graph LR
     subgraph " "
       subgraph subID1[**Metrics**]
       direction LR
-      REC1 --> PRO1
-      PRO1 --> PRO2
-      PRO2 --> PRO3
-      PRO3 --> PRO4
-      PRO4 --> EXP1
-      PRO4 --> EXP2
+      REC1 -- > PRO1
+      PRO1 -- > PRO2
+      PRO2 -- > PRO3
+      PRO3 -- > PRO4
+      PRO4 -- > EXP1
+      PRO4 -- > EXP2
       end
     end
 classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
 classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
 classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
 classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;
-```
+``` -->

content/en/conf/1-advanced-collector/2-building-resilience/2-2-test-environment.md

@@ -8,26 +8,25 @@ Next, we will configure our environment to be ready for testing the **File Stora
 
 {{% notice title="Exercise" style="green" icon="running" %}}
 
-**Start the Gateway**: In the **Gateway terminal** window navigate to the `[WORKSHOP]/4-resilience` directory and run:
+**Start the Gateway**: In the **Gateway terminal** window run:
 
 ```bash { title="Start the Gateway" }
 ../otelcol --config=gateway.yaml
 ```
 
-**Start the Agent**: In the **Agent terminal** window navigate to the `[WORKSHOP]/4-resilience` directory and run:
+**Start the Agent**: In the **Agent terminal** window run:
 
 ```bash { title="Start the Agent" }
 ../otelcol --config=agent.yaml
 ```
 
-**Send five test spans**: In the **Loadgen terminal** window navigate to the `[WORKSHOP]/4-resilience` directory and run:
+**Send five test spans**: In the **Loadgen terminal** window run:
 
 ```bash { title="Start Load Generator" }
 ../loadgen -count 5
 ```
 
 Both the **Agent** and **Gateway** should display debug logs, and the **Gateway** should create a `./gateway-traces.out` file.
 
-{{% /notice %}}
-
 If everything functions correctly, we can proceed with testing system resilience.
+{{% /notice %}}

content/en/conf/1-advanced-collector/2-building-resilience/2-3-failure.md

@@ -6,18 +6,12 @@ weight: 3
 
 To assess the **Agent's** resilience, we'll simulate a temporary **Gateway** outage and observe how the **Agent** handles it:
 
-**Summary**:
-
-1. **Send Traces to the Agent** – Generate traffic by sending traces to the **Agent**.
-2. **Stop the Gateway** – This will trigger the **Agent** to enter retry mode.
-3. **Restart the Gateway** – The **Agent** will recover traces from its persistent queue and forward them successfully. Without the persistent queue, these traces would have been lost permanently.
-
 {{% notice title="Exercise" style="green" icon="running" %}}
 
-**Simulate a network failure**: In the **Gateway terminal** stop the **Gateway** with `Ctrl-C` and wait until the gateway console shows that it has stopped:
+**Simulate a network failure**: In the **Gateway terminal** stop the **Gateway** with `Ctrl-C` and wait until the gateway console shows that it has stopped. The **Agent** will continue running, but it will not be able to send data to the gateway. The output in the **Gateway terminal** should look similar to this:
 
 ```text
-2025-01-28T13:24:32.785+0100  info  service@v0.120.0/service.go:309  Shutdown complete.
+2025-07-09T10:22:37.941Z        info    service@v0.126.0/service.go:345 Shutdown complete.      {"resource": {}}
 ```
 
 **Send traces**: In the **Loadgen terminal** window send five more traces using the `loadgen`.
@@ -31,16 +25,13 @@ Notice that the agent’s retry mechanism is activated as it continuously attemp
 **Stop the Agent**: In the **Agent terminal** window, use `Ctrl-C` to stop the agent. Wait until the agent’s console confirms it has stopped:
 
 ```text
-2025-01-28T14:40:28.702+0100  info  extensions/extensions.go:66  Stopping extensions...
-2025-01-28T14:40:28.702+0100  info  [email protected]/service.go:309  Shutdown complete.
+2025-07-09T10:25:59.344Z        info    [email protected]/service.go:345 Shutdown complete.      {"resource": {}}
 ```
 
 {{% /notice %}}
 
-{{% notice title="Tip" style="primary" icon="lightbulb" %}}
-Stopping the agent will halt its retry attempts and prevent any future retry activity.
+By stopping the agent will halt its retry attempts and prevent any future retry activity.
 
 If the agent runs for too long without successfully delivering data, it may begin dropping traces, depending on the retry configuration, to conserve memory. By stopping the agent, any metrics, traces, or logs currently stored in memory are lost before being dropped, ensuring they remain available for recovery.
 
 This step is essential for clearly observing the recovery process when the agent is restarted.
-{{% /notice %}}

content/en/conf/1-advanced-collector/2-building-resilience/_index.md

@@ -16,18 +16,3 @@ This solution will work for metrics as long as the connection downtime is brief
 For logs, there are plans to implement a more enterprise-ready solution in one of the upcoming Splunk OpenTelemetry Collector releases.
 
 {{% /notice %}}
-
-{{% notice title="Exercise" style="green" icon="running" %}}
-
-> [!IMPORTANT]
-> **Change _ALL_ terminal windows to the `[WORKSHOP]/2-building-resilience` directory.**
-
-Your directory structure will look like this:
-
-```text { title="Updated Directory Structure" }
-.
-├── agent.yaml
-└── gateway.yaml
-```
-
-{{% /notice %}}