moved pet clinic as preparation for intro to instrumentation

Author: hagen-p

content/en/other/s4r/pet-clinic/1-otel-collector.md

@@ -0,0 +1,76 @@
+---
+title: Installing the OpenTelemetry Collector
+linkTitle: 1. OpenTelemetry Collector
+weight: 1
+---
+
+## 1. Introduction
+
+The OpenTelemetry Collector is the core component of instrumenting infrastructure and applications.  Its role is to collect and send:
+
+* Infrastructure metrics (disk, cpu, memory, etc)
+* Application Performance Monitoring (APM) traces
+* Profiling data
+* Host and application logs
+
+Splunk Observability Cloud offers a wizard to walk you through the setup of the Collector on both your infrastructure and applications.
+
+{{% notice title="Delete any existing OpenTelemetry Collectors" style="warning" %}}
+If you have completed the Splunk IM workshop, please ensure you have deleted the collector running in Kubernetes before continuing. This can be done by running the following command:
+
+``` bash
+helm delete splunk-otel-collector
+```
+
+{{% /notice %}}
+
+## 2. Confirm environment variables
+
+To ensure your instance is configured correctly, we need to confirm that the required environment variables for this workshop are set correctly. In your terminal run the following command:
+
+``` bash
+env
+```
+
+In the output check the following environment variables are present and have values set:
+
+```text
+ACCESS_TOKEN
+REALM
+RUM_TOKEN
+```
+
+For this workshop, **all** of the above are required. If any are missing, please contact your instructor.
+
+## 3. Install the OpenTelemetry Collector
+
+We can then go ahead and install the Collector. Some additional parameters are passed to the install script, they are:
+
+* `--with-instrumentation` - This will install the agent from the Splunk distribution of OpenTelemetry Java, which is then loaded automatically when the PetClinic Java application starts up. No configuration is required!
+* `--deployment-environment` - Sets the resource attribute `deployment.environment` to the value passed. This is used to filter views in the UI.
+* `--enable-profiler` - Enables the profiler for the Java application. This will generate CPU profiles for the application.
+* `--enable-profiler-memory` - Enables the profiler for the Java application. This will generate memory profiles for the application.
+* `--enable-metrics` - Enables the exporting of Micrometer metrics
+
+``` bash
+curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
+sudo sh /tmp/splunk-otel-collector.sh --realm $REALM -- $ACCESS_TOKEN --mode agent --without-fluentd --with-instrumentation --deployment-environment $INSTANCE-petclinic --enable-profiler --enable-profiler-memory --enable-metrics
+```
+
+{{% notice style="info" title="AWS/EC2 instances" %}}
+If you are attempting this workshop on an AWS/EC2 instance you will have to patch the collector to expose the hostname of the instance:
+
+``` bash
+sudo sed -i 's/gcp, ecs, ec2, azure, system/system, gcp, ecs, ec2, azure/g' /etc/otel/collector/agent_config.yaml
+```
+
+Once the `agent_config.yaml` has been patched, you will need to restart the collector:
+
+``` bash
+sudo systemctl restart splunk-otel-collector
+```
+
+{{% /notice %}}
+Once the installation is completed, you can navigate to the **Hosts with agent installed** dashboard to see the data from your host, **Dashboards → Hosts with agent installed**.
+
+Use the dashboard filter and select `host.name` and type or select the hostname of your virtual machine. Once you see data flowing for your host, we are then ready to get started with the APM component.

content/en/other/s4r/pet-clinic/2-zero-configuration.md

@@ -0,0 +1,111 @@
+---
+title: Zero Configuration Auto Instrumentation for Java
+linkTitle: 2. Zero Configuration
+weight: 2
+---
+
+## 1. Spring PetClinic Application
+
+The first thing we need to set up APM is... well, an application. For this exercise, we will use the Spring PetClinic application. This is a very popular sample Java application built with Spring framework (Springboot).
+
+First, clone the PetClinic GitHub repository, and then we will compile, build, package and test the application:
+
+```bash
+git clone https://github.com/spring-projects/spring-petclinic
+```
+
+Change into the `spring-petclinic` directory (and checkout a specific commit):
+
+```bash
+cd spring-petclinic && git checkout 276880e
+```
+
+Start a MySQL database for PetClinic to use:
+
+```bash
+docker run -d -e MYSQL_USER=petclinic -e MYSQL_PASSWORD=petclinic -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=petclinic -p 3306:3306 docker.io/mysql:5.7.8
+```
+
+Next, we will start a Docker container running Locust that will generate some simple traffic to the PetClinic application. Locust is a simple load-testing tool that can be used to generate traffic to a web application.
+
+```bash
+docker run --network="host" -d -p 8090:8090 -v /home/ubuntu/workshop/petclinic:/mnt/locust docker.io/locustio/locust -f /mnt/locust/locustfile.py --headless -u 1 -r 1 -H http://127.0.0.1:8083
+```
+
+Next, run the `maven` command to compile/build/package PetClinic:
+
+```bash
+./mvnw package -Dmaven.test.skip=true
+```
+
+{{% notice style="info" %}}
+This will take a few minutes the first time you run, `maven` will download a lot of dependencies before it compiles the application. Future executions will be a lot quicker.
+{{% /notice %}}
+
+Once the compilation is complete, you can run the application with the following command. Notice that we are passing the `mysql` profile to the application, this will tell the application to use the MySQL database we started earlier. We are also setting the `otel.service.name` to a logical service name that will also be used in the UI for filtering:
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$(hostname)-petclinic-service \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+You can validate if the application is running by visiting `http://<VM_IP_ADDRESS>:8083`.
+
+## 2. AlwaysOn Profiling and Metrics
+
+When we installed the collector we configured it to enable AlwaysOn Profiling and Metrics. This means that the collector will automatically generate CPU and Memory profiles for the application and send them to Splunk Observability Cloud.
+
+When you start the PetClinic application you will see the collector automatically detect the application and instrument it for traces and profiling.
+
+{{% tab title="Example output" %}}
+
+``` text {wrap="false"}
+Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/lib/splunk-instrumentation/splunk-otel-javaagent.jar -Dsplunk.profiler.enabled=true -Dsplunk.profiler.memory.enabled=true
+OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
+[otel.javaagent 2023-06-26 13:32:04:661 +0100] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: splunk-1.25.0-otel-1.27.0
+[otel.javaagent 2023-06-26 13:32:05:294 +0100] [main] INFO com.splunk.javaagent.shaded.io.micrometer.core.instrument.push.PushMeterRegistry - publishing metrics for SignalFxMeterRegistry every 30s
+[otel.javaagent 2023-06-26 13:32:07:043 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger - -----------------------
+[otel.javaagent 2023-06-26 13:32:07:044 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger - Profiler configuration:
+[otel.javaagent 2023-06-26 13:32:07:047 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -                  splunk.profiler.enabled : true
+[otel.javaagent 2023-06-26 13:32:07:048 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -                splunk.profiler.directory : /tmp
+[otel.javaagent 2023-06-26 13:32:07:049 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -       splunk.profiler.recording.duration : 20s
+[otel.javaagent 2023-06-26 13:32:07:050 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -               splunk.profiler.keep-files : false
+[otel.javaagent 2023-06-26 13:32:07:051 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -            splunk.profiler.logs-endpoint : null
+[otel.javaagent 2023-06-26 13:32:07:053 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -              otel.exporter.otlp.endpoint : null
+[otel.javaagent 2023-06-26 13:32:07:054 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -           splunk.profiler.memory.enabled : true
+[otel.javaagent 2023-06-26 13:32:07:055 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -             splunk.profiler.tlab.enabled : true
+[otel.javaagent 2023-06-26 13:32:07:056 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -        splunk.profiler.memory.event.rate : 150/s
+[otel.javaagent 2023-06-26 13:32:07:057 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -      splunk.profiler.call.stack.interval : PT10S
+[otel.javaagent 2023-06-26 13:32:07:058 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -  splunk.profiler.include.internal.stacks : false
+[otel.javaagent 2023-06-26 13:32:07:059 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -      splunk.profiler.tracing.stacks.only : false
+[otel.javaagent 2023-06-26 13:32:07:059 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger - -----------------------
+[otel.javaagent 2023-06-26 13:32:07:060 +0100] [main] INFO com.splunk.opentelemetry.profiler.JfrActivator - Profiler is active.
+```
+
+{{% /tab %}}
+
+## 3. Review Profiling Data Collection
+
+You can now visit the Splunk APM UI and examine the application components, traces, profiling, DB Query performance and metrics. From the left-hand menu **APM** → Explore**, click the environment dropdown and select your environment (which is prefixed with the hostname of your instance).
+
+![APM Environment](../images/apm-environment.png)
+
+Once your validation is complete you can stop the application by pressing `Ctrl-c`.
+
+## 4. Adding Resource Attributes to Spans
+
+Resource attributes can be added to every reported span. For example `version=0.314`. A comma-separated list of resource attributes can also be defined e.g. `key1=val1,key2=val2`.
+
+Let's launch the PetClinic again using a new resource attribute. Note, that adding resource attributes to the run command will override what was defined when we installed the collector. Let's add a new resource attribute `version=0.314`:
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$INSTANCE-petclinic-service \
+-Dotel.resource.attributes=version=0.314 \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+Back in the Splunk APM UI we can drill down on a recent trace and see the new `version` attribute in a span.

content/en/other/s4r/pet-clinic/3-rum.md

@@ -0,0 +1,64 @@
+---
+title: 3. Real User Monitoring
+weight: 3
+---
+
+## 1. Enable RUM
+
+For the Real User Monitoring (RUM) instrumentation, we will add the Open Telemetry Javascript [https://github.com/signalfx/splunk-otel-js-web](https://github.com/signalfx/splunk-otel-js-web) snippet in the pages, we will use the wizard again **Data Management → Add Integration → RUM Instrumentation → Browser Instrumentation**.
+
+Select the preconfigured **RUM ACCESS TOKEN** from the dropdown, and click **Next**. Enter **App name** and **Environment** using the following syntax:
+
+- `[hostname]-petclinic-service` - replacing `[hostname]` with your actual hostname.
+- `[hostname]-petclinic-env` - replacing `[hostname]` with your actual hostname.
+
+Then you'll need to select the workshop RUM token and define the application and environment names. The wizard will then show a snippet of HTML code that needs to be placed at the top of the pages in the `<head>` section. In this example, we are using:
+
+- Application Name: `<hostname>-petclinic-service`
+- Environment: `<hostname>-petclinic-env`
+
+Copy the generated code snippet in the wizard or copy and edit the snippet below accordingly. You need to replace `<REALM>`, `<RUM_ACCESS_TOKEN>` and `<hostname>` with the actual values.
+
+``` html
+<script src="https://cdn.signalfx.com/o11y-gdi-rum/latest/splunk-otel-web.js" crossorigin="anonymous"></script>
+<script>
+SplunkRum.init({
+    beaconUrl: "https://rum-ingest.<REALM>.signalfx.com/v1/rum",
+    rumAuth: "<RUM_ACCESS_TOKEN>",
+    app: "<hostname>-petclinic-service",
+    environment: "<hostname>-petclinic-env"
+    });
+</script>
+```
+
+The Spring PetClinic application uses a single HTML page as the "layout" page, that is reused across all pages of the application. This is the perfect location to insert the Splunk RUM Instrumentation Library as it will be loaded in all pages automatically.
+
+Let's then edit the layout page:
+
+```bash
+vi src/main/resources/templates/fragments/layout.html
+```
+
+Next, insert the snippet we generated above in the `<head>` section of the page. Now we need to rebuild the application and run it again:
+
+## 2. Rebuild PetClinic
+
+Run the `maven` command to compile/build/package PetClinic:
+
+```bash
+./mvnw package -Dmaven.test.skip=true
+```
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$(hostname)-petclinic-service \
+-Dotel.resource.attributes=version=0.314 \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+Then let's visit the application using a browser to generate real-user traffic `http://<VM_IP_ADDRESS>:8083`, now we should see RUM traces being reported.
+
+Let's visit RUM and see some of the traces and metrics **Hamburger Menu → RUM** and you will see some of the Spring PetClinic URLs showing up in the UI.
+
+When you drill down into a RUM trace you will see a link to APM in the spans. Clicking on the trace ID will take you to the corresponding APM trace for the current RUM trace.

content/en/other/s4r/pet-clinic/4-log-observer-connect.md

@@ -0,0 +1,110 @@
+---
+title: 4. Log Observer
+weight: 4
+---
+
+## 1. Introduction
+
+For the Splunk Log Observer component, we will configure the Spring PetClinic application to write logs to a file in the filesystem and configure the Splunk OpenTelemetry Collect to read (tail) that log file and report the information to the Splunk Observability Platform.
+
+## 2. OpenTelemetry Filelog Configuration
+
+We need to configure the Splunk OpenTelemetry Collector to tail the Spring PetClinic log file and report the data to the Splunk Cloud HEC URL.
+
+The Splunk OpenTelemetry Collector uses the [Filelog Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md) to consume logs. We will need to edit the collectors' configuration file:
+
+``` bash
+sudo vi /etc/otel/collector/agent_config.yaml
+```
+
+Under `receivers:` create the Filelog Receiver (make sure to indent correctly):
+
+``` yaml {hl_lines="2-3"}
+receivers:
+  filelog:
+    include: [/home/ubuntu/spring-petclinic.log]
+```
+
+The under the `service:` section, find the `logs:` pipeline, replace `fluentforward` with`filelog` and remove `otlp`(again, make sure to indent correctly):
+
+``` yaml {hl_lines="2-7"}
+    logs:
+      receivers: [filelog]
+```
+
+Save the file and exit the editor. Next, we need to validate that the HEC Token and HEC URL are configured for the collector to use. We will inspect the `/etc/otel/collector/splunk-otel-collector.conf` file:
+
+```bash
+sudo cat /etc/otel/collector/splunk-otel-collector.conf
+```
+
+Make sure `SPLUNK_HEC_URL` and `SPLUNK_HEC_TOKEN` have values set, we can then restart the collector to apply the changes:
+
+``` bash
+sudo systemctl restart splunk-otel-collector
+```
+
+## 3. Logback Settings
+
+The Spring PetClinic application can be configured to use several different Java logging libraries. In this scenario, we are going to use logback. We just need to create a file named `logback.xml` in the configuration folder:
+
+```bash
+vi ~/spring-petclinic/src/main/resources/logback.xml
+```
+
+Copy and paste the following XML content:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE xml>
+<configuration scan="true" scanPeriod="30 seconds">
+  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
+      <resetJUL>true</resetJUL>
+  </contextListener>
+  <logger name="org.springframework.samples.petclinic" level="info"/>
+  <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
+    <file>/home/ubuntu/spring-petclinic.log</file>
+    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+      <fileNamePattern>springLogFile.%d{yyyy-MM-dd}.log</fileNamePattern>
+      <maxHistory>5</maxHistory>
+      <totalSizeCap>1GB</totalSizeCap>
+    </rollingPolicy>
+    <encoder>
+      <pattern>
+        %d{yyyy-MM-dd HH:mm:ss} - %logger{36} - %msg trace_id=%X{trace_id} span_id=%X{span_id} trace_flags=%X{trace_flags} %n service.name=%property{otel.resource.service.name}, deployment.environment=%property{otel.resource.deployment.environment}: %m%n
+      </pattern>
+    </encoder>
+  </appender>
+  <root level="info">
+    <appender-ref ref="file" />
+  </root>
+</configuration>
+```
+
+Now we need to rebuild the application and run it again:
+
+```bash
+./mvnw package -Dmaven.test.skip=true
+```
+
+Once the rebuild has been completed we can then run the application again:
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$(hostname)-petclinic-service \
+-Dotel.resource.attributes=version=0.314 \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+## 4. View Logs
+
+From the left-hand menu click on **Log Observer**. Click on **Index** and select **o11y-workshop-XXX.splunkcloud.com** (where **XXX** will be the realm you are running in). On the right-hand side select **petclinic-workshop** and then click **Apply**. You should see log messages being reported.
+
+Next, click **Add Filter** search for the field `service_name` select the value `<your host name>-petclinic-service` and click `=` (include). You should now see only the log messages from your PetClinic application.
+
+![Log Observer](../images/log-observer.png)
+
+## 4. Summary
+
+This is the end of the exercise and we have certainly covered a lot of ground. At this point, you should have metrics, traces (APM & RUM), logs, database query performance and code profiling being reported into Splunk Observability Cloud. **Congratulations**!