AppD and fixing S4R

Author: rcastley

content/en/other/12-appdynamics/1-petclinic/1-build-petclinic.md

@@ -0,0 +1,127 @@
+---
+title: Building the PetClinic Application
+linkTitle: 1. Build PetClinic
+weight: 1
+---
+
+## 1. Spring PetClinic Application
+
+The first thing we need to set up APM is... well, an application. For this exercise, we will use the Spring PetClinic application. This is a very popular sample Java application built with the Spring framework (Springboot).
+
+First, clone the PetClinic GitHub repository, and then we will compile, build, package and test the application:
+
+```bash
+git clone https://github.com/spring-projects/spring-petclinic
+```
+
+Change into the `spring-petclinic` directory (and checkout a specific commit):
+
+```bash
+cd spring-petclinic && git checkout 276880e
+```
+
+Start a MySQL database for PetClinic to use:
+
+```bash
+docker run -d -e MYSQL_USER=petclinic -e MYSQL_PASSWORD=petclinic -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=petclinic -p 3306:3306 docker.io/biarms/mysql:5.7
+```
+
+Next, we will start a Docker container running Locust that will generate some simple traffic to the PetClinic application. Locust is a simple load-testing tool that can be used to generate traffic to a web application.
+
+```bash
+docker run --network="host" -d -p 8090:8090 -v ~/workshop/petclinic:/mnt/locust docker.io/locustio/locust -f /mnt/locust/locustfile.py --headless -u 1 -r 1 -H http://127.0.0.1:8083
+```
+
+Next, run the `maven` command to compile/build/package PetClinic:
+
+```bash
+./mvnw package -Dmaven.test.skip=true
+```
+
+{{% notice style="info" %}}
+This will take a few minutes the first time you run, `maven` will download a lot of dependencies before it compiles the application. Future builds will be a lot quicker.
+{{% /notice %}}
+
+Once the build completes, you need to obtain the public IP address of the instance you are running on. You can do this by running the following command:
+
+```bash
+curl ifconfig.me
+```
+
+You will see an IP address returned, make a note of this as we will need it to validate that the application is running.
+
+We also need to obtain the `INSTANCE` environment variable value, as this is what is being used as the `otel.service.name` attribute. You can do this by running the following command:
+
+```bash
+echo $INSTANCE
+```
+
+Also, make a note of this value as we will need it to filter the data in the UI.
+
+You can now run the application with the following command. Notice that we are passing the `mysql` profile to the application, this will tell the application to use the MySQL database we started earlier. We are also setting the `otel.service.name` to a logical service name that will also be used in the UI for filtering:
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$INSTANCE-petclinic-service \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+You can validate if the application is running by visiting `http://<IP_ADDRESS>:8083` (replace `<IP_ADDRESS>` with the IP address you obtained earlier). You should see the PetClinic application running.
+
+## 2. AlwaysOn Profiling and Metrics
+
+When we installed the collector we configured it to enable **AlwaysOn Profiling** and **Metrics**. This means that the collector will automatically generate CPU and Memory profiles for the application and send them to Splunk Observability Cloud.
+
+When you start the PetClinic application you will see the collector automatically detect the application and instrument it for traces and profiling.
+
+{{% tab title="Example output" %}}
+
+``` text {wrap="false"}
+Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/lib/splunk-instrumentation/splunk-otel-javaagent.jar -Dsplunk.profiler.enabled=true -Dsplunk.profiler.memory.enabled=true
+OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
+[otel.javaagent 2023-06-26 13:32:04:661 +0100] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: splunk-1.25.0-otel-1.27.0
+[otel.javaagent 2023-06-26 13:32:05:294 +0100] [main] INFO com.splunk.javaagent.shaded.io.micrometer.core.instrument.push.PushMeterRegistry - publishing metrics for SignalFxMeterRegistry every 30s
+[otel.javaagent 2023-06-26 13:32:07:043 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger - -----------------------
+[otel.javaagent 2023-06-26 13:32:07:044 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger - Profiler configuration:
+[otel.javaagent 2023-06-26 13:32:07:047 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -                  splunk.profiler.enabled : true
+[otel.javaagent 2023-06-26 13:32:07:048 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -                splunk.profiler.directory : /tmp
+[otel.javaagent 2023-06-26 13:32:07:049 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -       splunk.profiler.recording.duration : 20s
+[otel.javaagent 2023-06-26 13:32:07:050 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -               splunk.profiler.keep-files : false
+[otel.javaagent 2023-06-26 13:32:07:051 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -            splunk.profiler.logs-endpoint : null
+[otel.javaagent 2023-06-26 13:32:07:053 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -              otel.exporter.otlp.endpoint : null
+[otel.javaagent 2023-06-26 13:32:07:054 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -           splunk.profiler.memory.enabled : true
+[otel.javaagent 2023-06-26 13:32:07:055 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -             splunk.profiler.tlab.enabled : true
+[otel.javaagent 2023-06-26 13:32:07:056 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -        splunk.profiler.memory.event.rate : 150/s
+[otel.javaagent 2023-06-26 13:32:07:057 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -      splunk.profiler.call.stack.interval : PT10S
+[otel.javaagent 2023-06-26 13:32:07:058 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -  splunk.profiler.include.internal.stacks : false
+[otel.javaagent 2023-06-26 13:32:07:059 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger -      splunk.profiler.tracing.stacks.only : false
+[otel.javaagent 2023-06-26 13:32:07:059 +0100] [main] INFO com.splunk.opentelemetry.profiler.ConfigurationLogger - -----------------------
+[otel.javaagent 2023-06-26 13:32:07:060 +0100] [main] INFO com.splunk.opentelemetry.profiler.JfrActivator - Profiler is active.
+```
+
+{{% /tab %}}
+
+## 3. Review Profiling Data Collection
+
+You can now visit the Splunk APM UI and examine the application components, traces, profiling, DB Query performance and metrics. From the left-hand menu **APM** → **Explore**, click the environment dropdown and select your environment e.g. `<INSTANCE>-petclinic` (where`<INSTANCE>` is replaced with the value you noted down earlier).
+
+![APM Environment](../images/apm-environment.png)
+
+Once your validation is complete you can stop the application by pressing `Ctrl-c`.
+
+## 4. Adding Resource Attributes to Spans
+
+Resource attributes can be added to every reported span. For example `version=0.314`. A comma-separated list of resource attributes can also be defined e.g. `key1=val1,key2=val2`.
+
+Let's launch the PetClinic again using new resource attributes. Note, that adding resource attributes to the run command will override what was defined when we installed the collector. Let's add two new resource attributes `deployment.environment=$INSTANCE-petclinic-env,version=0.314`:
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$INSTANCE-petclinic-service \
+-Dotel.resource.attributes=deployment.environment=$INSTANCE-petclinic-env,version=0.314 \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+Back in the Splunk APM UI we can drill down on a recent trace and see the new `version` attribute in a span.

content/en/other/12-appdynamics/1-petclinic/1-otel-collector.md

@@ -0,0 +1,41 @@
+---
+title: Installing the AppDynamics Agent
+linkTitle: 2. AppDynamics Agent
+weight: 2
+---
+
+## 1. Introduction
+
+## 2. Install AppDynamics Agent
+
+- Login to the AppDynamics Controller.
+- Under the **Home** menu, select **Getting Started**.
+- Click **Getting Started Wizard**.
+- Click **Java**.
+- Ensure the **JVM** is **JDK8+**.
+- Create a new application and enter application name. Please use the syntax `<initials>-petclinic-<date>` e.g. `rwc-petclinic-13-06-2024`.
+- Create a new tier and enter tier name. Please use the syntax `<initials>-petclinic-tier` e.g. `rwc-petclinic-tier`.
+- Enter a new Node Name. Please use the syntax `<initials>-petclinic-node` e.g. `rwc-petclinic-node`.
+- Click **Continue**.
+- Secure copy the zip file to your workshop instance.
+
+    ```bash
+    scp -P 2222 ~/Downloads/AppServerAgent-1.8-24.5.0.36037.zip [email protected]:/home/splunk
+    ```
+
+- SSH into the workshop instance.
+
+- `mkdir appd`
+- `unzip AppServerAgent-1.8-24.5.0.36037.zip -d opt/appdynamics`
+- `cd ~/spring-petclinic`
+- Start PetClinic with the AppDynamics agent.
+
+    ```bash
+    java \
+    -Dserver.port=8081 \
+    -javaagent:/home/splunk/appd/appagent/javaagent.jar \
+    -Dappdynamics.agent.applicationName="rwc-petclinic-13-06-2024"\
+    -Dappdynamics.agent.tierName="rwc-petclinic-tier" \
+    -Dappdynamics.agent.nodeName="rwc-petclinic-node" \
+    -jar target/spring-petclinic-*.jar
+    ```

content/en/other/12-appdynamics/1-petclinic/3-rum.md

@@ -0,0 +1,82 @@
+---
+title: 3. Real User Monitoring
+weight: 3
+---
+
+## 1. Enable RUM
+
+For the Real User Monitoring (RUM) instrumentation, we will add the Open Telemetry Javascript [**https://github.com/signalfx/splunk-otel-js-web**](https://github.com/signalfx/splunk-otel-js-web) snippet in the pages, we will use the wizard again **Data Management → Add Integration → RUM Instrumentation → Browser Instrumentation**.
+
+Your instructor will inform you which token to use from the dropdown, click **Next**. Enter **App name** and **Environment** using the following syntax:
+
+- `<INSTANCE>-petclinic-service` - replacing `<INSTANCE>` with the value you noted down earlier.
+- `<INSTANCE>-petclinic-env` - replacing `<INSTANCE>` with the value you noted down earlier.
+
+The wizard will then show a snippet of HTML code that needs to be placed at the top of the pages in the `<head>` section. The following is an example of the  (do not use this snippet, use the one generated by the wizard):
+
+``` html
+/*
+
+IMPORTANT: Replace the <version> placeholder in the src URL with a
+version from https://github.com/signalfx/splunk-otel-js-web/releases
+
+*/
+<script src="https://cdn.signalfx.com/o11y-gdi-rum/latest/splunk-otel-web.js" crossorigin="anonymous"></script>
+<script>
+    SplunkRum.init({
+        realm: "eu0",
+        rumAccessToken: "<redacted>",
+        applicationName: "petclinic-1be0-petclinic-service",
+        deploymentEnvironment: "petclinic-1be0-petclinic-env"
+    });
+</script>
+```
+
+The Spring PetClinic application uses a single HTML page as the "layout" page, that is reused across all pages of the application. This is the perfect location to insert the Splunk RUM Instrumentation Library as it will be loaded in all pages automatically.
+
+Let's then edit the layout page:
+
+```bash
+vi src/main/resources/templates/fragments/layout.html
+```
+
+Next, insert the snippet we generated above in the `<head>` section of the page. Make sure you don't include the comment and replace `<version>` in the source URL to `latest` e.g.
+
+```html
+<!doctype html>
+<html th:fragment="layout (template, menu)">
+
+<head>
+<script src="https://cdn.signalfx.com/o11y-gdi-rum/latest/splunk-otel-web.js" crossorigin="anonymous"></script>
+<script>
+    SplunkRum.init({
+        realm: "eu0",
+        rumAccessToken: "<redacted>",
+        applicationName: "petclinic-1be0-petclinic-service",
+        deploymentEnvironment: "petclinic-1be0-petclinic-env"
+    });
+</script>
+...
+```
+
+## 2. Rebuild PetClinic
+
+With the code changes complete, we need to rebuild the application and run it again. Run the `maven` command to compile/build/package PetClinic:
+
+```bash
+./mvnw package -Dmaven.test.skip=true
+```
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$INSTANCE-petclinic-service \
+-Dotel.resource.attributes=deployment.environment=$INSTANCE-petclinic-env,version=0.314 \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+Then let's visit the application using a browser to generate real-user traffic `http://<IP_ADDRESS>:8083`.
+
+In RUM, filter down into the environment as defined in the RUM snippet above and click through to the dashboard.
+
+When you drill down into a RUM trace you will see a link to APM in the spans. Clicking on the trace ID will take you to the corresponding APM trace for the current RUM trace.

content/en/other/12-appdynamics/1-petclinic/4-log-observer-connect.md

@@ -0,0 +1,112 @@
+---
+title: 4. Log Observer
+weight: 4
+---
+
+## 1. Introduction
+
+For the Splunk Log Observer component, we will configure the Spring PetClinic application to write logs to a file and configure the Splunk OpenTelemetry Collector to read (tail) that log file and send the logs to Splunk Cloud.
+
+## 2. OpenTelemetry Filelog Configuration
+
+We need to configure the Splunk OpenTelemetry Collector to tail the Spring PetClinic log file and send the log data to Splunk Cloud.
+
+The Splunk OpenTelemetry Collector uses **Fluentd** by default but we will change the configuration to use the OpenTelemetry [**Filelog Receiver**](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md) to consume the logs. We will need to edit the collectors' configuration file:
+
+``` bash
+sudo vi /etc/otel/collector/agent_config.yaml
+```
+
+Under the `receivers:` section create the Filelog Receiver (make sure to indent correctly):
+
+``` yaml {hl_lines="2-3"}
+receivers:
+  filelog:
+    include: [/tmp/spring-petclinic.log]
+```
+
+Then under the `service:` section, find the `logs:` pipeline, replace `fluentforward` with`filelog` (again, make sure to indent correctly):
+
+``` yaml {hl_lines="2-7"}
+    logs:
+      receivers: [filelog, otlp]
+```
+
+Save the file and exit the editor. Next, we need to validate that the HEC Token and HEC URL are configured for the collector to use. We will inspect the `/etc/otel/collector/splunk-otel-collector.conf` file:
+
+```bash
+sudo cat /etc/otel/collector/splunk-otel-collector.conf
+```
+
+Make sure `SPLUNK_HEC_URL` and `SPLUNK_HEC_TOKEN` have values set. If they are not set, please reach out to your instructor. With the configuration change complete and validated, we can now restart the collector:
+
+```bash
+sudo systemctl restart splunk-otel-collector
+```
+
+## 3. Logback Settings
+
+The Spring PetClinic application can be configured to use several different Java logging libraries. In this scenario, we are going to use `logback`. We just need to create a file named `logback.xml` in the configuration folder:
+
+```bash
+vi ~/spring-petclinic/src/main/resources/logback.xml
+```
+
+Copy and paste the following XML content:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE xml>
+<configuration scan="true" scanPeriod="30 seconds">
+  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
+      <resetJUL>true</resetJUL>
+  </contextListener>
+  <logger name="org.springframework.samples.petclinic" level="info"/>
+  <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
+    <file>/tmp/spring-petclinic.log</file>
+    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+      <fileNamePattern>springLogFile.%d{yyyy-MM-dd}.log</fileNamePattern>
+      <maxHistory>5</maxHistory>
+      <totalSizeCap>1GB</totalSizeCap>
+    </rollingPolicy>
+    <encoder>
+      <pattern>
+        %d{yyyy-MM-dd HH:mm:ss} - %logger{36} - %msg trace_id=%X{trace_id} span_id=%X{span_id} trace_flags=%X{trace_flags} %n service.name=%property{otel.resource.service.name}, deployment.environment=%property{otel.resource.deployment.environment}: %m%n
+      </pattern>
+    </encoder>
+  </appender>
+  <root level="info">
+    <appender-ref ref="file" />
+  </root>
+</configuration>
+```
+
+Now we need to rebuild the application and run it again:
+
+```bash
+./mvnw package -Dmaven.test.skip=true
+```
+
+Once the rebuild has been completed we can then run the application again:
+
+```bash
+java \
+-Dserver.port=8083 \
+-Dotel.service.name=$INSTANCE-petclinic-service \
+-Dotel.resource.attributes=deployment.environment=$INSTANCE-petclinic-env,version=0.314 \
+-jar target/spring-petclinic-*.jar --spring.profiles.active=mysql
+```
+
+## 4. View Logs
+
+From the left-hand menu click on **Log Observer** and ensure **Index** is set to **splunk4rookies-workshop**.
+
+Next, click **Add Filter** search for the field `service_name` select the value `<INSTANCE>-petclinic-service` and click `=` (include). You should now see only the log messages from your PetClinic application.
+
+![Log Observer](../images/log-observer.png)
+
+## 4. Summary
+
+This is the end of the workshop and we have certainly covered a lot of ground. At this point, you should have metrics, traces (APM & RUM), logs, database query performance and code profiling being reported into Splunk Observability Cloud.
+
+**Congratulations!**

content/en/other/12-appdynamics/1-petclinic/_index.md

@@ -0,0 +1,19 @@
+---
+title: PetClinic Java Workshop
+weight: 1
+description: Instrumenting PetClinic using AppDynamics.
+authors: ["Robert Castley"]
+time: 30 minutes
+---
+
+We will also show the steps about how to clone (download) a sample Java application (Spring PetClinic), as well as how to compile, package and run the application.
+
+TBD
+
+{{% notice title="Prerequisites" style="primary" icon="info" %}}
+
+* Outbound SSH access to port `2222`.
+* Outbound HTTP access to port `8083`.
+* Familiarity with the `bash` shell and `vi/vim` editor.
+
+{{% /notice %}}

content/en/other/12-appdynamics/1-petclinic/images/apm-environment.png

@@ -0,0 +1,7 @@
+---
+title: AppDynamics Workshops
+description: AppDynamics Workshops
+weight: 12
+---
+
+{{% children containerstyle="ul" depth="1" description="true" %}}