fixes for flow and scripts

Author: hagen-p

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/10-preparation.md

@@ -4,9 +4,32 @@ linkTitle: 10. Preparation
 weight: 10
 ---
 
-## 1. Deploying the prebuilt containers into Kubernetes
+## 1. Validate the settings for your workshop
 
-The first thing we need to set up is ... well, an application. The first deployment of our application will be using prebuilt containers to give us the base scenario: a Java microservices-based application running in Kubernetes.
+To ensure your instance is configured correctly, we need to confirm that the required environment variables for this workshop are set correctly. In your terminal run the following command:
+
+``` bash
+. ~/workshop/petclinic/scripts/check_env.sh
+```
+
+In the output check the following environment variables are present and have values set:
+
+```text
+ACCESS_TOKEN
+REALM 
+RUM_TOKEN 
+HEC_TOKEN
+HEC_URL
+INSTANCE
+```
+
+Please make a note of the `INSTANCE` environment variable value as this is the reference to you workshop instance and we will need it to filter the data in the **Splunk Observability Suite** UI.
+
+For this workshop, **all** of the above are required. If any are missing, please contact your instructor.
+
+## 2. Deploying the prebuilt containers into Kubernetes
+
+The second thing we need to do, well ..., is to set up our application. The first deployment of our application will be using prebuilt containers to give us the base scenario: a regular Java microservices-based application running in Kubernetes.
 
 So let's deploy our application:
 {{< tabs >}}
@@ -46,11 +69,7 @@ configmap/scriptfile created
 {{< /tabs >}}
 
 <!-- {{% notice title="In case of error Unable to read /etc/rancher/k3s/k3s.yaml" style="warning" %}}
-In rare occasions, you may encounter the above error at this point, this is due to incorrect file permission on the Kubernetes config file. This can easily be resolved by running the following command:
-
-``` bash
-sudo chmod 777 /etc/rancher/k3s/k3s.yaml
-```
+In rare occasions, you may encounter the above error at this point.  please lpg out and back in, and verify the above env variables are all set correctly. If not please, please contact your instructor.
 
 {{% /notice %}} -->
 At this point we can verify the deployment by checking if the Pods are running:
@@ -101,7 +120,7 @@ Change into the `spring-petclinic` directory:
 cd ~/spring-petclinic-microservices
 ```
 
-Next, run the script that will use the `maven` command to compile/build the PetClinic microservices:
+Next, lets test the download and run the script that will use the `maven` command to compile/build the PetClinic microservices:
 {{< tabs >}}
 {{% tab title="Running maven" %}}
 
@@ -141,7 +160,7 @@ This will take a few minutes the first time you run, `maven` will download a lot
 
 ## 3. Set up a local Docker Repository
 
-Once we have our Auto instrumentation up and running, we are going to use show some  of the additional instrumentation features of Opentelemetry Java. This will be the first time we will touch the source code and add some annotations to it to get even more valuable data from our Java application. Kubernetes will need to pull these new images from somewhere, so let's setup a local repository, so Kubernetes can pull these local images.
+Once we have our Auto instrumentation up and running with the existing containers, we are going to use show some  of the additional instrumentation features of Opentelemetry Java. That will be the first time we will touch the source code and add some annotations to it to get even more valuable data from our Java application. Kubernetes will need to pull these new images from somewhere, so let's setup a local repository, so Kubernetes can pull those local images.
 
 {{< tabs >}}
 {{% tab title="Install Docker Repository" %}}

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/20-otel-collector.md

@@ -13,41 +13,20 @@ The Splunk OpenTelemetry Collector is the core component of instrumenting infras
 * Profiling data
 * Host and Application logs
 
+To get Observability signals into the **Splunk Observability Cloud** we need to add an OpenTelemetry Collector to our Kubernetes cluster.
+
 {{% notice title="Delete any existing OpenTelemetry Collectors" style="warning" %}}
-If you have completed a Splunk Observability workshop using this EC2 instance, please ensure you have deleted the collector running in Kubernetes before continuing. This can be done by running the following command:
+If you have completed a Splunk Observability workshop using this EC2 instance, please ensure you have deleted the collector running in Kubernetes before continuing with this workshop. This can be done by running the following command:
 
 ``` bash
 helm delete splunk-otel-collector
 ```
 
 {{% /notice %}}
 
-## 2. Confirm environment variables
-
-To ensure your instance is configured correctly, we need to confirm that the required environment variables for this workshop are set correctly. In your terminal run the following command:
-
-``` bash
-. ~/workshop/petclinic/scripts/check_env.sh
-```
-
-In the output check the following environment variables are present and have values set:
-
-```text
-ACCESS_TOKEN
-REALM 
-RUM_TOKEN 
-HEC_TOKEN
-HEC_URL
-INSTANCE
-```
-
-Please make a note of the `INSTANCE` environment variable value as this is the reference to you workshop instance and we will need it to filter the data in the **Splunk Observability Suite** UI.
-
-For this workshop, **all** of the above are required. If any are missing, please contact your instructor.
-
-## 3. Install the OpenTelemetry Collector using Helm
+## 2. Install the OpenTelemetry Collector using Helm
 
-We are going to install the OpenTelemetry Collector in Operator mode using the Splunk Kubernetes Helm Chart for the Opentelemetry collector. First, we need to add the Splunk Helm chart repository to Helm and update so it knows where to find it:
+We are going to install the Splunk distribution of the OpenTelemetry Collector in Operator mode using the Splunk Kubernetes Helm Chart for the Opentelemetry collector. First, we need to add the Splunk Helm chart repository to Helm and update so it knows where to find it:
 
 {{< tabs >}}
 {{% tab title="Helm Repo Add" %}}
@@ -73,7 +52,7 @@ Update Complete. ⎈Happy Helming!⎈
 {{% /tab %}}
 {{< /tabs >}}
 
-Splunk Observability Cloud offers wizards in the **Splunk Observability Suite** UI to walk you through the setup of the Collector on both your infrastructure including Kubernetes, but in interest of time, we will use a setup created earlier and are going to install the OpenTelemetry Collector with the OpenTelemetry Collector Helm chart with some additional options:
+Splunk Observability Cloud offers wizards in the **Splunk Observability Suite** UI to walk you through the setup of the Collector on your infrastructure including Kubernetes, but in interest of time, we will use a setup created earlier. As we want the auto instrumentation to be available, we will install the OpenTelemetry Collector with the OpenTelemetry Collector Helm chart with some additional options:
 
 * --set="operator.enabled=true" - this will install the Opentelemetry operator, that will be used to handle auto instrumentation
 * --set="certmanager.enabled=true" - This will install the required certificate manager for the operator.
@@ -192,7 +171,7 @@ helm delete splunk-otel-collector
 
 {{% /notice %}}
 
-## 4. Verify the installation by checking Metrics and Logs
+## 3. Verify the installation by checking Metrics and Logs
 
 Once the installation is completed, you can login into the  **Splunk Observability Cloud** with the URL provided by the Instructor.
 First, Navigate to **Kubernetes Navigator** in the **Infrastructure**![infra](../images/infra-icon.png?classes=inline&height=25px) section to see the metrics from your cluster in the **K8s nodes** pane. Change the *Time* filter to the last 15 Minutes (-15m) to focus on the lates data.

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/30-auto-instrumentation.md

@@ -31,9 +31,9 @@ The resulting output should say:
 Image:         quay.io/phagen/spring-petclinic-api-gateway:0.0.2
 ```
 
-This container is pulled from a remote repository `quay.io` and as you will see 
+This container is pulled from a remote repository `quay.io` and was not build to send traces to the **Splunk Observability Cloud**  
 
-Lets add the Java auto instrumentation TAG to the api-gateway service first with the `kubectl patch deployment` command.
+Lets enable the Java auto instrumentation on the api-gateway service first by adding the `inject-java` tag to kubernetes with the `kubectl patch deployment` command.
 {{< tabs >}}
 {{% tab title="Patch api-gateway service" %}}
 
@@ -108,7 +108,33 @@ deployment.apps/api-gateway patched (no change)
 {{% /tab %}}
 {{< /tabs >}}
 
-It will take the Petclinic Microservice application a few minutes to start up and fully synchronise, but after its fully initialized, you now should see all the different services in Splunk APM:
+It will take the Petclinic Microservice application a few minutes to start up and fully synchronise.
+Lets monitor the  load generator container until its capable to generate load as show in the output tab.
+
+{{< tabs >}}
+{{% tab title="Tail Log" %}}
+
+``` bash
+. ~/workshop/petclinic/scripts/tail_logs.sh
+```
+
+{{% /tab %}}
+{{% tab title="Tail Log Output" %}}
+
+```text
+{"severity":"info","msg":"Welcome Text = "Welcome to Petclinic"}
+{"severity":"info","msg":"@ALL"
+{"severity":"info","msg":"@owner details page"}
+{"severity":"info","msg":"@pet details page"}
+{"severity":"info","msg":"@add pet page"}
+{"severity":"info","msg":"@veterinarians page"}
+{"severity":"info","msg":"cookies was"}
+```
+
+{{% /tab %}}
+{{< /tabs >}}
+
+Once the services are fully initialized, you now should see all the different services appear in Splunk APM:
 ![all services](../images/apm-full-service.png)
 Of course, we want to check the Dependency map by clicking Explore:
 ![full map](../images/apm-map-full.png)

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/60-log-observer-connect.md

@@ -203,7 +203,12 @@ To see the changes in effect, we need to redeploy the services,  First let chang
 
 The result is a new file on disk called **petclinic-local.yaml**
 Let switch to the local version by applying the local version of the deployment yaml.
+ First delete the old deplyment with 
 
+```bash
+kubectl delete -f ~/workshop/petclinic/petclinic-local.yaml
+```
+followed by 
 ```bash
 kubectl apply -f ~/workshop/petclinic/petclinic-local.yaml
 ```
@@ -214,7 +219,7 @@ This will cause the containers to be replaced with the local version, you can ve
 kubectl describe pods api-gateway |grep Image:
 ```
 
-The resulting output should say:
+The resulting output should say ( again if you see double, its the old container being terminated, give it a few seconds):
 
 ```text
   Image:         ghcr.io/signalfx/splunk-otel-java/splunk-otel-java:v1.30.0
@@ -223,6 +228,30 @@ The resulting output should say:
 
 ## 6. View Logs
 
+First give the service time to get back into sync and lets tail the load generator log again
+{{< tabs >}}
+{{% tab title="Tail Log" %}}
+
+``` bash
+. ~/workshop/petclinic/scripts/tail_logs.sh
+```
+
+{{% /tab %}}
+{{% tab title="Tail Log Output" %}}
+
+```text
+{"severity":"info","msg":"Welcome Text = "Welcome to Petclinic"}
+{"severity":"info","msg":"@ALL"
+{"severity":"info","msg":"@owner details page"}
+{"severity":"info","msg":"@pet details page"}
+{"severity":"info","msg":"@add pet page"}
+{"severity":"info","msg":"@veterinarians page"}
+{"severity":"info","msg":"cookies was"}
+```
+
+{{% /tab %}}
+{{< /tabs >}}
+
 From the left-hand menu click on **Log Observer** and ensure **Index** is set to **splunk4rookies-workshop**.
 
 Next, click **Add Filter** search for the field `service_name` select the value `<INSTANCE>-petclinic-service` and click `=` (include). You should now see only the log messages from your PetClinic application.

workshop/petclinic/scripts/update_logback.sh

@@ -1,36 +1,12 @@
 #!/bin/bash
 
-# '<?xml version="1.0" encoding="UTF-8"?>
-# <configuration>
-#     <include resource="org/springframework/boot/logging/logback/base.xml"/>
-#     <!-- Required for Loglevel managment into the Spring Petclinic Admin Server-->
-#     <jmxConfigurator/>
-#     <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
-#         <encoder>
-#           <pattern>
-#             %d{yyyy-MM-dd HH:mm:ss} - %logger{36} - %msg trace_id=%X{trace_id} span_id=%X{span_id} trace_flags=%X{trace_flags} %n service.name=%property{otel.resource.service.name}, deployment.environment=%property{otel.resource.deployment.environment}: %m%n
-#           </pattern>
-#         </encoder> 
-#     </appender>
-
-#     <!-- Just wrap your logging appender, for example ConsoleAppender, with OpenTelemetryAppender -->
-#     <appender name="OTEL" class="io.opentelemetry.instrumentation.logback.mdc.v1_0.OpenTelemetryAppender">
-#       <appender-ref ref="CONSOLE"/>
-#     </appender>
-
-#      <!-- Use the wrapped "OTEL" appender instead of the original "CONSOLE" one -->
-#      <root level="INFO">
-#        <appender-ref ref="OTEL"/>
-#      </root>
-# </configuration>'
-
-
 xml_content='<?xml version="1.0" encoding="UTF-8"?>
 <configuration>
     <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
         <encoder>
             <pattern>
-                logback: %d{HH:mm:ss.SSS} [%thread] %level %logger{36} - trace_id=%X{trace_id} span_id=%X{span_id} service.name=%property{otel.resource.service.name} trace_flags=%X{trace_flags} - %msg %kvp{DOUBLE}%n
+                logback: %d{HH:mm:ss.SSS} [%thread] severity=%-5level %logger{36} - trace_id=%X{trace_id} span_id=%X{span_id} service.name=%property{otel.resource.service.name} trace_flags=%X{trace_flags} - %msg %kvp{DOUBLE}%n
+            </pattern>
         </encoder>
     </appender>
     <appender name="OpenTelemetry"