flow and working on logs

Author: hagen-p

content/en/other/1-imt/_index.md

@@ -15,7 +15,7 @@ The workshops also introduce you to dashboards, editing and creating charts, cre
 
 By the end of these technical workshops, you will have a good understanding of some of the key features and capabilities of the Splunk Observability Cloud.
 
-Here are the instructions on how to access your pre-configured [AWS](./initial-setup/)/EC2 instance](./initial-setup/)
+Here are the instructions on how to access your pre-configured [AWS](./initial-setup/)/[EC2 instance](./initial-setup/)
 
 ![Splunk Architecture](images/architecture.png)
 

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/10-preparation.md

@@ -45,14 +45,14 @@ configmap/scriptfile created
 {{% /tab %}}
 {{< /tabs >}}
 
-{{% notice title="In case of error Unable to read /etc/rancher/k3s/k3s.yaml" style="warning" %}}
+<!-- {{% notice title="In case of error Unable to read /etc/rancher/k3s/k3s.yaml" style="warning" %}}
 In rare occasions, you may encounter the above error at this point, this is due to incorrect file permission on the Kubernetes config file. This can easily be resolved by running the following command:
 
 ``` bash
 sudo chmod 777 /etc/rancher/k3s/k3s.yaml
 ```
 
-{{% /notice %}}
+{{% /notice %}} -->
 At this point we can verify the deployment by checking if the Pods are running:
 {{< tabs >}}
 {{% tab title="kubectl get pods" %}}

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/20-otel-collector.md

@@ -195,9 +195,9 @@ helm delete splunk-otel-collector
 ## 4. Verify the installation by checking Metrics and Logs
 
 Once the installation is completed, you can login into the  **Splunk Observability Cloud** with the URL provided by the Instructor.
-First, Navigate to **Kubernetes Navigator** in the **Infrastructure** section to see the metrics from your cluster inthe **K8s nodes** pane an change the *Time* filter to the last 15 Minutes (-15m).
+First, Navigate to **Kubernetes Navigator** in the **Infrastructure**![infra](../images/infra-icon.png?classes=inline&height=25px) section to see the metrics from your cluster in the **K8s nodes** pane. Change the *Time* filter to the last 15 Minutes (-15m) to focus on the lates data.
 
-Use the regular filter option and select `k8s.cluster.name`**(1)**and type or select the cluster name of your workshop instance (you can get the unique part from your cluster name by using the `INSTANCE` from the output from the shell script you run earlier). (You can also select you cluster by clicking on on its image in the cluster pane.)
+Use the regular filter option at the top of the Navigator and select `k8s.cluster.name` **(1)** and type or select the cluster name of your workshop instance (you can get the unique part from your cluster name by using the `INSTANCE` from the output from the shell script you run earlier). (You can also select you cluster by clicking on on its image in the cluster pane.)
 
 You should see metrics **(3)** & log events **(4)** related to your cluster. Also a `Mysql` pane **(5)** should appear,when you click on that pane, you can see the MySQL related metrics from you database.
 

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/30-auto-instrumentation.md

@@ -4,7 +4,12 @@ linkTitle: 30. Auto-instrumentation & APM
 weight: 30
 ---
 
-In the previous chapter, we enabled the OpenTelemetry Collector for Kubernetes and configured it to send metrics to the **Splunk Observability Cloud**.  The next step is to enable auto-instrumentation for our Java apps running in the pods in our cluster.
+In the previous chapter, we enabled the OpenTelemetry Collector for Kubernetes on our cluster and configured it to send metrics both Kubernetes and Mysql metrics to the **Splunk Observability Cloud**. The next step is to enable auto-instrumentation for our Java apps running in the pods in our cluster.
+
+{{% notice title="Zero Config Auto Instrumentation" style="info" %}}
+It is important to understand that Zero Config Auto instrumentation is designed to get Trace/Span & Profiling data out of your existing application, without requiring  you to change your code or requiring to rebuild.
+{{% /notice %}}
+
 
 For this workshop we will be using the Zero config option of the Opentelemetry Collector in Kubernetes.
 This means that the Collector monitors you pods running in Kubernetes, and if they match certain criteria, they will en able auto instrumentation on the pod.
@@ -62,7 +67,7 @@ Image:         quay.io/phagen/spring-petclinic-api-gateway:0.0.2
 
 Once the container is patched it will be restarted, let's go back to  the **Splunk Observability Cloud** with the URL provided by the Instructor to check our service.
 
-First, Navigate to the **APM** section to see the traces from your service in the **Explore** Pane. Use the filter option and change the *environment* filter **(1)** and search for the name of your workshop instance in the drop down box, it should be the [INSTANCE]-workshop. (where `INSTANCE` is the value from the shell script you run earlier). Make sure it is the only one selected.
+First, Navigate to the **APM** ![APM](../images/apm-icon.png?classes=inline&height=25px) section to see the traces from your service in the **Explore** Pane. Use the filter option and change the *environment* filter **(1)** and search for the name of your workshop instance in the drop down box, it should be the [INSTANCE]-workshop. (where `INSTANCE` is the value from the shell script you run earlier). Make sure it is the only one selected.
 ![apm](../images/apm-api-gateway-overview.png)
 You should see the name **(2)** of the service and metrics in the Latency and Request & Errors charts. (You can ignore the Critical Alert, Its caused by the sudden request increase generated by the load generator, It will become the norm in a bit and go away).
 

content/en/other/3-auto-instrumentation/3-java-microservices-pet-clinic/60-log-observer-connect.md

@@ -6,19 +6,35 @@ weight: 60
 
 ## 1. Introduction
 
-For the Splunk Log Observer component, we will configure the Spring PetClinic application to use an Otel Based format to write logs, This will allow the (Auto)-instrumentation to add Otel relevant information to the logs that can be used to correlate Metric, traces and logs. 
+Until this point, we have not touched or changed our code, yet we did receive Trace & Profiling/DB Query performance information.
+If we want to get more out of our Java application, we can introduce a small change to our application log setup.
+
+This change will configure the Spring PetClinic application to use an Otel Based format to write logs, This will allow the (Auto)-instrumentation to add Otel relevant information into the logs.
+
+The Splunk Log Observer component is used to view the logs and with this information can automatically relate logs information with APM Services and Traces. This feature called **Related Content** will also work with Infra structure.
 
 ## 2. Update Logback config for the services
 
-The Spring PetClinic application can be configured to use several different Java logging libraries. In this scenario, the application is using `logback`.  to make sure we get the otel information in the logs we just need to update a file named `logback.xml` for each of the services in the petclinc microservices folders.
+The Spring PetClinic application can be configured to use several different Java logging libraries. In this scenario, the application is using `logback`.  To make sure we get the otel information in the logs we need to update a file named `logback.xml` with the log structure, and add an Otel dependency to the `pom.xml` of each of the services in the petclinic microservices folders.
+
+First lets set the Log Structure/Format:
+
+Spring boot will allow you to set a global template, but for ease of use, we will replace the existing content of the `logback-spring.xml` files of each service with the following XML content using a prepared script:
+Note the following entries that will be added:  
 
-Spring boot will allow you to set a global template, but for ease of use,  we will replace the existing content of the `logback-spring.xml` files of each service with the following XML content using a prepared script:
+- trace_id
+- span_id
+- trace_flags
+- service.name
+- deployment.environment
+
+These fields allows the **Splunk Observability Cloud Suite** to display **Related Content**:
 
 ```xml
 <?xml version="1.0" encoding="UTF-8"?>
 <configuration>
     <include resource="org/springframework/boot/logging/logback/base.xml"/>
-    <!-- Required for Loglevel managment into the Spring Petclinic Admin Server-->
+    <!-- Required for Loglevel management into the Spring Petclinic Admin Server-->
     <jmxConfigurator/>
     <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
         <encoder>
@@ -40,6 +56,8 @@ Spring boot will allow you to set a global template, but for ease of use,  we wi
 </configuration>
 ```
 
+So lets run the script that will update our log structure with the format above:
+
 {{< tabs >}}
 {{% tab title="Update Logback files" %}}
 
@@ -70,9 +88,15 @@ We can verify if the replacement has been successful by examining the spring-log
 cat /home/ubuntu/spring-petclinic-microservices/spring-petclinic-customers-service/src/main/resources/logback-spring.xml
 ```
 
-## 3. Recompile and store the services locally
+## 3. Reconfigure and build the services locally
+
+Before we can build the new services with the updated log format we need to add the dependency to the `Pom.xml`:
 
-Next, run the script that will use the `maven` command to compile/build/package the PetClinic microservices:
+```bash
+. ~/workshop/petclinic/scripts/add_otel.sh
+```
+
+The Services are now ready to be build, so run the script that will use the `maven` command to compile/build/package the PetClinic microservices (Note the -P buildDocker, this will build the new containers):
 {{< tabs >}}
 {{% tab title="Running maven" %}}
 
@@ -109,7 +133,7 @@ Successfully tagged quay.io/phagen/spring-petclinic-api-gateway:latest
 {{% /tab %}}
 {{< /tabs >}}
 
-Next, run the script that will push the newly build containers into our local repository:
+Given that Kubernetes needs to pull these freshly build images from somewhere, we are going to store them in the repository we set up earlier. To do this, run the script that will push the newly build containers into our local repository:
 
 {{< tabs >}}
 {{% tab title="pushing Containers" %}}
@@ -171,12 +195,31 @@ The result should be :
 
 ## 5. Deploy new services to kubernetes
 
-To see the changes in effect, we need to redeploy the services, by applying the local version of the deployment yaml.
+To see the changes in effect, we need to redeploy the services,  First let change the location of the images from the exterenal repo  to the local one by running the following script:
+
+```bash
+. ~/workshop/petclinic/scripts/set_local.sh
+```
+
+The result is a new file on disk called **petclinic-local.yaml**
+Let switch to the local version by applying the local version of the deployment yaml.
 
 ```bash
 kubectl apply -f ~/workshop/petclinic/petclinic-local.yaml
 ```
-this will cause the containers to be replaced with the local version, you can verify this by chcking the continers
+
+This will cause the containers to be replaced with the local version, you can verify this by checking the containers:
+
+```bash
+kubectl describe pods api-gateway |grep Image:
+```
+
+The resulting output should say:
+
+```text
+  Image:         ghcr.io/signalfx/splunk-otel-java/splunk-otel-java:v1.30.0
+  Image:         localhost:5000/spring-petclinic-api-gateway:local
+```
 
 ## 6. View Logs
 
@@ -192,11 +235,8 @@ This is the end of the workshop and we have certainly covered a lot of ground. A
 
 **Congratulations!**
 
-
-
 docker system prune -a --volumes
 
-
   81  . ~/workshop/petclinic/scripts/add_otel.sh
    82  . ~/workshop/petclinic/scripts/update_logback.sh
    83  ./mvnw clean install -DskipTests -P buildDocker
@@ -210,4 +250,4 @@ docker system prune -a --volumes
    91  kubectl delete -f ~/workshop/petclinic/petclinic-local.yaml
    92  kubectl apply -f ~/workshop/petclinic/petclinic-local.yaml
    93  k9s
-   94  kubectl get deployments -l app.kubernetes.io/part-of=spring-petclinic -o name | xargs -I % kubectl patch % -p "{\"spec\": {\"template\":{\"metadata\":{\"annotations\":{\"instrumentation.opentelemetry.io/inject-java\":\"true\"}}}}}"
+   94  kubectl get deployments -l app.kubernetes.io/part-of=spring-petclinic -o name | xargs -I % kubectl patch % -p "{\"spec\": {\"template\":{\"metadata\":{\"annotations\":{\"instrumentation.opentelemetry.io/inject-java\":\"true\"}}}}}"