Release Application Observability Code Challenge 2

Author: cbos

README.md

@@ -4,7 +4,7 @@
 ![Badge](https://goto.ceesbos.nl/badge/github/cbos/application-observability-code-challenges)
 
 Application Observability Code Challenges are a way to challenge yourself to improve the observability of a sample application. Of course, this application has some surprises that you probably did not expect.
-These challenges will help you develop an observability mindset and become more familiar with your observability tools.
+These puzzlers will help you develop an observability mindset and become more familiar with your observability tools.
 
 See the announcement blog about [Application Observability Code Challenges](https://goto.ceesbos.nl/aocc) for more information.
 
@@ -53,3 +53,4 @@ The sample application is prepared to send data using OpenTelemetry, so any setu
 # Challenges
 
 - [Challenge 1](challenge-01/README.md) - Spring boot with Jersey/JAX-RS REST endpoint with some strange behaviour 🤔
+- [Challenge 2](challenge-02/README.md) - Spot the difference between system thread and virtual thread usage for the same application with some problems

challenge-02/.env

@@ -0,0 +1,2 @@
+# Keep inline with Application Observability Code Challenges, otherwise dashboards might break
+OTEL_VERSION=2.15.0

challenge-02/.gitattributes

@@ -0,0 +1,2 @@
+/mvnw text eol=lf
+*.cmd text eol=crlf

challenge-02/.gitignore

@@ -0,0 +1,33 @@
+HELP.md
+target/
+!.mvn/wrapper/maven-wrapper.jar
+!**/src/main/**/target/
+!**/src/test/**/target/
+
+### STS ###
+.apt_generated
+.classpath
+.factorypath
+.project
+.settings
+.springBeans
+.sts4-cache
+
+### IntelliJ IDEA ###
+.idea
+*.iws
+*.iml
+*.ipr
+
+### NetBeans ###
+/nbproject/private/
+/nbbuild/
+/dist/
+/nbdist/
+/.nb-gradle/
+build/
+!**/src/main/**/build/
+!**/src/test/**/build/
+
+### VS Code ###
+.vscode/

challenge-02/.mvn/wrapper/maven-wrapper.properties

@@ -0,0 +1,3 @@
+wrapperVersion=3.3.4
+distributionType=only-script
+distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.11/apache-maven-3.9.11-bin.zip

challenge-02/README.md

@@ -0,0 +1,168 @@
+![Challenge 02](images/aocc_challenge02.png)
+
+# Application Observability Code Challenge 2
+![Badge](https://goto.ceesbos.nl/badge/github/cbos/application-observability-code-challenges/challenge-02)
+
+This is the second puzzler of the Application Observability Code Challenges.   
+See the [announcement blog](https://goto.ceesbos.nl/aocc) or the [repository README](../README.md) for more information about the challenges in general.
+Check out the first challenge as well, if you did not complete it yet.
+
+> 🚨 **Challenge**:
+> - Run the sample applications
+> - Run the tests to see what happens
+> - Try to find out what happens, make a hypothesis❗
+> - **Improve the observability** of the applications to **prove the hypothesis**
+> - Optional: fix the problem and **prove it with observability data that it is really fixed**
+> - Optional, but highly appreciated 🙏: Share your findings, insights you learned and potential solution, either as a ['discussion'](https://github.com/cbos/application-observability-code-challenges/discussions) or as a pull request
+
+An online guided environment is available with KillerCode, see [https://killercoda.com/observability-code-challenges](https://goto.ceesbos.nl/aocckk)
+
+## Challenge
+
+## Summary
+This challenge will learn you more about the observability of regular threads and virtual threads and how to make it more observable.
+
+## Setup in this repository
+
+- The setup is a set of applications with actual the same code base, but running with different configuration and Java versions
+- The applications are instrumented using OpenTelemetry auto instrumentation.
+- You can run the applications with Docker or directly.
+
+## Pre-requisites
+
+- Java 21 (if you want to use a lower version, you will need to modify the pom.xml)
+- Docker/Podman
+- Just CLI (optional, but recommended)
+- [K6](https://grafana.com/docs/k6/latest/set-up/install-k6/) for load testing
+
+## Prepare the environment
+
+### Clone the repository 
+Clone the repository to your local machine and go to the folder of `challenge-02`.
+
+```shell
+git clone https://github.com/cbos/application-observability-code-challenges
+cd application-observability-code-challenges/challenge-02
+```
+
+### Download the OpenTelemetry Java agent jar
+
+```shell
+just download-otel
+```
+This downloads the OpenTelemetry Java agent jar to the `.otel` directory.
+
+### Build the application
+
+```shell
+just build 
+# or if you want to do it manually
+./mvnw clean verify
+```
+
+### Observability Toolkit or your own stack
+Launch your observability stack or use the Observability Toolkit.
+The sample application assumes that you have an OpenTelemetry endpoint running at `localhost:4318`.
+
+If you don't have any observability tools running, you can run a preconfigured setup with the following commands:
+
+```shell
+git clone https://github.com/cbos/observability-toolkit
+cd observability-toolkit
+just up # or docker-compose up -d 
+```
+Now you can open http://localhost:3000 to open Grafana.
+
+## Run the application setup
+
+The setup of application contains 4 applications:
+- `common-backend`: A common backend used by three other applications
+- `regular-threads`: A version of the application with Java 21 and using regular threads to execute the requests
+- `virtual-threads-21`: A version of the application with Java 21 and using `Virtual Threads` to execute the requests
+- `virtual-threads-25`: A version of the application with Java 25 and using `Virtual Threads` to execute the requests
+
+![Setup](images/application_setup.png)
+
+To run the setup, you can use the following command:
+
+```shell
+# or if you want to run it with Docker
+just up  
+just down # to stop the docker container
+just ps # to see the status of the container
+just logs <application_name> # to see the logs of the specific applications
+
+# Without Just you can use:
+# docker-compose -f docker/docker-compose.yml up -d --build
+# docker-compose -f docker/docker-compose.yml down 
+# docker-compose -f docker/docker-compose.yml ps
+# docker-compose -f docker/docker-compose.yml logs <application_name>
+```
+The applications are using auto-instrumentation of OpenTelemetry, which provides a foundation for observability, but is it enough for this challenge?
+
+# Test runs with k6
+
+To get more information about the problems, a set of test scripts is available.
+These testscripts are implemented with [K6](https://grafana.com/oss/k6/), a load testing tool that can also be used for performance testing.
+K6 also produces metrics as well in OpenTelemetry format.
+
+## Run load scripts
+
+The [first challenge](../challenge-01/README.md) contains a detailed description on how to run the tests and how the output can be viewed and interpreted.    
+Please read that first, if you are not familiar with K6.
+
+To run the load tests, you can use the following command:  
+```shell
+# Just a putting load on the '/' endpoint of the 3 services
+just k6-scenario-1
+# Or run k6 with the following command, but this does not have metrics pushed:
+# k6 run k6/scenario1.js 
+
+# Put load on the /random/<id> endpoint of the 3 services
+just k6-scenario-2
+````
+These tests give load and will give you more insights in the Grafana dashboards.
+
+### K6 Load test - Grafana Dashboard
+
+See [http://localhost:3000/d/o11ytk-k6-load-test/k6-load-test](http://localhost:3000/d/o11ytk-k6-load-test/k6-load-test) for the K6 dashboard.
+The first challenge also explains how to use this dashboard.
+
+### Service details - Grafana Dashboard
+
+See [http://localhost:3000/d/o11ytk-service-details/service-details](http://localhost:3000/d/o11ytk-service-details/service-details) for the Service details dashboard.   
+The dashboard has details about incoming requests with the RED metrics. Rate, errors and duration.   
+It also shows a heat map of request duration for incoming requests.
+
+There are more details per endpoint of the service, but that is not relevant for this test.   
+And there are details for the outgoing requests, but that is not relevant for this test either.
+
+There is also a section with JVM metrics such as memory usage (heap and non-heap), garbage collection and threads.
+This gives you more information about the behaviour of the service.   
+And there is information about Tomcat, the number of threads specific to Tomcat request handling.
+
+And as you might have noticed already, this challenge is about regular threads and virtual threads, these sections are relevant for this challenge.
+
+## Run the final scenario
+
+The last scenario is the actual scenario you need to run, this again puts load on the 3 applications and with that you will see response times of these services will go up, while the response times of the common backend will be fine and stable.   
+
+```shell
+just k6-scenario-3
+```
+
+In the setup you can see that the 3 applications are using the common-backend.
+That is what you can see from the node graph as well in Grafana:
+
+Open Grafana > Explore > Select 'Tempo' as datasource > Select 'Service Graph' tab:
+
+![Node graph](images/node_graph.png)
+
+
+Goals for this challenge:
+1) Check if you can find the Java version in the observability data (logs, metrics, traces)
+1) Can you spot the latency gap between the services in traces?
+1) Can you find the problem based on metrics?
+1) And the most important: Can you see the difference between the 3 application `regular-threads` and `virtual-threads-21` and `virtual-threads-25`?
+2) What can be done to make it more observable?
+

challenge-02/docker/Dockerfile

@@ -0,0 +1,23 @@
+ARG JAVA_VERSION="21.0.7_6-jre-alpine"
+
+FROM docker.io/eclipse-temurin:${JAVA_VERSION}
+
+# Expose the port to the external world
+EXPOSE 8080
+
+RUN mkdir /app
+
+# create a nonroot user and group
+RUN addgroup -S javauser && adduser -S -s /bin/false -G javauser javauser
+
+# Copy the application
+COPY target/*.jar /app/otel-demo-application.jar
+WORKDIR /app
+
+# Change ownership to non-root user
+RUN chown -R javauser:javauser /app
+
+## set the nonroot user as the default user
+USER javauser
+
+CMD "java" "-jar" "otel-demo-application.jar"

challenge-02/docker/docker-compose.yml

@@ -0,0 +1,68 @@
+x-otel-base: &otel-base
+  env_file:
+    - opentelemetry.properties
+  extra_hosts:
+    # host.containers.internal is standard for podman, with this mapping it is also available for docker
+    - "host.containers.internal:host-gateway"
+  volumes:
+    # file is shared with multiple containers, adding 'z' attribute
+    - ../.otel/opentelemetry-javaagent.jar:/app/opentelemetry-javaagent.jar:ro,z
+  mem_limit: 1gb
+
+name: OpenTelemetry-demo-app
+
+services:
+  common-backend:
+    <<: *otel-base
+    image: ceesbos.nl/common-backend
+    build:
+      context: ..
+      dockerfile: docker/Dockerfile
+    environment:
+      - OTEL_SERVICE_NAME=common-backend
+      - SPRING_APPLICATION_NAME=common-backend
+
+  regular-threads:
+    <<: *otel-base
+    image: ceesbos.nl/regular-threads
+    build:
+      context: ..
+      dockerfile: docker/Dockerfile
+    environment:
+      - OTEL_SERVICE_NAME=regular-threads
+      - SPRING_APPLICATION_NAME=regular-threads
+      - DOWNSTREAM_HOSTNAME=common-backend
+    ports:
+      - "8081:8080"
+
+  virtual-threads-21:
+    <<: *otel-base
+    image: ceesbos.nl/virtual-threads-21
+    build:
+      context: ..
+      dockerfile: docker/Dockerfile
+    environment:
+      - OTEL_SERVICE_NAME=virtual-threads-21
+      - SPRING_APPLICATION_NAME=virtual-threads-21
+      - DOWNSTREAM_HOSTNAME=common-backend
+      # Set spring.threads.virtual.enabled=true via a environment variable
+      - SPRING_THREADS_VIRTUAL_ENABLED=true
+    ports:
+      - "8082:8080"
+
+  virtual-threads-25:
+    <<: *otel-base
+    image: ceesbos.nl/virtual-threads-25
+    build:
+      context: ..
+      dockerfile: docker/Dockerfile
+      args:
+        - JAVA_VERSION=25_36-jre-alpine
+    environment:
+      - OTEL_SERVICE_NAME=virtual-threads-25
+      - SPRING_APPLICATION_NAME=virtual-threads-25
+      - DOWNSTREAM_HOSTNAME=common-backend
+      # Set spring.threads.virtual.enabled=true via a environment variable
+      - SPRING_THREADS_VIRTUAL_ENABLED=true
+    ports:
+      - "8083:8080"

challenge-02/docker/opentelemetry.properties

@@ -0,0 +1,19 @@
+JAVA_TOOL_OPTIONS=-javaagent:/app/opentelemetry-javaagent.jar
+OTEL_METRICS_EXPORTER=otlp
+OTEL_TRACES_EXPORTER=otlp
+OTEL_LOGS_EXPORTER=otlp
+OTEL_TRACES_SAMPLER=always_on
+OTEL_EXPORTER_OTLP_ENDPOINT=http://host.containers.internal:4318
+OTEL_EXPORTER_OTLP_METRICS_TIMEOUT=3000
+OTEL_METRIC_EXPORT_INTERVAL=5000
+
+# Tomcat JMX must be enabled in application.properties with server.tomcat.mbeanregistry.enabled=true
+OTEL_JMX_ENABLED=true
+OTEL_JMX_TARGET_SYSTEM=tomcat
+OTEL_JMX_DISCOVERY_DELAY=1m
+
+
+# https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/runtime-telemetry/README.md
+# These settings enable more details
+# OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_JAVA17_ENABLED=true
+# OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_JAVA17_ENABLE_ALL=true