Skip to content

Commit e561b7c

Browse files
zeitlingercyrille-leclerctrask
authored
add example for OTLP logging via stdout and k8s (#547)
Co-authored-by: Cyrille Le Clerc <[email protected]> Co-authored-by: Trask Stalnaker <[email protected]>
1 parent 8bc81c6 commit e561b7c

File tree

18 files changed

+495
-0
lines changed

18 files changed

+495
-0
lines changed

‎.github/renovate.json5‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,5 +35,18 @@
3535
"matchPackageNames": ["eclipse-temurin"],
3636
"enabled": false
3737
}
38+
],
39+
"customManagers": [
40+
{
41+
"customType": "regex",
42+
"description": "Update _VERSION variables in Dockerfiles",
43+
"fileMatch": [
44+
"(^|/|\\.)Dockerfile$",
45+
"(^|/)Dockerfile\\.[^/]*$"
46+
],
47+
"matchStrings": [
48+
"# renovate: datasource=(?<datasource>[a-z-]+?)(?: depName=(?<depName>.+?))? packageName=(?<packageName>.+?)(?: versioning=(?<versioning>[a-z-]+?))?\\s(?:ENV|ARG) .+?_VERSION=(?<currentValue>.+?)\\s"
49+
]
50+
}
3851
]
3952
}

‎.github/scripts/run-oats-tests.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
pushd logging-k8s-stdout-otlp-json
6+
../gradlew assemble
7+
popd
8+
9+
wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
10+
11+
cd oats/yaml
12+
go install github.com/onsi/ginkgo/v2/ginkgo@latest
13+
export TESTCASE_TIMEOUT=5m
14+
export TESTCASE_BASE_PATH=../..
15+
ginkgo -r

‎.github/workflows/oats-tests.yml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: OATS Tests
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
paths:
8+
- .github/workflows/oats-tests.yml
9+
- 'logging-k8s-stdout-otlp-json/**'
10+
workflow_dispatch:
11+
12+
jobs:
13+
acceptance-tests:
14+
runs-on: ubuntu-24.04
15+
steps:
16+
- name: Check out
17+
uses: actions/checkout@v4
18+
19+
- name: Set up JDK for running Gradle
20+
uses: actions/setup-java@v4
21+
with:
22+
distribution: temurin
23+
java-version: 17
24+
25+
- name: Set up gradle
26+
uses: gradle/actions/setup-gradle@v4
27+
with:
28+
cache-read-only: ${{ github.event_name == 'pull_request' }}
29+
30+
- name: Check out oats
31+
uses: actions/checkout@v4
32+
with:
33+
repository: grafana/oats
34+
ref: v0.1.0
35+
path: oats
36+
37+
- name: Set up Go
38+
uses: actions/setup-go@v5
39+
with:
40+
go-version: '1.23'
41+
cache-dependency-path: oats/go.sum
42+
43+
- name: Run acceptance tests
44+
run: .github/scripts/run-oats-tests.sh
45+
46+
- name: upload log file
47+
uses: actions/upload-artifact@v4
48+
if: failure()
49+
with:
50+
name: OATS logs
51+
path: oats/yaml/build/**/*.log

‎logging-k8s-stdout-otlp-json/Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM eclipse-temurin:21-jre
2+
3+
WORKDIR /usr/src/app/
4+
5+
# renovate: datasource=github-releases depName=opentelemetry-java-instrumentation packageName=open-telemetry/opentelemetry-java-instrumentation
6+
ENV OPENTELEMETRY_JAVA_INSTRUMENTATION_VERSION=v2.10.0
7+
8+
ADD build/libs/*SNAPSHOT.jar ./app.jar
9+
ADD --chmod=644 https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/$OPENTELEMETRY_JAVA_INSTRUMENTATION_VERSION/opentelemetry-javaagent.jar ./opentelemetry-javaagent.jar
10+
ENV JAVA_TOOL_OPTIONS=-javaagent:./opentelemetry-javaagent.jar
11+
12+
EXPOSE 8080
13+
ENTRYPOINT [ "java", "-jar", "./app.jar" ]

‎logging-k8s-stdout-otlp-json/README.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Exporting Application logs using JSON logging in Kubernetes
2+
3+
If you want to get logs from your Java application ingested into an
4+
OpenTelemetry-compatible logs backend, the easiest and recommended way is using
5+
an OpenTelemetry protocol (OTLP) exporter,
6+
which is explained in the [logging](../logging) example.
7+
8+
However, some scenarios require logs
9+
to be output to files or stdout due to organizational or reliability needs.
10+
Refer to [Collecting OpenTelemetry-compliant Java logs from files](https://opentelemetry.io/blog/2024/collecting-otel-compliant-java-logs-from-files/) for more details.
11+
12+
This example contains
13+
14+
- a Java application that uses the experimental
15+
[experimental-otlp/stdout](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md#in-development-exporter-selection) logs exporter
16+
- an OpenTelemetry collector configuration that uses the
17+
[OTLP/JSON connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/otlpjsonconnector) to turn Pod logs into `OTLP`
18+
19+
## Architecture
20+
21+
![OTLP/JSON Architecture](otlpjson-architecture.png)
22+
23+
The OpenTelemetry Collector pipeline:
24+
25+
![OpenTelemetry Collector Pipeline](otel-collector-otlpjson-pipeline.png)
26+
27+
## Getting Started
28+
29+
The k8s directory contains the Kubernetes manifests to deploy the application and the collector.
30+
31+
Ignore the `lgtm.yaml` file, which is only used for running locally with
32+
[LGTM](https://github.com/grafana/docker-otel-lgtm/)
33+
and automated testing using [OATs](https://github.com/grafana/oats).
34+
35+
## Running locally
36+
37+
You can run the application locally using the following steps:
38+
39+
1. Run [k3d.sh](./k3d.sh) to start a local Kubernetes cluster with all the necessary components.
40+
2. Generate traffic using [generate-traffic.sh](./generate-traffic.sh)
41+
3. Log in to [http://localhost:3000](http://localhost:3000)
42+
4. Go to "Explore"
43+
5. Select "Loki" as data source to view the logs

‎logging-k8s-stdout-otlp-json/build.gradle.kts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
import org.springframework.boot.gradle.plugin.SpringBootPlugin
2+
import org.springframework.boot.gradle.tasks.bundling.BootJar
3+
4+
plugins {
5+
id("java")
6+
id("org.springframework.boot") version "3.4.0"
7+
}
8+
9+
description = "OpenTelemetry Example for Java Agent with Stdout logging"
10+
val moduleName by extra { "io.opentelemetry.examples.javagent.logging-k8s-stdout-otlp-json" }
11+
12+
java {
13+
toolchain {
14+
languageVersion.set(JavaLanguageVersion.of(17))
15+
}
16+
}
17+
18+
dependencies {
19+
implementation(platform(SpringBootPlugin.BOM_COORDINATES))
20+
21+
implementation("org.springframework.boot:spring-boot-starter-web")
22+
implementation("org.springframework.boot:spring-boot-starter-actuator")
23+
}
24+

‎logging-k8s-stdout-otlp-json/generate-traffic.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/bash
2+
3+
watch 'curl -s http://localhost:8080/rolldice'

‎logging-k8s-stdout-otlp-json/k3d.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
docker build -f Dockerfile -t "dice:1.1-SNAPSHOT" .
6+
k3d cluster create jsonlogging || k3d cluster start jsonlogging
7+
k3d image import -c jsonlogging dice:1.1-SNAPSHOT
8+
9+
kubectl apply -f k8s/
10+
11+
kubectl wait --for=condition=ready pod -l app=dice
12+
kubectl wait --for=condition=ready --timeout=5m pod -l app=lgtm
13+
14+
kubectl port-forward service/dice 8080:8080 &
15+
kubectl port-forward service/lgtm 3000:3000 &

‎logging-k8s-stdout-otlp-json/k8s/collector-configmap.yaml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: otel-collector-config
5+
data:
6+
otel-collector-config.yaml: |-
7+
receivers:
8+
otlp:
9+
protocols:
10+
grpc:
11+
endpoint: 0.0.0.0:4317
12+
http:
13+
endpoint: 0.0.0.0:4318
14+
prometheus/collector: # needed if you use the docker-lgtm image
15+
config:
16+
scrape_configs:
17+
- job_name: 'opentelemetry-collector'
18+
static_configs:
19+
- targets: [ 'localhost:8888' ]
20+
filelog/otlp-json-logs:
21+
include:
22+
- /var/log/pods/*/*/*.log
23+
include_file_path: true
24+
operators:
25+
- id: container-parser
26+
type: container
27+
28+
processors:
29+
batch:
30+
resourcedetection:
31+
detectors: [ "env", "system" ]
32+
override: false
33+
34+
k8sattributes:
35+
# Config details in
36+
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor
37+
connectors:
38+
otlpjson:
39+
40+
exporters:
41+
otlphttp/metrics:
42+
endpoint: http://localhost:9090/api/v1/otlp
43+
otlphttp/traces:
44+
endpoint: http://localhost:4418
45+
otlphttp/logs:
46+
endpoint: http://localhost:3100/otlp
47+
debug/metrics:
48+
verbosity: detailed
49+
debug/traces:
50+
verbosity: detailed
51+
debug/logs:
52+
verbosity: detailed
53+
nop:
54+
55+
service:
56+
pipelines:
57+
traces:
58+
receivers: [ otlp ]
59+
processors: [ k8sattributes, resourcedetection, batch ]
60+
exporters: [ otlphttp/traces ]
61+
metrics:
62+
receivers: [ otlp, prometheus/collector ]
63+
processors: [ k8sattributes, resourcedetection, batch ]
64+
exporters: [ otlphttp/metrics ]
65+
logs/raw_otlpjson:
66+
receivers: [ filelog/otlp-json-logs ]
67+
# No need for processors before the otlpjson connector
68+
# Declare processors in the shared "logs" pipeline below
69+
processors: [ ]
70+
exporters: [ otlpjson ]
71+
logs/otlp:
72+
receivers: [ otlp, otlpjson ]
73+
processors: [ k8sattributes, resourcedetection, batch ]
74+
exporters: [ otlphttp/logs ]
75+
# exporters: [ otlphttp/logs, debug/logs ] # Uncomment this line to enable debug logging

‎logging-k8s-stdout-otlp-json/k8s/dice.yaml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: dice
5+
spec:
6+
selector:
7+
app: dice
8+
ports:
9+
- protocol: TCP
10+
port: 8080
11+
targetPort: 8080
12+
---
13+
apiVersion: apps/v1
14+
kind: Deployment
15+
metadata:
16+
name: dice
17+
spec:
18+
replicas: 1
19+
selector:
20+
matchLabels:
21+
app: dice
22+
template:
23+
metadata:
24+
labels:
25+
app: dice
26+
spec:
27+
containers:
28+
- name: dice
29+
image: dice:1.1-SNAPSHOT
30+
imagePullPolicy: Never
31+
ports:
32+
- containerPort: 8080
33+
env:
34+
- name: OTEL_EXPORTER_OTLP_ENDPOINT
35+
value: "http://lgtm:4318"
36+
- name: OTEL_LOGS_EXPORTER
37+
value: "experimental-otlp/stdout"
38+
- name: OTEL_RESOURCE_ATTRIBUTES
39+
value: service.name=dice,service.namespace=shop,service.version=1.1,deployment.environment=staging
40+
- name: OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_LOG_ATTRIBUTES
41+
value: "true"
42+
- name: OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_CAPTURE_KEY_VALUE_PAIR_ATTRIBUTES
43+
value: "true"
44+
- name: OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_CAPTURE_MDC_ATTRIBUTES
45+
value: "true"
46+
- name: SERVICE_NAME
47+
value: dice
48+

0 commit comments

Comments
 (0)