Skip to content

Commit 3669468

Browse files
cpuguy83cpuguy83
committed
ci: Fix trace setup and switch to otel collector
otel-collector will export a standard file format which can be collected and imported into any tool we want. Also adds configurations that people can use locally to take the generated trace file from a CI run and load it locally into jaeger. NOTE: It looks like we had a bad copy/paste that didn't get caught when we tried to switch to the ghcr mirror we have of jaeger, so tracing hasn't been working properly since then. With this change we also just add a warning to the build that we couldn't setup tracing if it fails rather than exiting non-zero and failing the build--no need for a mirror. Signed-off-by: Brian Goff <cpuguy83@gmail.com>
1 parent afe3864 commit 3669468

File tree

7 files changed

+145
-11
lines changed

7 files changed

+145
-11
lines changed

‎.github/otel-collector-ci.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
receivers:
2+
otlp:
3+
protocols:
4+
http:
5+
endpoint: 0.0.0.0:4318
6+
grpc:
7+
endpoint: 0.0.0.0:4317
8+
9+
exporters:
10+
file:
11+
path: /data/traces.jsonl
12+
format: json
13+
14+
service:
15+
pipelines:
16+
traces:
17+
receivers: [otlp]
18+
exporters: [file]

‎.github/workflows/ci.yml‎

Lines changed: 28 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -131,26 +131,41 @@ jobs:
131131
username: ${{ github.actor }}
132132
password: ${{ secrets.GITHUB_TOKEN }}
133133

134-
- name: Setup jaeger
134+
- name: Setup otel-collector
135135
run: |
136136
set -e
137-
docker run -d --net=host --restart=always --name jaeger -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:1.62.0
138-
docker0_ip="$(ip -f inet addr show docker0 | grep -Po 'inet \K[\d.]+')"
139-
echo "OTEL_EXPORTER_OTLP_ENDPOINT=ghcr.io/project-dalec/dalec/mirror/dockerhub/jaegertracing/all-in-one:1.62.0" >> "${GITHUB_ENV}"
137+
mkdir -p /tmp/otel-traces
138+
chmod 777 /tmp/otel-traces
139+
140+
if ! docker run -d --net=host --restart=always --name otel-collector \
141+
-v ${{ github.workspace }}/.github/otel-collector-ci.yaml:/etc/otelcol-contrib/config.yaml:ro \
142+
-v /tmp/otel-traces:/data:rw \
143+
otel/opentelemetry-collector-contrib:0.144.0; then
144+
echo "::warning::Failed to start otel-collector for tracing, skipping trace configuration"
145+
exit 0
146+
fi
147+
148+
echo "OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318" >> "${GITHUB_ENV}"
140149
echo "OTEL_SERVICE_NAME=dalec-integration-test" >> "${GITHUB_ENV}"
141150
142151
tmp="$(mktemp)"
143152
echo "[Service]" > "${tmp}"
144-
echo "Environment=\"OTEL_EXPORTER_OTLP_ENDPOINT=http://${docker0_ip}:4318\"" >> "${tmp}"
153+
echo "Environment=\"OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318\"" >> "${tmp}"
145154
146155
sudo mkdir -p /etc/systemd/system/docker.service.d
147156
sudo mkdir -p /etc/systemd/system/containerd.service.d
148157
sudo cp "${tmp}" /etc/systemd/system/docker.service.d/otlp.conf
149158
sudo cp "${tmp}" /etc/systemd/system/containerd.service.d/otlp.conf
150159
151-
sudo systemctl daemon-reload
152-
sudo systemctl restart containerd
153-
sudo systemctl restart docker
160+
if ! sudo systemctl daemon-reload; then
161+
echo "::warning::Failed to reload systemd daemon for tracing configuration"
162+
fi
163+
if ! sudo systemctl restart containerd; then
164+
echo "::warning::Failed to restart containerd with tracing configuration"
165+
fi
166+
if ! sudo systemctl restart docker; then
167+
echo "::warning::Failed to restart docker with tracing configuration"
168+
fi
154169
155170
- name: download deps
156171
run: go mod download
@@ -259,9 +274,11 @@ jobs:
259274
run: |
260275
set -ex
261276
mkdir -p /tmp/reports
262-
curl -sSLf localhost:16686/api/traces?service=${OTEL_SERVICE_NAME} > /tmp/reports/jaeger-tests.json
263-
curl -sSLf localhost:16686/api/traces?service=containerd > /tmp/reports/jaeger-containerd.json
264-
curl -sSLf localhost:16686/api/traces?service=docker > /tmp/reports/jaeger-docker.json
277+
if [ -f /tmp/otel-traces/traces.jsonl ]; then
278+
cp /tmp/otel-traces/traces.jsonl /tmp/reports/traces.jsonl
279+
else
280+
echo "::warning::No traces file found"
281+
fi
265282
- name: Upload reports
266283
if: always()
267284
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0

‎contrib/traces/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Trace Replay Setup
2+
3+
This directory contains a Docker Compose setup to replay CI trace files locally
4+
using Jaeger UI.
5+
6+
## Usage
7+
8+
1. Download the trace artifact from a CI run (e.g., `integration-test-reports-<suite>`)
9+
2. Extract `traces.jsonl` to this directory
10+
3. Run:
11+
```bash
12+
docker compose up
13+
```
14+
4. Open Jaeger UI at http://localhost:16686
15+
16+
The otel-collector will read the `.jsonl` file and export traces to Jaeger. The
17+
Jaeger UI can then be used to browse and analyze the traces.
18+
19+
## Files
20+
21+
- `docker-compose.yaml` - Orchestrates Jaeger and OTEL Collector
22+
- `otel-collector-config.yaml` - Collector config for reading JSON files and exporting to Jaeger
23+
- `jaeger-config.yaml` - Jaeger v2 configuration with memory storage
24+
- `jaeger-ui-config.json` - Jaeger UI configuration

‎contrib/traces/docker-compose.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
services:
2+
jaeger:
3+
image: jaegertracing/jaeger:2
4+
volumes:
5+
- ./jaeger-config.yaml:/etc/jaeger/config.yaml:ro
6+
- ./jaeger-ui-config.json:/etc/jaeger/ui-config.json:ro
7+
command: ["--config", "/etc/jaeger/config.yaml"]
8+
ports:
9+
- "16686:16686" # Jaeger UI
10+
- "4317:4317" # OTLP gRPC
11+
- "4318:4318" # OTLP HTTP
12+
13+
otel-collector:
14+
image: otel/opentelemetry-collector-contrib:0.144.0
15+
volumes:
16+
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
17+
- ./:/data:ro
18+
depends_on:
19+
- jaeger

‎contrib/traces/jaeger-config.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
service:
2+
extensions: [jaeger_storage, jaeger_query]
3+
pipelines:
4+
traces:
5+
receivers: [otlp]
6+
processors: [batch]
7+
exporters: [jaeger_storage_exporter]
8+
9+
extensions:
10+
jaeger_storage:
11+
backends:
12+
memory_storage:
13+
memory:
14+
max_traces: 50000
15+
16+
jaeger_query:
17+
storage:
18+
traces: memory_storage
19+
ui:
20+
config_file: /etc/jaeger/ui-config.json
21+
22+
receivers:
23+
otlp:
24+
protocols:
25+
grpc:
26+
http:
27+
28+
processors:
29+
batch:
30+
31+
exporters:
32+
jaeger_storage_exporter:
33+
trace_storage: memory_storage

‎contrib/traces/jaeger-ui-config.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"search": {
3+
"maxLimit": 50000
4+
}
5+
}

‎contrib/traces/otel-collector-config.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
receivers:
2+
otlpjsonfile:
3+
include:
4+
- /data/*.jsonl
5+
6+
exporters:
7+
otlp/jaeger:
8+
endpoint: jaeger:4317
9+
tls:
10+
insecure: true
11+
debug:
12+
verbosity: detailed
13+
14+
service:
15+
pipelines:
16+
traces:
17+
receivers: [otlpjsonfile]
18+
exporters: [debug, otlp/jaeger]

0 commit comments

Comments
 (0)