signoz-otlp-verify

signoz-otlp-verify is a small Kubernetes-focused CLI for proving whether a self-hosted SigNoz deployment can receive OTLP traces.

It is designed for the gap after installation: the SigNoz pods may be healthy, but users still need to verify whether OTLP traffic reaches the collector and whether the trace becomes queryable in SigNoz.

Why this exists

SigNoz documents a broad self-hosted Kubernetes surface and separate ingestion troubleshooting steps. Their Trace API is documented at POST /api/v5/query_range, while OTLP ingestion commonly uses 4317 for gRPC and 4318 for HTTP.

This tool automates the first support checks:

• Discover SigNoz services, pods, endpoints, and OTLP ports in a namespace.
• Port-forward the collector service and send a synthetic OTLP/HTTP trace.
• Optionally run an OTLP/gRPC send check through telemetrygen when installed.
• Query SigNoz for the synthetic run.id when SIGNOZ_API_KEY is provided.
• Emit JSON or Markdown reports with concrete remediation hints.

Demo

Watch the full verification sequence in action:

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e .
signoz-otlp-verify run --namespace platform --execution-mode local --format markdown

For API verification:

export SIGNOZ_API_KEY="..."
signoz-otlp-verify run --namespace platform --execution-mode local --format markdown

To save reports:

signoz-otlp-verify run --namespace platform --execution-mode local --format json --output report.json
signoz-otlp-verify run --namespace platform --execution-mode local --format markdown --output report.md

To verify from inside the cluster:

docker build -t signoz-otlp-verify:latest .
kubectl apply -f deploy/job.yaml
kubectl logs -n platform job/signoz-otlp-verify

Commands

signoz-otlp-verify discover --namespace <namespace>
signoz-otlp-verify send --namespace <namespace>
signoz-otlp-verify check --namespace <namespace> --run-id <run-id>
signoz-otlp-verify run --namespace <namespace>

Requirements

• Python 3.10+
• Kubernetes Python client
• kubectl configured for the target cluster
• Access to the namespace where SigNoz is installed
• Optional: SIGNOZ_API_KEY for API-level trace visibility checks
• Optional: telemetrygen for an actual OTLP/gRPC send test on 4317

Without SIGNOZ_API_KEY, the tool still validates Kubernetes discovery, service endpoints, OTLP port exposure, and OTLP/HTTP collector acceptance.

Execution modes

• --execution-mode local: loads local kubeconfig and uses port-forwarding for OTLP/API checks.
• --execution-mode in-cluster: loads the pod service account and talks to *.svc.cluster.local directly.
• --execution-mode auto: tries in-cluster first, then local kubeconfig, then kubectl fallback.

In-cluster mode is useful when you need to isolate SigNoZ internal networking from ingress, load balancer, or local port-forwarding issues.

Current scope

Version 0.1.0 verifies traces first because traces provide the clearest end-to-end path:

1. Generate a unique run.id.
2. Send one synthetic trace to the collector.
3. Query SigNoz for that run.id when credentials are available.
4. Explain where the path fails.

Metrics and logs can be added after the first public proof.

Synthetic data hygiene

The verifier sends one diagnostic trace with explicit tags:

• service.name=signoz-diagnostic-tool
• run.id=<unique-run-id>
• is_synthetic=true
• tool.name=signoz-otlp-verify
• tool.version=0.1.0

Use these filters in SigNoz to find or exclude the diagnostic trace.

Example output

# SigNoz OTLP Verification Report

- Run ID: signoz-verify-20260510-104501-a1b2c3d4
- Namespace: platform

| Severity | Check | Result | Remediation |
| --- | --- | --- | --- |
| info | service.discovery | Found collector service signoz-otel-collector | none |
| pass | otlp.http.send | Collector accepted OTLP/HTTP trace on 4318 | none |
| warning | otlp.grpc.send | telemetrygen not found; only TCP reachability checked | install telemetrygen for full gRPC validation |
| pass | signoz.trace.query | Trace found through /api/v5/query_range | none |