Azure Data Explorer Integration#2854
Closed
roy2392 wants to merge 37 commits intoopen-telemetry:mainfrom
Closed
Conversation
- Add Terraform version constraints (>= 1.5.0) - Configure Azure providers (azurerm, azuread, random, local) - Add .gitignore for Terraform sensitive files
- Create Azure Data Explorer cluster with streaming ingestion - Define tables for traces, metrics, and logs - Configure retention policies (365 days) and hot cache (30 days) - Add JSON ingestion mappings for OTLP format
- Create Azure Kubernetes Service cluster with autoscaling - Configure Azure CNI networking and RBAC - Add Log Analytics workspace for monitoring - Enable Container Insights solution
- Create Azure AD application and service principal - Configure client secret with 180-day rotation - Grant ADX Database Ingestor and Viewer roles
- Orchestrate ADX, AKS, and identity modules - Create resource group and virtual network - Generate Kubernetes secrets YAML from Terraform - Output deployment instructions and next steps
- Document all configurable variables with examples - Add kubernetes/azure directory for generated secrets
- Create Azure-specific OTel collector configuration - Configure ADX exporter for traces, metrics, and logs - Add batch processor for efficient ingestion - Configure spanmetrics connector for service metrics
- Add template for ADX connection settings - Add Service Principal authentication variables - Update .gitignore to exclude .env.azure.local with secrets
- Create secrets template for ADX credentials - Document all required Azure authentication values
- Configure Kubernetes-aware OTel collector for ADX - Add k8sattributes processor for pod metadata - Configure spanmetrics connector for service metrics
- Create automated deployment script with Terraform integration - Support manual deployment with --skip-terraform flag - Add colorized output and status reporting
- Configure Azure Data Explorer datasource for Grafana - Add separate datasource views for traces, metrics, and logs - Support Service Principal authentication
- Create comprehensive APM dashboard for ADX - Add service overview stats (spans, errors, latency) - Add request rate and P95 latency time series - Add error logs visualization - Add service performance summary table
- Create standalone KQL schema for ADX tables - Define traces, metrics, and logs table structures - Add JSON ingestion mappings for OTLP format - Configure retention and caching policies
- Add service overview queries - Add trace analysis examples - Add latency and error analysis - Add log correlation queries - Add metrics analysis examples
- Add architecture overview and prerequisites - Document Terraform and manual deployment options - Include configuration reference tables - Add troubleshooting section - Document cost estimation
- Add service-flow image reference - Add mermaid architecture diagram showing data flow - Add mermaid repo structure diagram for Azure-specific files - Document benefits vs SaaS observability tools - Link to detailed Azure deployment guide
|
|
github-actions
bot
added
the
helm-update-required
added 2 commits
January 5, 2026 10:56
- Change auto_scaling_enabled to enable_auto_scaling - Change automatic_upgrade_channel to automatic_channel_upgrade
feat: integrate Azure Data Explorer as telemetry backend
- Update default Kubernetes version from 1.29 to 1.32 (1.29 now requires LTS) - Add explicit service_cidr (172.16.0.0/16) to avoid overlap with VNet - Add dns_service_ip (172.16.0.10) for AKS DNS service
Update helm
- Add comprehensive INTEGRATE_YOUR_SERVICES.md with examples for: - Python, Node.js, Java, .NET, and Go instrumentation - OTel Collector configuration for ADX - Kubernetes and Docker deployment examples - KQL queries for monitoring - Grafana integration - Update README and AZURE_DEPLOYMENT.md with links to new guide
- Add ADX configuration section to values.yaml with cluster URI, database, tables - Add Azure authentication settings (tenantId, clientId, clientSecret, existingSecret) - Update otel-collector-deployment.yaml to inject ADX env vars from secret - Create adx-secret.yaml template for automatic secret creation - Create grafana-adx-datasource.yaml for automatic datasource provisioning - Create values-azure.yaml example file with full Azure configuration - Update README with Azure Data Explorer quick start guide
- Add Terraform resource to generate values-generated.yaml with ADX credentials - Update outputs.tf with Helm installation commands - Update deploy-to-aks.sh to use Helm by default (with kubectl fallback) - Add generated files to .gitignore (secrets.yaml, values-generated.yaml) - Update README with integrated deployment workflow Terraform now generates a complete Helm values file that the deployment script automatically uses, creating a seamless infrastructure-to-deployment pipeline.
Migrate from Service Principal with client secret to Azure AD Workload Identity for secure, secret-less authentication to Azure Data Explorer. Changes: - Enable OIDC issuer and Workload Identity on AKS cluster - Replace Azure AD App/Service Principal with User-Assigned Managed Identity - Add Federated Identity Credential linking K8s service account to identity - Update OTel Collector config to use `use_azure_auth: true` - Update Helm chart with conditional Workload Identity support - Remove namespace.yaml template (conflicts with Helm --create-namespace) - Fix RBAC ClusterRoleBinding service account name Benefits: - No secrets stored in Kubernetes - Automatic token rotation (tokens expire in ~1 hour) - Reduced blast radius (tokens only valid for specific pod) - Full Azure AD audit logging - Simplified secret management
Member
|
Hello @roy2392, I think this PR was pointed to the wrong repo. What we have is a list of links to each vendor on the main README: https://github.com/open-telemetry/opentelemetry-demo?tab=readme-ov-file#demos-featuring-the-astronomy-shop Feel free to have a fork of the Demo and open a PR adding a link to it in here. As of now, I'm closing this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR integrates Azure Data Explorer (ADX) as the telemetry backend for the OpenTelemetry Demo, providing a cost-effective alternative to SaaS observability tools like Datadog and Coralogix for organizations using Azure.
Motivation
Changes
Infrastructure (Terraform)
OpenTelemetry Collector
otelcol-config-azure.ymlwith ADX exporterKubernetes
deploy-to-aks.sh)Grafana
Documentation
Architecture
Microservices (17) --> OTel Collector --> Azure Data Explorer --> Grafana
(ADX Exporter) (Traces/Metrics/Logs) (KQL Dashboards)
Files Changed
terraform/kubernetes/azure/src/otel-collector/otelcol-config-azure.ymlsrc/grafana/provisioning/adx/scripts/deploy-to-aks.shdocs/AZURE_DEPLOYMENT.mdHow to Test
./scripts/deploy-to-aks.sh
OTelTraces | count
OTelMetrics | count
OTelLogs | count
kubectl port-forward -n otel-demo svc/grafana 3000:3000
Checklist
Related Links