Skip to content

Commit 7ec07c4

Browse files
IshwarKanseIshwarKanse
committed
Add documentation for OpenShift tests
Assisted by Cursor IDE using Claude AI model.
1 parent 431dc28 commit 7ec07c4

File tree

9 files changed

+1513
-0
lines changed

9 files changed

+1513
-0
lines changed

‎tests/e2e-openshift/README.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# OpenTelemetry Operator OpenShift End-to-End Test Suite
2+
3+
This directory contains a comprehensive set of OpenShift-specific end-to-end tests for the OpenTelemetry Operator. These tests serve as **configuration blueprints** for customers to understand and deploy various OpenTelemetry observability patterns on OpenShift.
4+
5+
## 🎯 Purpose
6+
7+
These test scenarios provide OpenTelemetry configuration blueprints that demonstrate:
8+
- Integration with OpenShift-specific features (Routes, Monitoring, Security)
9+
- Real-world observability patterns and configurations
10+
- Step-by-step deployment instructions for various use cases
11+
12+
## 📋 Test Scenarios Overview
13+
14+
| Scenario | Purpose | Key Features |
15+
|----------|---------|-------------|
16+
| [route](./route/) | External Access via OpenShift Routes | Route ingress, OTLP HTTP/gRPC endpoints |
17+
| [scrape-in-cluster-monitoring](./scrape-in-cluster-monitoring/) | Prometheus Metrics Federation | In-cluster monitoring integration, metrics scraping |
18+
| [otlp-metrics-traces](./otlp-metrics-traces/) | OTLP Endpoint with Tempo | Metrics & traces collection, Tempo integration |
19+
| [multi-cluster](./multi-cluster/) | Secure Multi-Cluster Communication | TLS certificates, cross-cluster telemetry |
20+
| [must-gather](./must-gather/) | Diagnostic Information Collection | Must-gather functionality, target allocator |
21+
| [monitoring](./monitoring/) | Platform Monitoring Integration | OpenShift monitoring stack integration |
22+
| [kafka](./kafka/) | Messaging Layer for Telemetry | Kafka-based telemetry distribution |
23+
| [export-to-cluster-logging-lokistack](./export-to-cluster-logging-lokistack/) | Log Export to LokiStack | Log shipping to OpenShift logging |
24+
25+
## 🔗 OpenTelemetry Collector Components Tests
26+
27+
For detailed component-specific configurations and testing patterns, see the **OpenTelemetry Component E2E Test Suite** in the [distributed-tracing-qe](https://github.com/openshift/distributed-tracing-qe.git) repository:
28+
29+
**📡 Receivers:**
30+
- [filelog](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filelog) - File-based log collection from Kubernetes pods
31+
- [hostmetricsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/hostmetricsreceiver) - Host system metrics (CPU, memory, disk, network)
32+
- [journaldreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/journaldreceiver) - Systemd journal log collection
33+
- [k8sclusterreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/k8sclusterreceiver) - Kubernetes cluster-wide metrics
34+
- [k8seventsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/k8seventsreceiver) - Kubernetes events collection
35+
- [k8sobjectsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/k8sobjectsreceiver) - Kubernetes objects monitoring
36+
- [kubeletstatsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/kubeletstatsreceiver) - Kubelet and container metrics
37+
- [otlpjsonfilereceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/otlpjsonfilereceiver) - OTLP JSON file log ingestion
38+
39+
**📤 Exporters:**
40+
- [awscloudwatchlogsexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/awscloudwatchlogsexporter) - AWS CloudWatch Logs integration
41+
- [awsxrayexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/awsxrayexporter) - AWS X-Ray tracing export
42+
- [googlemanagedprometheusexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/googlemanagedprometheusexporter) - Google Cloud Managed Prometheus
43+
- [loadbalancingexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/loadbalancingexporter) - High availability load balancing
44+
- [prometheusremotewriteexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/prometheusremotewriteexporter) - Prometheus remote write integration
45+
46+
**⚙️ Processors:**
47+
- [batchprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/batchprocessor) - Batching for performance optimization
48+
- [filterprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filterprocessor) - Selective data filtering
49+
- [groupbyattrsprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/groupbyattrsprocessor) - Attribute-based data grouping
50+
- [memorylimiterprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/memorylimiterprocessor) - Memory usage protection
51+
- [resourceprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/resourceprocessor) - Resource attribute manipulation
52+
- [tailsamplingprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/tailsamplingprocessor) - Intelligent trace sampling
53+
- [transformprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/transformprocessor) - Advanced data transformation
54+
55+
**🔗 Connectors:**
56+
- [forwardconnector](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/forwardconnector) - Data forwarding between pipelines
57+
- [routingconnector](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/routingconnector) - Conditional data routing
58+
- [countconnector](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/countconnector) - Metrics generation from telemetry data
59+
60+
**🔧 Extensions:**
61+
- [oidcauthextension](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/oidcauthextension) - OIDC authentication
62+
- [filestorageextension](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filestorageextension) - Persistent file storage
63+
64+
These component test blueprints provide configurations for individual OpenTelemetry components that can be combined with the OpenShift integration patterns documented here.
65+
66+
## 🚀 Quick Start
67+
68+
### Prerequisites
69+
70+
- OpenShift cluster (4.12+)
71+
- OpenTelemetry Operator installed
72+
- `oc` CLI tool configured
73+
- Appropriate cluster permissions
74+
75+
### Running Tests
76+
77+
These tests use [Chainsaw](https://kyverno.github.io/chainsaw/) for end-to-end testing:
78+
79+
```bash
80+
# Run all OpenShift tests
81+
chainsaw test --test-dir tests/e2e-openshift/
82+
83+
# Run specific test scenario
84+
chainsaw test --test-dir tests/e2e-openshift/route/
85+
```
86+
87+
### Using as Configuration Templates
88+
89+
Each test directory contains:
90+
- **Configuration Files**: YAML configuration blueprints
91+
- **README.md**: Step-by-step deployment instructions
92+
- **Scripts**: Verification and setup automation
93+
94+
## 📁 Directory Structure
95+
96+
```
97+
tests/e2e-openshift/
98+
├── README.md # This overview
99+
├── route/ # External access patterns
100+
├── scrape-in-cluster-monitoring/ # Prometheus integration
101+
├── otlp-metrics-traces/ # OTLP with Tempo
102+
├── multi-cluster/ # Cross-cluster telemetry
103+
├── must-gather/ # Diagnostic collection
104+
├── monitoring/ # Platform monitoring
105+
├── kafka/ # Messaging patterns
106+
└── export-to-cluster-logging-lokistack/ # Log export patterns
107+
```
108+
109+
## 🔧 Configuration Patterns
110+
111+
### Common OpenShift Integrations
112+
113+
1. **Security Context Constraints (SCCs)**
114+
- Automated SCC annotations for namespaces
115+
- Service account configurations
116+
117+
2. **OpenShift Routes**
118+
- TLS termination options
119+
- External endpoint exposure
120+
121+
3. **Monitoring Stack Integration**
122+
- Prometheus federation
123+
- Platform monitoring labels
124+
125+
4. **RBAC Configurations**
126+
- Cluster roles and bindings
127+
- Service account permissions
128+
129+
## 📖 Documentation
130+
131+
Each test scenario includes:
132+
- **Configuration blueprints** for reference and adaptation
133+
- **Step-by-step instructions** for manual deployment
134+
- **Verification steps** to ensure proper operation
135+
- **Troubleshooting guidance** for common issues
136+
137+
## 🏷️ Labels and Annotations
138+
139+
OpenShift-specific labels and annotations used across scenarios:
140+
- `openshift.io/cluster-monitoring=true` - Enable platform monitoring
141+
- `openshift.io/sa.scc.uid-range` - UID range for security contexts
142+
- `openshift.io/sa.scc.supplemental-groups` - Supplemental groups for SCCs
143+
144+
## 🤝 Contributing
145+
146+
When adding new test scenarios:
147+
1. Include comprehensive README with step-by-step instructions
148+
2. Provide configuration blueprint examples
149+
3. Add verification scripts for testing
150+
4. Document OpenShift-specific considerations
151+
152+
## 📝 Documentation Note
153+
154+
The comprehensive READMEs in this test suite were generated using Claude AI to provide detailed, step-by-step configuration blueprints for OpenTelemetry deployments on OpenShift. These AI-generated guides aim to accelerate user adoption by providing clear, actionable documentation for complex observability scenarios.
155+
156+
## 📚 Additional Resources
157+
158+
- [OpenTelemetry Operator Documentation](https://github.com/open-telemetry/opentelemetry-operator)
159+
- [OpenShift Documentation](https://docs.openshift.com/)
160+
- [Chainsaw Testing Framework](https://kyverno.github.io/chainsaw/)

‎tests/e2e-openshift/export-to-cluster-logging-lokistack/README.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Export to Cluster Logging LokiStack Test
2+
3+
This test demonstrates how to export OpenTelemetry logs to OpenShift's cluster logging infrastructure using LokiStack for centralized log management and analysis.
4+
5+
## Test Overview
6+
7+
This test creates:
8+
1. A MinIO instance for LokiStack object storage
9+
2. A LokiStack instance for log storage and querying
10+
3. An OpenTelemetry Collector that processes and exports logs to LokiStack
11+
4. Log generation to test the end-to-end flow
12+
5. Integration with OpenShift logging UI plugin
13+
14+
## Prerequisites
15+
16+
- OpenShift cluster (4.12+)
17+
- OpenTelemetry Operator installed
18+
- Red Hat OpenShift Logging Operator installed
19+
- `oc` CLI tool configured
20+
- Appropriate cluster permissions
21+
22+
## Configuration Resources
23+
24+
### MinIO Object Storage
25+
26+
Deploy MinIO for LokiStack storage backend:
27+
28+
**Configuration:** [`install-minio.yaml`](./install-minio.yaml)
29+
30+
Creates MinIO infrastructure:
31+
- 2Gi PersistentVolumeClaim for storage
32+
- MinIO deployment with demo credentials (tempo/supersecret)
33+
- Service for internal cluster access
34+
- Secret with S3-compatible access configuration
35+
36+
### LokiStack Instance
37+
38+
Deploy LokiStack for log storage:
39+
40+
**Configuration:** [`install-loki.yaml`](./install-loki.yaml)
41+
42+
Creates a LokiStack with:
43+
- 1x.demo size for testing environments
44+
- S3-compatible storage via MinIO
45+
- v13 schema with openshift-logging tenant mode
46+
- Integration with cluster logging infrastructure
47+
48+
### OpenTelemetry Collector Configuration
49+
50+
Deploy collector with LokiStack integration:
51+
52+
**Configuration:** [`otel-collector.yaml`](./otel-collector.yaml)
53+
54+
Configures a complete collector setup with:
55+
- Service account with LokiStack write permissions
56+
- ClusterRole for accessing pods, namespaces, and nodes
57+
- Bearer token authentication for LokiStack gateway
58+
- k8sattributes processor for Kubernetes metadata
59+
- Transform processor for log level normalization
60+
- Dual pipeline: one for LokiStack export, one for debug output
61+
62+
### Log Generation
63+
64+
Generate test logs to validate the pipeline:
65+
66+
**Configuration:** [`generate-logs.yaml`](./generate-logs.yaml)
67+
68+
Creates a job that:
69+
- Generates 20 structured log entries in OTLP format
70+
- Sends logs via HTTP POST to the collector
71+
- Includes service metadata and custom attributes
72+
- Uses proper OTLP JSON structure for compatibility
73+
74+
### Logging UI Plugin
75+
76+
Enable the logging UI plugin for log visualization:
77+
78+
**Configuration:** [`logging-uiplugin.yaml`](./logging-uiplugin.yaml)
79+
80+
Configures the OpenShift console plugin for:
81+
- Log visualization in the OpenShift web console
82+
- Integration with LokiStack for log querying
83+
- Enhanced logging user interface experience
84+
85+
## Deployment Steps
86+
87+
1. **Install MinIO for object storage:**
88+
```bash
89+
oc apply -f install-minio.yaml
90+
```
91+
92+
2. **Deploy LokiStack instance:**
93+
```bash
94+
oc apply -f install-loki.yaml
95+
```
96+
97+
3. **Deploy OpenTelemetry Collector with RBAC:**
98+
```bash
99+
oc apply -f otel-collector.yaml
100+
```
101+
102+
4. **Enable logging UI plugin:**
103+
```bash
104+
oc apply -f logging-uiplugin.yaml
105+
```
106+
107+
5. **Generate test logs:**
108+
```bash
109+
oc apply -f generate-logs.yaml
110+
```
111+
112+
## Expected Resources
113+
114+
The test creates and verifies these resources:
115+
116+
### Storage Infrastructure
117+
- **MinIO**: Object storage backend with `tempo` bucket
118+
- **PVC**: 2Gi persistent volume for MinIO storage
119+
- **Secret**: `logging-loki-s3` with MinIO access credentials
120+
121+
### Logging Stack
122+
- **LokiStack**: `logging-loki` instance in demo mode
123+
- **Gateway**: HTTP gateway for log ingestion
124+
- **Storage Schema**: v13 schema with 2023-10-15 effective date
125+
126+
### OpenTelemetry Integration
127+
- **Service Account**: `otel-collector-deployment` with logging permissions
128+
- **Collector**: `otel-collector` with LokiStack exporter
129+
- **RBAC**: Cluster role for writing to LokiStack
130+
131+
### Log Generation
132+
- **Job**: `generate-logs` creating test log entries
133+
- **UI Plugin**: Logging view plugin for OpenShift console
134+
135+
## Testing the Configuration
136+
137+
The test includes verification logic in the Chainsaw test configuration.
138+
139+
**Verification Script:** [`check_logs.sh`](./check_logs.sh)
140+
141+
The script verifies:
142+
- Log generation job completes successfully
143+
- Collector exports logs to LokiStack gateway
144+
- LokiStack gateway receives and processes log entries
145+
- End-to-end log flow from generation to storage
146+
147+
## Additional Verification Commands
148+
149+
Verify the logging infrastructure:
150+
151+
```bash
152+
# Check LokiStack status
153+
oc get lokistack logging-loki -o yaml
154+
155+
# Check MinIO deployment
156+
oc get deployment minio -o yaml
157+
158+
# View LokiStack gateway service
159+
oc get svc logging-loki-gateway-http
160+
161+
# Check collector service account permissions
162+
oc auth can-i create application --as=system:serviceaccount:openshift-logging:otel-collector-deployment
163+
164+
# Port forward to MinIO for bucket verification
165+
oc port-forward svc/minio 9000:9000 &
166+
curl -u tempo:supersecret http://localhost:9000/minio/health/live # Using demo test credentials
167+
168+
# Check collector metrics
169+
oc port-forward svc/otel-collector 8888:8888 &
170+
curl http://localhost:8888/metrics | grep otelcol_exporter
171+
```
172+
173+
## Verification
174+
175+
The test verifies:
176+
- ✅ MinIO is deployed and accessible as object storage
177+
- ✅ LokiStack instance is ready and configured
178+
- ✅ OpenTelemetry Collector has proper RBAC permissions
179+
- ✅ Collector is configured with LokiStack OTLP exporter
180+
- ✅ Bearer token authentication is working
181+
- ✅ Log generation job completes successfully
182+
- ✅ Logs are processed through k8sattributes and transform processors
183+
- ✅ Logs are successfully exported to LokiStack
184+
- ✅ Logging UI plugin is enabled for log visualization
185+
186+
## Key Features
187+
188+
- **LokiStack Integration**: Native integration with OpenShift cluster logging
189+
- **Object Storage**: MinIO backend for log persistence
190+
- **Authentication**: Bearer token authentication with service accounts
191+
- **Log Processing**: Kubernetes attributes and log transformation
192+
- **OTLP Protocol**: Uses OTLP HTTP for log export to LokiStack
193+
- **Multi-Pipeline**: Separate pipelines for LokiStack export and debug output
194+
- **UI Integration**: Logging console plugin for log visualization
195+
196+
## Configuration Notes
197+
198+
- LokiStack runs in `1x.demo` size for testing environments
199+
- MinIO uses ephemeral storage with demo credentials (tempo/supersecret) - **FOR TESTING ONLY**
200+
- Collector uses bearer token authentication for LokiStack access
201+
- Service CA certificate is used for TLS communication with LokiStack
202+
- Log processors add Kubernetes metadata and normalize severity levels
203+
- The application log type is set for proper tenant routing in LokiStack

0 commit comments

Comments
 (0)